Usually, if you know enough about your algorithms to select an appropriate float alternative, you also know enough to fix your float code and that's what you should actually do.
That said, some of these aren't alternatives. Symbolic computation is a different thing entirely. Interval arithmetic can be built atop floats (e.g. IEEE-1788) and has its own zoo of unintuitive behaviors. BCD is better called a historical artifact than an alternative these days.
It's really just rationals and decimal floats in this list, which probably don't solve the issues you have if you're considering float alternatives.
You can't "fix" floating point code if you are looking for deterministic answers. You just have to use other data types to handle money or complex mathematical operations like 0.2+0.1, no ifs and buts.
> Floating point is deterministic, what are you talking about?
Order of operations can change a result, for example. I suspect you mean that the algorithm never changes. While op means that mathematical operations which most folks would expect to be reliable are not.
There are enough problems for a 44 page paper titled "What Every Computer Scientist Should Know About Floating-Point Arithmetic"[1] I don't quibble on the language because I know what people mean.
Most folks won't encounter most of the issues, generally. But expose your code to a large enough dataset, or be like me and write a CAD/CAM system with motion control and experience most of them.
No. It's been quite some time since I realized that all language is a pidgin used to translate between individuals' unique lived experiences and points of reference. And find communication much more fluid and less confrontational when the focus is on shared meaning rather than perfect word choice. Especially when working with non-native speakers, but also just people in general. Stephen Fry captures the feeling: <a href="https://www.youtube.com/watch?v=Ovi7uQbtKas" rel="nofollow">https://www.youtube.com/watch?v=Ovi7uQbtKas
When TZubiri made their original comment, I understood they were speaking about some or all of the issues outlined in the paper I linked. If you didn't, that's ok. If you think the referenced paper missed something, it's OK to add that.
> You make it really hard to take you seriously.
Thanks for following the thread. I'll clarify on my intended meaning was indeed a strict actual definition of determinism, but a broader definition of floating point, to include its actual usage. But fwiw, it was indeed possible that I was someone who confuses determinism for precision, but no.
When I said that floating points are not deterministic, I wasn't very precise, but I do think that broadly speaking, floating point arithmetic, as used today, foregoes determinism, and this results from the very ethos of the foundational IEEE754 data type, the goal is to have a data type for approximate answers, turns out that when exact answers are sacrificed in the name of speed, so is determinism. And this has huge effects on modern day, Floating Point is used on separate hardware with parallel operations, and there's race conditions that make most Machine Learning and AI computing irreproducible, and that indeed seems to be a consequence, as you mention, of the lack of associativity of FP.
So, that said, I would make two clarifications:
>-- Floating Points
>++ Floating Point computing
where by Floating Point computing would mean the actual application computing that we build, as opposed to "Floating points" referring to the ideal ancient standardized hardware layer abstractions.
And if necessary:
> -- is
> ++ tends to be
In order to be perfectly correct, which after all, is what we are going after.
So if pressed, I wouldn't say "Floating points are not deterministic" but "Floating Point computing tends to be non-deterministic", but I would feel very comfortable shorthanding it to "Floating Points are non-deterministic" anyways.
The paper cited is a bit hard for me, so I can't verify if it matches what I'm saying. But I imagine by the date, it wouldn't be able to address the issues that we can empirically from the advent of ML systems, but maybe it did foresee from a theoretical standpoint some of their limitations.
There's a between-the-lines thesis here that there's two main schools of computing nowadays, one that seeks perfection, and another that seeks approximations, the CPU/GPU dichotomy is roughly analogous to the Mathematics/Physics vs Engineering/Industrial dichotomy.rroot@t14:/mnt/c/Users/TomZubiri/Desktop# cat fixed.txt
Thanks for following the thread. I'll clarify on my intended meaning was indeed a strict actual definition of determinism, but a broader definition of floating point, to include its actual usage. But fwiw, it was indeed possible that I was someone who confuses determinism for precision, but no.
When I said that floating points are not deterministic, I wasn't very precise, but I do think that broadly speaking, floating point arithmetic, as used today, foregoes determinism, and this results from the very ethos of the foundational IEEE754 data type, the goal is to have a data type for approximate answers, turns out that when exact answers are sacrificed in the name of speed, so is determinism. And this has huge effects on modern day, Floating Point is used on separate hardware with parallel operations, and there's race conditions that make most Machine Learning and AI computing irreproducible, and that indeed seems to be a consequence, as you mention, of the lack of associativity of FP.
So, that said, I would make two clarifications:
>-- Floating Points
>++ Floating Point computing
where by Floating Point computing would mean the actual application computing that we build, as opposed to "Floating points" referring to the ideal ancient standardized hardware layer abstractions.
And if necessary:
> -- is
> ++ tends to be
In order to be perfectly correct, which after all, is what we are going after.
So if pressed, I wouldn't say "Floating points are not deterministic" but "Floating Point computing tends to be non-deterministic", but I would feel very comfortable shorthanding it to "Floating Points are non-deterministic" anyways.
The paper cited is a bit hard for me, so I can't verify if it matches what I'm saying. But I imagine by the date, it wouldn't be able to address the issues that we can empirically from the advent of ML systems, but maybe it did foresee from a theoretical standpoint some of their limitations.
There's a between-the-lines thesis here that there's two main schools of computing nowadays, one that seeks perfection, and another that seeks approximations, the CPU/GPU dichotomy is roughly analogous to the Mathematics/Physics vs Engineering/Industrial dichotomy.
> when exact answers are sacrificed in the name of speed
I have found in <a href="https://github.com/timschmidt/hyperreal" rel="nofollow">https://github.com/timschmidt/hyperreal and the geometry layers built atop it, that exactness and speed are not fundamentally opposed. Hyperreal retains exact mathematical structure at the scalar level, and at each level higher up through vertices, vectors, matrices, triangles, curves, etc. And uses it to choose optimal datatypes, dispatch against the fastest algorithms, and eliminate large chunks of needless calculation where simpler proofs are sufficient.
The downside is a bit of additional code and API complexity. But the slightly more complex API accurately represents issues like uncertainty in decidable equality of arbitrary computables in bounded time, which is a limitation of math itself, so representing it accurately in the API permits the algorithms to be more robust.
> Floating Point is used on separate hardware with parallel operations, and there's race conditions that make most Machine Learning and AI computing irreproducible
Deterministic reproducibility is, I think, one of the most awesome superpowers of a system like hyperreal. It makes simulation much more interesting and useful. As well as cross-platform use.
floating point arithmetic, as used today, foregoes determinism, and this results from the very ethos of the foundational IEEE754 data type
This was somewhat true in the past, but the situation has been improving dramatically in recent years to the point where FP determinism is completely feasible. The remaining hurdles are primarily on the toolchains/kernel side. I have a library called rfloat that you can drop into most C/C++ code for practical determinism without thought (subject to documented caveats), for example. You can do the same thing manually with some more careful attention.
I'm in a very remote corner of the world on bad Wi-Fi though, so you'll have to forgive omitted links.
>in recent years to the point where FP determinism is completely feasible.
Ok, sure, it is feasible, but is that how it's actually used? Or is it used in GPUs with thousands of processors running in parallel, and batching different operations, (and with temperature settings that add even more pseudo-indeterminism purposefully).
It's funny that the customs actually go in the opposite direction of fabricating even more indeterminism, whether it is for being an accountability sink, or for fudging data to claim IP over the new mungled data, the FP/GPU/ML folk want magic, not determinism.
AlotOfReading · · focus · HN ↗
That said, some of these aren't alternatives. Symbolic computation is a different thing entirely. Interval arithmetic can be built atop floats (e.g. IEEE-1788) and has its own zoo of unintuitive behaviors. BCD is better called a historical artifact than an alternative these days.
It's really just rationals and decimal floats in this list, which probably don't solve the issues you have if you're considering float alternatives.
TZubiri · · focus · HN ↗
messe · · focus · HN ↗
> You just have to use other data types to handle money or complex mathematical operations like 0.2+0.1
Such as... decimal floating point.
timschmidt · · focus · HN ↗
Order of operations can change a result, for example. I suspect you mean that the algorithm never changes. While op means that mathematical operations which most folks would expect to be reliable are not.
messe · · focus · HN ↗
timschmidt · · focus · HN ↗
Most folks won't encounter most of the issues, generally. But expose your code to a large enough dataset, or be like me and write a CAD/CAM system with motion control and experience most of them.
That's why I wrote hyperreal[2]
1: <a href="https://www.cs.tufts.edu/cs/40/docs/WhatEveryComputerScientistShouldKnowAboutFloatingPointArithmetic.pdf" rel="nofollow">https://www.cs.tufts.edu/cs/40/docs/WhatEveryComputerScienti...
2: <a href="https://github.com/timschmidt/hyperreal" rel="nofollow">https://github.com/timschmidt/hyperreal
drfloyd51 · · focus · HN ↗
You make it really hard to take you seriously.
timschmidt · · focus · HN ↗
When TZubiri made their original comment, I understood they were speaking about some or all of the issues outlined in the paper I linked. If you didn't, that's ok. If you think the referenced paper missed something, it's OK to add that.
> You make it really hard to take you seriously.
Same, bud.
TZubiri · · focus · HN ↗
When I said that floating points are not deterministic, I wasn't very precise, but I do think that broadly speaking, floating point arithmetic, as used today, foregoes determinism, and this results from the very ethos of the foundational IEEE754 data type, the goal is to have a data type for approximate answers, turns out that when exact answers are sacrificed in the name of speed, so is determinism. And this has huge effects on modern day, Floating Point is used on separate hardware with parallel operations, and there's race conditions that make most Machine Learning and AI computing irreproducible, and that indeed seems to be a consequence, as you mention, of the lack of associativity of FP.
So, that said, I would make two clarifications:
>-- Floating Points
>++ Floating Point computing
where by Floating Point computing would mean the actual application computing that we build, as opposed to "Floating points" referring to the ideal ancient standardized hardware layer abstractions.
And if necessary:
> -- is
> ++ tends to be
In order to be perfectly correct, which after all, is what we are going after.
So if pressed, I wouldn't say "Floating points are not deterministic" but "Floating Point computing tends to be non-deterministic", but I would feel very comfortable shorthanding it to "Floating Points are non-deterministic" anyways.
The paper cited is a bit hard for me, so I can't verify if it matches what I'm saying. But I imagine by the date, it wouldn't be able to address the issues that we can empirically from the advent of ML systems, but maybe it did foresee from a theoretical standpoint some of their limitations.
There's a between-the-lines thesis here that there's two main schools of computing nowadays, one that seeks perfection, and another that seeks approximations, the CPU/GPU dichotomy is roughly analogous to the Mathematics/Physics vs Engineering/Industrial dichotomy.rroot@t14:/mnt/c/Users/TomZubiri/Desktop# cat fixed.txt Thanks for following the thread. I'll clarify on my intended meaning was indeed a strict actual definition of determinism, but a broader definition of floating point, to include its actual usage. But fwiw, it was indeed possible that I was someone who confuses determinism for precision, but no.
When I said that floating points are not deterministic, I wasn't very precise, but I do think that broadly speaking, floating point arithmetic, as used today, foregoes determinism, and this results from the very ethos of the foundational IEEE754 data type, the goal is to have a data type for approximate answers, turns out that when exact answers are sacrificed in the name of speed, so is determinism. And this has huge effects on modern day, Floating Point is used on separate hardware with parallel operations, and there's race conditions that make most Machine Learning and AI computing irreproducible, and that indeed seems to be a consequence, as you mention, of the lack of associativity of FP.
So, that said, I would make two clarifications:
>-- Floating Points
>++ Floating Point computing
where by Floating Point computing would mean the actual application computing that we build, as opposed to "Floating points" referring to the ideal ancient standardized hardware layer abstractions.
And if necessary:
> -- is
> ++ tends to be
In order to be perfectly correct, which after all, is what we are going after.
So if pressed, I wouldn't say "Floating points are not deterministic" but "Floating Point computing tends to be non-deterministic", but I would feel very comfortable shorthanding it to "Floating Points are non-deterministic" anyways.
The paper cited is a bit hard for me, so I can't verify if it matches what I'm saying. But I imagine by the date, it wouldn't be able to address the issues that we can empirically from the advent of ML systems, but maybe it did foresee from a theoretical standpoint some of their limitations.
There's a between-the-lines thesis here that there's two main schools of computing nowadays, one that seeks perfection, and another that seeks approximations, the CPU/GPU dichotomy is roughly analogous to the Mathematics/Physics vs Engineering/Industrial dichotomy.
timschmidt · · focus · HN ↗
I have found in <a href="https://github.com/timschmidt/hyperreal" rel="nofollow">https://github.com/timschmidt/hyperreal and the geometry layers built atop it, that exactness and speed are not fundamentally opposed. Hyperreal retains exact mathematical structure at the scalar level, and at each level higher up through vertices, vectors, matrices, triangles, curves, etc. And uses it to choose optimal datatypes, dispatch against the fastest algorithms, and eliminate large chunks of needless calculation where simpler proofs are sufficient.
The downside is a bit of additional code and API complexity. But the slightly more complex API accurately represents issues like uncertainty in decidable equality of arbitrary computables in bounded time, which is a limitation of math itself, so representing it accurately in the API permits the algorithms to be more robust.
> Floating Point is used on separate hardware with parallel operations, and there's race conditions that make most Machine Learning and AI computing irreproducible
Deterministic reproducibility is, I think, one of the most awesome superpowers of a system like hyperreal. It makes simulation much more interesting and useful. As well as cross-platform use.
AlotOfReading · · focus · HN ↗
I'm in a very remote corner of the world on bad Wi-Fi though, so you'll have to forgive omitted links.
TZubiri · · focus · HN ↗
Ok, sure, it is feasible, but is that how it's actually used? Or is it used in GPUs with thousands of processors running in parallel, and batching different operations, (and with temperature settings that add even more pseudo-indeterminism purposefully).
It's funny that the customs actually go in the opposite direction of fabricating even more indeterminism, whether it is for being an accountability sink, or for fudging data to claim IP over the new mungled data, the FP/GPU/ML folk want magic, not determinism.