> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
Imagine the first time electric calculators calculated the square root of 5. I'm sure people would verify again and again if what the circuits calculated was right.
Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.
How crazy will the world be once our [current gen] ANN are like that!
It is insane how many times I see this false analogy repeated on HN over and over (analogy of a deterministic-by-design calculator device (or a compiler, etc.) and a non-deterministic-by-design LLM software).
You're missing the forest for the trees here. The point OP is trying to make is that calculators originally were essentially non-deterministic. Technology will go from unreliable to reliable.
LLMs are not non-deterministic by design. The randomness in the output is artificially injected for ergonomic reasons. (Yes, the non-determinism in production systems is different, but also not by design)
You don’t train calculators on how to add. All calculators will more or less come up with the same answer to some deterministic and quantifiable level of accuracy and precision.
pyridines · · focus · HN ↗
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
xtracto · · focus · HN ↗
Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.
How crazy will the world be once our [current gen] ANN are like that!
What an amazing thought.
sedan_baklazhan · · focus · HN ↗
CrimsonRain · · focus · HN ↗
Nothing is stopping LLMs to be more deterministic/correct over time.
Also you yourself is nondeterministic :)
sedan_baklazhan · · focus · HN ↗
All people are. That is how automation appeared to begin with - to provide deterministic behavior.
CrimsonRain · · focus · HN ↗
So what's stopping "non deterministic" AIs from doing the same, at faster/larger scale? :)
freecodeio · · focus · HN ↗
you clearly don't know how transistors work, which is ironic because it's a thread in hackernews about LLMs and mathematics
CrimsonRain · · focus · HN ↗
uejfiweun · · focus · HN ↗
sedan_baklazhan · · focus · HN ↗
xorcist · · focus · HN ↗
hackinthebochs · · focus · HN ↗
goalieca · · focus · HN ↗