> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
> What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.
This assumes other technological breakthroughs don't also happen in parallel. I'll go out on a limb with nothing more than a hunch that we are in the beginning stages of understanding mathematics, physics, and biology.
It does not. This is basic thermodynamics. Unless we are using computers made out of something other than matter, powered by something other than energy, thermodynamics bounds us all.
Ok, but what if the AI figures out a new thermodynamics? A completely new thermodynamics, that's vastly more powerful than human thermodynamics, but of course humans won't be able to comprehend it, because it's so much more powerful, that you need the superintelligence to handle it. With a thermodynamics like that, anything could be possible, and remember: what we have now is the worst it will ever be.
pyridines · · focus · HN ↗
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
breuleux · · focus · HN ↗
For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.
jasondigitized · · focus · HN ↗
stouset · · focus · HN ↗
Past that is believing in magic.
meindnoch · · focus · HN ↗