> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
Some of it has to do with the fact that many people pushed forward training themselves in a way they understood would better themselves for the future. They put in time, and now their sense of value is in question. They're forced to think about things like value (of life, themselves, and hopefully others), and the things they want to remain valuable in the future.
It's interesting to me that you only mentioned the people using 'AI' in the 'short-term' ways, and not the ones that use it to better themselves in the 'long-term' ways. You can spend your own time focusing on either usage, it's really up to you and your concerns. Either group's sense of value is what determines their behavior. Where they spend their time and thinking must be elsewhere, and you disagree with it. Who judges the quality of time spent? You do.
Is it more useful to think about self-improvement, and how to navigate the future in ways that might help you re-establish value of yourself, life, and others? Acquiring knowledge is a struggle, the author mentioned this. There is also Plato's Allegory of the Cave, which highlights some of that struggle, a resistance to change. And we're all limited by time, our genes, our station in life.
The only way to help anyone out of the cave, is to help them believe something different about themselves. To help them believe there is good reason to spend time going deeper into knowledge, or at the very least, allow others with the passion and station for it to do so.
If not for the very least reason that it keeps us 'in the loop' of some central idea behind intelligence (prediction?). Or because we feel it keeps us safer, as a fallback measure because we acknowledge we have to trust other's knowledge to exist.
pyridines · · focus · HN ↗
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
jazzprogramming · · focus · HN ↗
But suppose some future holy grail AI can do much more than that.
Suppose it could find a cure for cancer, fix the climate, build fusion plants, Dyson spheres and so on.
But nobody can understand anymore how any of it works. We just ask and then trust the AI to deliver (as it always has).
Isn't it fun to imagine how life would look like in that scenario?
We would probably no longer care about code, engineering or even physics and mathematics among other things. We would probably mainly care about
birdsongs · · focus · HN ↗
> Isn't it fun to imagine how life would look like in that scenario?
This is horrifying to me.
Loquebantur · · focus · HN ↗
People succumb to defeatism and acquiesce to regressing to zoo animals, with AI as their caretakers.
They simply cannot help but to apply the economic gauge of short term profits to value the alternatives.
Even though, obviously, here long term human survival and living conditions are at stake, necessitating an entirely different set of considerations.
itsalwaysgood · · focus · HN ↗
It's interesting to me that you only mentioned the people using 'AI' in the 'short-term' ways, and not the ones that use it to better themselves in the 'long-term' ways. You can spend your own time focusing on either usage, it's really up to you and your concerns. Either group's sense of value is what determines their behavior. Where they spend their time and thinking must be elsewhere, and you disagree with it. Who judges the quality of time spent? You do.
Is it more useful to think about self-improvement, and how to navigate the future in ways that might help you re-establish value of yourself, life, and others? Acquiring knowledge is a struggle, the author mentioned this. There is also Plato's Allegory of the Cave, which highlights some of that struggle, a resistance to change. And we're all limited by time, our genes, our station in life.
The only way to help anyone out of the cave, is to help them believe something different about themselves. To help them believe there is good reason to spend time going deeper into knowledge, or at the very least, allow others with the passion and station for it to do so.
If not for the very least reason that it keeps us 'in the loop' of some central idea behind intelligence (prediction?). Or because we feel it keeps us safer, as a fallback measure because we acknowledge we have to trust other's knowledge to exist.
LtWorf · · focus · HN ↗