> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
A metaphor I'm constantly drawn to is the transition from agrarian to urban societies following the Industrial Revolution. Somebody who somehow saw the Industrial Revolution coming from the perspective of somebody living in an agrarian society might have envisioned it leading to 'super farms.' And it did.
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
Technology doesn’t make slavery obsolete, it just makes your slaves more efficient.
There had always been abolitionists who were against slavery simply because they thought it was immoral, not because they thought slaves weren’t needed anymore.
The problem that moral opposition (or support) of something runs into is that when the economic benefit is positive, those who are willing to set aside morals will have more influence, power, and wealth than those who are unwilling. So it makes difficult for things to ever really change, especially in a permanent way. Even between nations themselves - those that side aside morals will tend to end up with more influence than those that try to act morally.
The economic problem with slavery is that you're paying for absolutely everything for a slave - their food, housing, healthcare, and more. And if they ever get sick, injured, or whatever else then you're also paying for that. And that's after you pay a huge lump sum up front to purchase them in the first place. In the US a run of the mill slave cost about the same as a house, and it continued to track that price oddly closely.
The industrial revolution created a wide availability of cheap hourly laborers where the person hiring paid only their wages, and had no obligation for any of their expenses. People utilizing this labor suddenly not only had morality on their side, but even had an economic edge over those still clinging onto slavery.
----
Another relevant point I think is that cognitive dissonance also tends to kick in when something is lucrative. For instance I think Aristotle's work, Politics, is amazingly insightful and informative. Yet his defense of slavery was not only the weakest part of his work, by far, but basically just garbage. Unlike his other arguments which clearly started from first principles and worked their way forward, it's clear with slavery that he started with an assumption that he wanted to be true, and concocted an argument going backwards from there.
Incidentally, he also predicted the end of slavery with an industrial revolution, some 2500 years ago: "For if every instrument could accomplish its own work, obeying or anticipating the will of others, like the statues of Daedalus, or the tripods of Hephaestus, which, says the poet, 'Of their own accord entered the assembly of the Gods.' If, in like manner, the shuttle would weave and the plectrum touch the lyre without a hand to guide them, chief workmen would not want servants, nor masters slaves." [1]
pyridines · · focus · HN ↗
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
somenameforme · · focus · HN ↗
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
nkrisc · · focus · HN ↗
There had always been abolitionists who were against slavery simply because they thought it was immoral, not because they thought slaves weren’t needed anymore.
somenameforme · · focus · HN ↗
The economic problem with slavery is that you're paying for absolutely everything for a slave - their food, housing, healthcare, and more. And if they ever get sick, injured, or whatever else then you're also paying for that. And that's after you pay a huge lump sum up front to purchase them in the first place. In the US a run of the mill slave cost about the same as a house, and it continued to track that price oddly closely.
The industrial revolution created a wide availability of cheap hourly laborers where the person hiring paid only their wages, and had no obligation for any of their expenses. People utilizing this labor suddenly not only had morality on their side, but even had an economic edge over those still clinging onto slavery.
----
Another relevant point I think is that cognitive dissonance also tends to kick in when something is lucrative. For instance I think Aristotle's work, Politics, is amazingly insightful and informative. Yet his defense of slavery was not only the weakest part of his work, by far, but basically just garbage. Unlike his other arguments which clearly started from first principles and worked their way forward, it's clear with slavery that he started with an assumption that he wanted to be true, and concocted an argument going backwards from there.
Incidentally, he also predicted the end of slavery with an industrial revolution, some 2500 years ago: "For if every instrument could accomplish its own work, obeying or anticipating the will of others, like the statues of Daedalus, or the tripods of Hephaestus, which, says the poet, 'Of their own accord entered the assembly of the Gods.' If, in like manner, the shuttle would weave and the plectrum touch the lyre without a hand to guide them, chief workmen would not want servants, nor masters slaves." [1]
[1] - <a href="https://classics.mit.edu/Aristotle/politics.mb.txt" rel="nofollow">https://classics.mit.edu/Aristotle/politics.mb.txt