> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
A metaphor I'm constantly drawn to is the transition from agrarian to urban societies following the Industrial Revolution. Somebody who somehow saw the Industrial Revolution coming from the perspective of somebody living in an agrarian society might have envisioned it leading to 'super farms.' And it did.
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
> I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing).
If an AI can replace me on the mental aspects of work, and robotics are on their way to replacing humanity on the physical aspects of work... then what's left? When there was agrarian societies, there were writers, priests, bankers, merchants, and laborers before and after - I really don't think things were that unclear even at the time. Now that we have machines that are close to exceeding humans in every way, what good are humans?
I've struggled with this question as well, and this is why I reject the premise of the historical pattern of technology enabling us to "move up" to something else.
At the same time, I agree with the original comment as well. I don't think this necessarily leads to some doomsday scenario. Whatever happens it'll likely be better for us and imo we will merge with the AIs at some point, so it won't be a question of us vs them.
Only if you have a zero-sum game mindset. I suppose it does make sense that AI-native intelligences are just smarter than hybrid AI-human entities (biologically enhanced or uploads) but that doesn't mean there needs to be conflict between the two. The universe is huge. :)
I personally don't think there will be a conflict, but more just an extinguishing of all things human, outside of niche forms of entertainment and care. It won't be machines that cause the conflict, but the frustrated and bored masses fighting over the scraps left behind by the machines and ultra-wealthy.
I actually see it more likely the opposite - a subset of humans will retreat into a (vastly smaller) human-first world, and AI will be of a separate world concerned with material/scientific/research concerns and appeases the masses who are still around.
We can already see this today in the form of upper class parents (like myself) who vastly restrict their kids screen time and ban social media. We may see AI-free societies that form, but supported by AI. The jobs and roles people play within that society may deal with services that really need to be done by a human, being a buffer from AI to the rest of this society, or just straight up fake.
I personally think this is close to a doomsday scenario - but it doesn't end with a big bang, but more of slow quiet death.
I figure both will happen. Some people will want to live naturally, some to merge. I sometimes think with the data centers in space discussion that it'll end up with AIs in space and the Earth as something like a nature reserve.
I think many people will be quite content not merging with AI and having AI do everything for them. Perhaps only a small subset of people will want to merge with AI. They may not be allowed to though, if majority of people fine not merging then they may ban those who want to merge.
The subset of people who will want to merge will be people who will not be content with having no control over their future. No ability to create new inventions, new discoveries, new ways of working. They will not be content with just having everything provided to them, playing the social games that will remain and think their environment is a prison.
pyridines · · focus · HN ↗
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
somenameforme · · focus · HN ↗
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
gWPVhyxPHqvk · · focus · HN ↗
If an AI can replace me on the mental aspects of work, and robotics are on their way to replacing humanity on the physical aspects of work... then what's left? When there was agrarian societies, there were writers, priests, bankers, merchants, and laborers before and after - I really don't think things were that unclear even at the time. Now that we have machines that are close to exceeding humans in every way, what good are humans?
bilater · · focus · HN ↗
At the same time, I agree with the original comment as well. I don't think this necessarily leads to some doomsday scenario. Whatever happens it'll likely be better for us and imo we will merge with the AIs at some point, so it won't be a question of us vs them.
ogogmad · · focus · HN ↗
bilater · · focus · HN ↗
gWPVhyxPHqvk · · focus · HN ↗
tim333 · · focus · HN ↗
gWPVhyxPHqvk · · focus · HN ↗
I actually see it more likely the opposite - a subset of humans will retreat into a (vastly smaller) human-first world, and AI will be of a separate world concerned with material/scientific/research concerns and appeases the masses who are still around.
We can already see this today in the form of upper class parents (like myself) who vastly restrict their kids screen time and ban social media. We may see AI-free societies that form, but supported by AI. The jobs and roles people play within that society may deal with services that really need to be done by a human, being a buffer from AI to the rest of this society, or just straight up fake.
I personally think this is close to a doomsday scenario - but it doesn't end with a big bang, but more of slow quiet death.
tim333 · · focus · HN ↗
pizzly · · focus · HN ↗
The subset of people who will want to merge will be people who will not be content with having no control over their future. No ability to create new inventions, new discoveries, new ways of working. They will not be content with just having everything provided to them, playing the social games that will remain and think their environment is a prison.
GPerson · · focus · HN ↗