> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
A metaphor I'm constantly drawn to is the transition from agrarian to urban societies following the Industrial Revolution. Somebody who somehow saw the Industrial Revolution coming from the perspective of somebody living in an agrarian society might have envisioned it leading to 'super farms.' And it did.
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
> I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing).
If an AI can replace me on the mental aspects of work, and robotics are on their way to replacing humanity on the physical aspects of work... then what's left? When there was agrarian societies, there were writers, priests, bankers, merchants, and laborers before and after - I really don't think things were that unclear even at the time. Now that we have machines that are close to exceeding humans in every way, what good are humans?
I've struggled with this question as well, and this is why I reject the premise of the historical pattern of technology enabling us to "move up" to something else.
At the same time, I agree with the original comment as well. I don't think this necessarily leads to some doomsday scenario. Whatever happens it'll likely be better for us and imo we will merge with the AIs at some point, so it won't be a question of us vs them.
Only if you have a zero-sum game mindset. I suppose it does make sense that AI-native intelligences are just smarter than hybrid AI-human entities (biologically enhanced or uploads) but that doesn't mean there needs to be conflict between the two. The universe is huge. :)
I personally don't think there will be a conflict, but more just an extinguishing of all things human, outside of niche forms of entertainment and care. It won't be machines that cause the conflict, but the frustrated and bored masses fighting over the scraps left behind by the machines and ultra-wealthy.
I actually see it more likely the opposite - a subset of humans will retreat into a (vastly smaller) human-first world, and AI will be of a separate world concerned with material/scientific/research concerns and appeases the masses who are still around.
We can already see this today in the form of upper class parents (like myself) who vastly restrict their kids screen time and ban social media. We may see AI-free societies that form, but supported by AI. The jobs and roles people play within that society may deal with services that really need to be done by a human, being a buffer from AI to the rest of this society, or just straight up fake.
I personally think this is close to a doomsday scenario - but it doesn't end with a big bang, but more of slow quiet death.
I figure both will happen. Some people will want to live naturally, some to merge. I sometimes think with the data centers in space discussion that it'll end up with AIs in space and the Earth as something like a nature reserve.
I think many people will be quite content not merging with AI and having AI do everything for them. Perhaps only a small subset of people will want to merge with AI. They may not be allowed to though, if majority of people fine not merging then they may ban those who want to merge.
The subset of people who will want to merge will be people who will not be content with having no control over their future. No ability to create new inventions, new discoveries, new ways of working. They will not be content with just having everything provided to them, playing the social games that will remain and think their environment is a prison.
“Everything else”. Societies get to decide exactly how much is, can, or will be automated, what decisions must be left to humans and what role we will play.
My position is there is "nothing else" other than entertainment and elder/childcare. And even large portions of those can be automated. There is also zero chance that "society" is going to get to decide, either - we've already failed that test with social media and carbon dioxide.
There is a huge amount of pending work in health research that is nowadays ignored because smart people have better paying jobs available.
Once many of those jobs get automated, I bet there will be many more people working in health research, which hopefully should lead to better health outcomes for society as a whole.
This is true, and it's strange that people don't think much about the fact that approximately zero % of people on planet earth are actually trying to cure diseases.
So when our lives will become so much emptier because we are leaving all intellectual and physical work to machines, at least on the way there we can reallocate all our greatest minds to improve healthcare, so we can turn 200 in our paradise where we don’t need to work or think?
The question is how long before machines can exceed humans in every way and how long will it take before legal and cultural restrictions allow this to happen. It still could be a long time away and jobs will probably increase in the short/medium term as AI unlocks more what humans can do at a rate faster than AI can automate away.
But lets consider what that end state could look like when machines exceed humans in everyway, what good are humans?
One possibility which I don't particularly enjoy is humans will be good for status games, art, creativity, story telling, IRL experiences, everything that involves human to human interaction and connection with other humans. If AI can supply everything in abundance than like rare cards there is only so many humans in the planet at any particular time then human's value increases. I really do believe connections with other humans will have a premium. As a nerd who is borderline anti-social, non artistic, creative, etc this does not appeal at all to me and properly to many others who read HN as well. For the majority of other people with some adaption time I think they will be fine. They were fine adapting to hunters and gathers to agricultural society, they were fine adapting to agricultural society to industrial society, they were find adapting from industrial society to information society and they will will be fine adapting from information society to the social/creative economy. Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
If a computer wins a chess game it doesn't enjoy winning nearly as much as a human would enjoy losing from another human. And that's just chess. Now extrapolate to everything else.
> Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
How do you propose this brave new world of yours will work economically?
I'm not yet even in my 50s, yet when I was young digital work mostly did not exist. Somebody who claimed the future would have people sitting in front of screens typing away being a huge part of the economy would have seemed a bit kooky. To say nothing of there being trillion dollar companies whose primary 'product' is selling ad space on the fledgling internet, an internet that it took forever for 'normies' to even realize existed.
It seems almost a forgone conclusion at this point that LLMs are going to wreck most cognitive work, particularly in the digital domain. And a world where e.g. software creation can be almost entirely automated for very little cost doesn't mean software companies become absurdly wealthy by being able to automate away all their labor - it means it's a world where software's value approaches $0.
And that would broadly apply to all things that LLMs can automate - which is going to be most of the problem-solving digital domain. So as the value of these things decreases, the value of other things will tend to increase. And, outside of entertainment, it seems highly probable that we're looking at a return to the 'real world' being where the value of problem solving will be found.
I think that's quite predictable. You're speaking of a scenario here where people did all the things society says you're supposed to do to end up living a very comfortable life, and the outcome is anything but comfortable. That's going to be quite difficult for people to accept, and so many simply will choose not to.
It may not even be a bad decision. It's not completely clear on what timeframe this will all play out, even when it does start playing out on a larger scale there will be some software jobs, and it's not most people have great alternatives so they'd be stressing themselves out over something that they probably can't do much of anything about.
> If an AI can replace me on the mental aspects of work, and robotics are on their way to replacing humanity on the physical aspects of work... then what's left?
At the risk of sounding new age, my answer to this is "emotional work".
What exactly that would mean in the equivalent of the post-agrarian society that LLMs might bring, I cannot know.
You can't ignore how economies shift in response to the surplus of things. When software or other such things can be produced trivially, its cost and value is going to approach as close to $0 as it possibly can. By contrast the value of other things will go up.
So for instance in the past it used to be that literacy was a high value skill. Transcription and other such services were highly valued. Now that literacy is ubiquitous, such skills have basically no value in and of themselves. But instead, we've built a mountain of stuff on top of those skills and in domains which simply did not even exist before, and those things are now valued. But what we value now a days is no more permanent or inherent in its value than anything else that ultimately failed to stand the test of time.
I never thought that 'Terminator' was very realistic in that it was the AI that triggered the apocalypse. I always thought it was far more realistic that those that possessed the keys to the AI and control of atomic weapons decided to solve the greenhouse problem in a different way. And to set their robots up to clean up the mess so they could reboot the planet with a few thousand lucky ones. The other option was to find a 'B Ark' and some planet to ship the rest of us off to but that takes a lot of effort.
That was of course a complete fantasy. Now let's see, who is in charge of the nukes these days...
I like to think of AIs as the third revolution for us humans.
1st - Writing. This spread the power of memory to everyone. Socrates famously maligns writing as decreasing memory for the people. I'd say we have gotten along just fine. We still use memory, we still need it for ourselves, and using a tremendous amount of memory is still quite expensive.
2nd - Engines. This spread the power of muscles to everyone. No longer was it tied up in horses and mules and Goliaths. Even the most disease ridden pencil neck could trench and mine and build lavish houses now. The power to make pyramids is within the reach of a small company now. We still need muscles, of course, we still use them for little odd jobs, for health and well being, and using a lot of power is still quite expensive.
3rd - AIs. This is more speculative, of course. But I see AIs and LLMs as just like the other two revolutions. Intelligence (probably poorly defined by us these days) will be democratized and spread out. No longer will you have to study for decades or hire really expensive people to get it. You can just use some for relatively cheap. You'll still need intelligence of course, use it around the house and with friends and family, and using a lot of intelligence will likely still be quite expensive.
4th - Who knows! But I think that us humans have a lot of things that we do and say and make and see that aren't really well understood right now. Empathy, love, morality, care, soul, sin, mind, war, justice, honor, god, family, emotion, education, desire, courage, citizenship, chance, beauty, art, divinity, there is a lot!
Games, art, athletics, food, fashion, sex, social interaction. People still play chess even though computers surpassed us a while ago. We still climb mountains even though helicopters exist. Check out Ian Banks' Culture series for an interesting take on post-scarcity life with superintelligences.
I'm aghast. I guess I'd like to pose a question to you - what is the purpose of your life? I found my purpose years ago, but it took me a while to figure it out. Here it is: the purpose of my life is to give love to others. Through words, through actions. Notice that it's wholly driven from an emotional underpinning, and not related to success, or fame, or money, or what little ephemeral things we humans physically build in our lives.
From that lens, a human driven by this purpose can never have LLMs/AI take any purpose from us. Raising children, helping the needy, and caring for our spouse and parents suddenly becomes divine work. It's work that the world needs. The world does not need more software or more buildings. There is currently a lack of human connection and compassion and care, perhaps bigger than there's been in the history of our world. And that's why I think this purpose should be paramount, especially today in the age we are living in.
Take raising children. If you love your children, if you truly care for them and raise them with compassion and understanding, they will raise their children that way as well. And they will be good human beings. They will be kind. And your love can be a sort of benevolent virus that will ring through the centuries, through the generations ahead.
pyridines · · focus · HN ↗
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
somenameforme · · focus · HN ↗
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
gWPVhyxPHqvk · · focus · HN ↗
If an AI can replace me on the mental aspects of work, and robotics are on their way to replacing humanity on the physical aspects of work... then what's left? When there was agrarian societies, there were writers, priests, bankers, merchants, and laborers before and after - I really don't think things were that unclear even at the time. Now that we have machines that are close to exceeding humans in every way, what good are humans?
bilater · · focus · HN ↗
At the same time, I agree with the original comment as well. I don't think this necessarily leads to some doomsday scenario. Whatever happens it'll likely be better for us and imo we will merge with the AIs at some point, so it won't be a question of us vs them.
ogogmad · · focus · HN ↗
bilater · · focus · HN ↗
gWPVhyxPHqvk · · focus · HN ↗
tim333 · · focus · HN ↗
gWPVhyxPHqvk · · focus · HN ↗
I actually see it more likely the opposite - a subset of humans will retreat into a (vastly smaller) human-first world, and AI will be of a separate world concerned with material/scientific/research concerns and appeases the masses who are still around.
We can already see this today in the form of upper class parents (like myself) who vastly restrict their kids screen time and ban social media. We may see AI-free societies that form, but supported by AI. The jobs and roles people play within that society may deal with services that really need to be done by a human, being a buffer from AI to the rest of this society, or just straight up fake.
I personally think this is close to a doomsday scenario - but it doesn't end with a big bang, but more of slow quiet death.
tim333 · · focus · HN ↗
pizzly · · focus · HN ↗
The subset of people who will want to merge will be people who will not be content with having no control over their future. No ability to create new inventions, new discoveries, new ways of working. They will not be content with just having everything provided to them, playing the social games that will remain and think their environment is a prison.
GPerson · · focus · HN ↗
therealpygon · · focus · HN ↗
gWPVhyxPHqvk · · focus · HN ↗
giovannibonetti · · focus · HN ↗
Once many of those jobs get automated, I bet there will be many more people working in health research, which hopefully should lead to better health outcomes for society as a whole.
danielmarkbruce · · focus · HN ↗
spockz · · focus · HN ↗
pizzly · · focus · HN ↗
But lets consider what that end state could look like when machines exceed humans in everyway, what good are humans?
One possibility which I don't particularly enjoy is humans will be good for status games, art, creativity, story telling, IRL experiences, everything that involves human to human interaction and connection with other humans. If AI can supply everything in abundance than like rare cards there is only so many humans in the planet at any particular time then human's value increases. I really do believe connections with other humans will have a premium. As a nerd who is borderline anti-social, non artistic, creative, etc this does not appeal at all to me and properly to many others who read HN as well. For the majority of other people with some adaption time I think they will be fine. They were fine adapting to hunters and gathers to agricultural society, they were fine adapting to agricultural society to industrial society, they were find adapting from industrial society to information society and they will will be fine adapting from information society to the social/creative economy. Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
jacquesm · · focus · HN ↗
> Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
How do you propose this brave new world of yours will work economically?
somenameforme · · focus · HN ↗
It seems almost a forgone conclusion at this point that LLMs are going to wreck most cognitive work, particularly in the digital domain. And a world where e.g. software creation can be almost entirely automated for very little cost doesn't mean software companies become absurdly wealthy by being able to automate away all their labor - it means it's a world where software's value approaches $0.
And that would broadly apply to all things that LLMs can automate - which is going to be most of the problem-solving digital domain. So as the value of these things decreases, the value of other things will tend to increase. And, outside of entertainment, it seems highly probable that we're looking at a return to the 'real world' being where the value of problem solving will be found.
jacquesm · · focus · HN ↗
And yet, a good chunk of HN is totally in denial about this.
somenameforme · · focus · HN ↗
It may not even be a bad decision. It's not completely clear on what timeframe this will all play out, even when it does start playing out on a larger scale there will be some software jobs, and it's not most people have great alternatives so they'd be stressing themselves out over something that they probably can't do much of anything about.
pelario · · focus · HN ↗
At the risk of sounding new age, my answer to this is "emotional work".
What exactly that would mean in the equivalent of the post-agrarian society that LLMs might bring, I cannot know.
jacquesm · · focus · HN ↗
somenameforme · · focus · HN ↗
So for instance in the past it used to be that literacy was a high value skill. Transcription and other such services were highly valued. Now that literacy is ubiquitous, such skills have basically no value in and of themselves. But instead, we've built a mountain of stuff on top of those skills and in domains which simply did not even exist before, and those things are now valued. But what we value now a days is no more permanent or inherent in its value than anything else that ultimately failed to stand the test of time.
jacquesm · · focus · HN ↗
That was of course a complete fantasy. Now let's see, who is in charge of the nukes these days...
Terr_ · · focus · HN ↗
Not sure how much more I can say without spoiling it for people who haven't played.
Balgair · · focus · HN ↗
1st - Writing. This spread the power of memory to everyone. Socrates famously maligns writing as decreasing memory for the people. I'd say we have gotten along just fine. We still use memory, we still need it for ourselves, and using a tremendous amount of memory is still quite expensive.
2nd - Engines. This spread the power of muscles to everyone. No longer was it tied up in horses and mules and Goliaths. Even the most disease ridden pencil neck could trench and mine and build lavish houses now. The power to make pyramids is within the reach of a small company now. We still need muscles, of course, we still use them for little odd jobs, for health and well being, and using a lot of power is still quite expensive.
3rd - AIs. This is more speculative, of course. But I see AIs and LLMs as just like the other two revolutions. Intelligence (probably poorly defined by us these days) will be democratized and spread out. No longer will you have to study for decades or hire really expensive people to get it. You can just use some for relatively cheap. You'll still need intelligence of course, use it around the house and with friends and family, and using a lot of intelligence will likely still be quite expensive.
4th - Who knows! But I think that us humans have a lot of things that we do and say and make and see that aren't really well understood right now. Empathy, love, morality, care, soul, sin, mind, war, justice, honor, god, family, emotion, education, desire, courage, citizenship, chance, beauty, art, divinity, there is a lot!
rphv · · focus · HN ↗
Games, art, athletics, food, fashion, sex, social interaction. People still play chess even though computers surpassed us a while ago. We still climb mountains even though helicopters exist. Check out Ian Banks' Culture series for an interesting take on post-scarcity life with superintelligences.
quadragenarian · · focus · HN ↗
From that lens, a human driven by this purpose can never have LLMs/AI take any purpose from us. Raising children, helping the needy, and caring for our spouse and parents suddenly becomes divine work. It's work that the world needs. The world does not need more software or more buildings. There is currently a lack of human connection and compassion and care, perhaps bigger than there's been in the history of our world. And that's why I think this purpose should be paramount, especially today in the age we are living in.
Take raising children. If you love your children, if you truly care for them and raise them with compassion and understanding, they will raise their children that way as well. And they will be good human beings. They will be kind. And your love can be a sort of benevolent virus that will ring through the centuries, through the generations ahead.