> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
A metaphor I'm constantly drawn to is the transition from agrarian to urban societies following the Industrial Revolution. Somebody who somehow saw the Industrial Revolution coming from the perspective of somebody living in an agrarian society might have envisioned it leading to 'super farms.' And it did.
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
> I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing).
If an AI can replace me on the mental aspects of work, and robotics are on their way to replacing humanity on the physical aspects of work... then what's left? When there was agrarian societies, there were writers, priests, bankers, merchants, and laborers before and after - I really don't think things were that unclear even at the time. Now that we have machines that are close to exceeding humans in every way, what good are humans?
The question is how long before machines can exceed humans in every way and how long will it take before legal and cultural restrictions allow this to happen. It still could be a long time away and jobs will probably increase in the short/medium term as AI unlocks more what humans can do at a rate faster than AI can automate away.
But lets consider what that end state could look like when machines exceed humans in everyway, what good are humans?
One possibility which I don't particularly enjoy is humans will be good for status games, art, creativity, story telling, IRL experiences, everything that involves human to human interaction and connection with other humans. If AI can supply everything in abundance than like rare cards there is only so many humans in the planet at any particular time then human's value increases. I really do believe connections with other humans will have a premium. As a nerd who is borderline anti-social, non artistic, creative, etc this does not appeal at all to me and properly to many others who read HN as well. For the majority of other people with some adaption time I think they will be fine. They were fine adapting to hunters and gathers to agricultural society, they were fine adapting to agricultural society to industrial society, they were find adapting from industrial society to information society and they will will be fine adapting from information society to the social/creative economy. Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
If a computer wins a chess game it doesn't enjoy winning nearly as much as a human would enjoy losing from another human. And that's just chess. Now extrapolate to everything else.
> Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
How do you propose this brave new world of yours will work economically?
I'm not yet even in my 50s, yet when I was young digital work mostly did not exist. Somebody who claimed the future would have people sitting in front of screens typing away being a huge part of the economy would have seemed a bit kooky. To say nothing of there being trillion dollar companies whose primary 'product' is selling ad space on the fledgling internet, an internet that it took forever for 'normies' to even realize existed.
It seems almost a forgone conclusion at this point that LLMs are going to wreck most cognitive work, particularly in the digital domain. And a world where e.g. software creation can be almost entirely automated for very little cost doesn't mean software companies become absurdly wealthy by being able to automate away all their labor - it means it's a world where software's value approaches $0.
And that would broadly apply to all things that LLMs can automate - which is going to be most of the problem-solving digital domain. So as the value of these things decreases, the value of other things will tend to increase. And, outside of entertainment, it seems highly probable that we're looking at a return to the 'real world' being where the value of problem solving will be found.
I think that's quite predictable. You're speaking of a scenario here where people did all the things society says you're supposed to do to end up living a very comfortable life, and the outcome is anything but comfortable. That's going to be quite difficult for people to accept, and so many simply will choose not to.
It may not even be a bad decision. It's not completely clear on what timeframe this will all play out, even when it does start playing out on a larger scale there will be some software jobs, and it's not most people have great alternatives so they'd be stressing themselves out over something that they probably can't do much of anything about.
pyridines · · focus · HN ↗
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
somenameforme · · focus · HN ↗
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
gWPVhyxPHqvk · · focus · HN ↗
If an AI can replace me on the mental aspects of work, and robotics are on their way to replacing humanity on the physical aspects of work... then what's left? When there was agrarian societies, there were writers, priests, bankers, merchants, and laborers before and after - I really don't think things were that unclear even at the time. Now that we have machines that are close to exceeding humans in every way, what good are humans?
pizzly · · focus · HN ↗
But lets consider what that end state could look like when machines exceed humans in everyway, what good are humans?
One possibility which I don't particularly enjoy is humans will be good for status games, art, creativity, story telling, IRL experiences, everything that involves human to human interaction and connection with other humans. If AI can supply everything in abundance than like rare cards there is only so many humans in the planet at any particular time then human's value increases. I really do believe connections with other humans will have a premium. As a nerd who is borderline anti-social, non artistic, creative, etc this does not appeal at all to me and properly to many others who read HN as well. For the majority of other people with some adaption time I think they will be fine. They were fine adapting to hunters and gathers to agricultural society, they were fine adapting to agricultural society to industrial society, they were find adapting from industrial society to information society and they will will be fine adapting from information society to the social/creative economy. Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
jacquesm · · focus · HN ↗
> Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
How do you propose this brave new world of yours will work economically?
somenameforme · · focus · HN ↗
It seems almost a forgone conclusion at this point that LLMs are going to wreck most cognitive work, particularly in the digital domain. And a world where e.g. software creation can be almost entirely automated for very little cost doesn't mean software companies become absurdly wealthy by being able to automate away all their labor - it means it's a world where software's value approaches $0.
And that would broadly apply to all things that LLMs can automate - which is going to be most of the problem-solving digital domain. So as the value of these things decreases, the value of other things will tend to increase. And, outside of entertainment, it seems highly probable that we're looking at a return to the 'real world' being where the value of problem solving will be found.
jacquesm · · focus · HN ↗
And yet, a good chunk of HN is totally in denial about this.
somenameforme · · focus · HN ↗
It may not even be a bad decision. It's not completely clear on what timeframe this will all play out, even when it does start playing out on a larger scale there will be some software jobs, and it's not most people have great alternatives so they'd be stressing themselves out over something that they probably can't do much of anything about.