> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
A metaphor I'm constantly drawn to is the transition from agrarian to urban societies following the Industrial Revolution. Somebody who somehow saw the Industrial Revolution coming from the perspective of somebody living in an agrarian society might have envisioned it leading to 'super farms.' And it did.
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
Or will it be the cumulative total of various advances?
I've equated Claude Code, or Codex, to the looms that made fine fabric more affordable during the Industrial Revolution; life-changing, but not society-changing. Neither the steam engine nor the automobile.
Perhaps I've answered my own question in that it's the LLM technology itself that equates to the steam engine, and it will power superfarm analogs that have yet to emerge. I'm still curious what you think they will be.
From my perspective, the superfarms are the things that we get when we try to predict the future by just pushing the present forward, but without thinking about newly emergent industries, societal shifts, and so on. So in other words, just seeing the same stuff as the present, but bigger, better, and more efficient.
So an obvious example there would be software. It's certainly true (if we assume LLMs reach their 'potential') that software will be able to reach new heights, and with a far smaller headcount driving the development. So some people see this as economically catastrophic for software developers, or an economic boon for certain large software companies.
But I think that when software can be built at the drop of a hat, software itself will no eventually no longer really matter in economic terms. Yet things you can build on top of it will matter more than ever. Those things are difficult to see from here, but I expect they will be the giants of the economy of tomorrow.
But what have we built on top of software so far? My first thought is that we have made new communities. But most of them are of much poorer quality than what we had in person.
I hope we start to rebuild in person connections again with this technology.
Well it has certainly transformed bureaucratic operations. And I don't mean that in a bad way; superior bureaucracy and logistics was a key to the British Empire.
Software allows us to push computation (intelligence) into our environment.
Telecommunications did that, not really software. Twitter started out as an SMS compatible service remember.
I do think there's something there though: I've spent the last 2 days building an app I've always wanted to build for myself with my computer in the corner running Claude and Claude Remote. Prototypes land on my phone and I don't even look at it for more then a few minutes before doing something else.
It's software, actually useful software, which doesn't take 100s of hours to build.
So I didn't even really spend two days on it: I mostly didn't look at all. I'll spend more time setting the result up on my home server.
I agree we can't predict, and I explain it this way - it is like a football game, where the ball will be 10 seconds later depends on what every player is doing. It is the same with AI, what we will do with it depends on what others will do with it, the reason we can't foresee it.
Many goods/products have seen a transition from scarcity to abundance. The analogy of fiber looms was already made. Humans used to spend a huge amount of time producing clothe. It was even used as currency in many cultures. No cheap clothing is so abundant we don't even think about where it comes from.
Salt is another one. Used to be payment for Roman soldier, now you can just take it from a McDonalds if you want.
I think software will be particularly unique in the abundance world because the exact same thing that made it so profitable is what will make it so 'worthless' - it has practically no direct real world cost. Cloth is abundant, yet you still pay a significant chunk of change for a shirt or pair of pants. But that's because there's still a significant real world cost there. Even if the production and processing can be heavily automated, there's still a very long process to go from cotton to pants at a store.
And then there's brand marketing and so on, which can drive prices to absurdly high levels. But when it comes to problem solving software. You just want the cheapest possible solution that works. And if we reach a level where you can even spin that solution yourself with a potentially poorly phrased, inconsistent, or contradictory spec (of the sort that developers already have to work with), then software as an industry is basically done. Not done as in dead, but done as is in there's just nowhere to go from here. Relatively few industries have any sort of comparable 'finish line', let alone the chances of us viably hitting it.
pyridines · · focus · HN ↗
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
somenameforme · · focus · HN ↗
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
curious303 · · focus · HN ↗
Or will it be the cumulative total of various advances?
I've equated Claude Code, or Codex, to the looms that made fine fabric more affordable during the Industrial Revolution; life-changing, but not society-changing. Neither the steam engine nor the automobile.
Perhaps I've answered my own question in that it's the LLM technology itself that equates to the steam engine, and it will power superfarm analogs that have yet to emerge. I'm still curious what you think they will be.
somenameforme · · focus · HN ↗
So an obvious example there would be software. It's certainly true (if we assume LLMs reach their 'potential') that software will be able to reach new heights, and with a far smaller headcount driving the development. So some people see this as economically catastrophic for software developers, or an economic boon for certain large software companies.
But I think that when software can be built at the drop of a hat, software itself will no eventually no longer really matter in economic terms. Yet things you can build on top of it will matter more than ever. Those things are difficult to see from here, but I expect they will be the giants of the economy of tomorrow.
treespace8 · · focus · HN ↗
I hope we start to rebuild in person connections again with this technology.
anon84873628 · · focus · HN ↗
Software allows us to push computation (intelligence) into our environment.
XorNot · · focus · HN ↗
I do think there's something there though: I've spent the last 2 days building an app I've always wanted to build for myself with my computer in the corner running Claude and Claude Remote. Prototypes land on my phone and I don't even look at it for more then a few minutes before doing something else.
It's software, actually useful software, which doesn't take 100s of hours to build.
So I didn't even really spend two days on it: I mostly didn't look at all. I'll spend more time setting the result up on my home server.
visarga · · focus · HN ↗
anon84873628 · · focus · HN ↗
Salt is another one. Used to be payment for Roman soldier, now you can just take it from a McDonalds if you want.
Seems like we can have an abundance of software.
somenameforme · · focus · HN ↗
And then there's brand marketing and so on, which can drive prices to absurdly high levels. But when it comes to problem solving software. You just want the cheapest possible solution that works. And if we reach a level where you can even spin that solution yourself with a potentially poorly phrased, inconsistent, or contradictory spec (of the sort that developers already have to work with), then software as an industry is basically done. Not done as in dead, but done as is in there's just nowhere to go from here. Relatively few industries have any sort of comparable 'finish line', let alone the chances of us viably hitting it.