Autonomous recursive self-improvement needs to happen. It doesn't have to happen tomorrow, or in 10 years even, but it has to happen sometime in the near future, or we're kinda screwed.
Why say this? Well, of the plausible futures from here, the worst one is where AI plateus just good enough to make us all useless (or just destroy the pipeline of a critical mass of capable humans for assisted RSI), but at a point dumb enough that we don't get human-superior AI. This might happen or not, but there's no way to predict when something that doesn't exist will, so it's one of those horrors that can keep you up at night. Some of those skeptical about RSI have likened it to the Great Filter.
The probly is believing our problems are just calculus.
They are not. Our problems are akin to cellular automata, and just the same. We can project a few steps forward but we cannot say this configuration will evolve to that configuration.
Eugenics is similar; the belief that we can measure physical chatacteristics and selectbased on the same for some desirable future configuration is just a self delusion.
Orboids in a swarm the whole does not direct the sum.
These are all basics that humans struggle against. AI will just head in adirection, and itll be as meaningless as an automata swarm. Youll recognize payterns but they wont make you sure of the next step.
That seems like an OK future to me? The AI-addled can have their fun, the rest of us can eventually step in to pick up the pieces. I know it can seem bleak at times but the knowledge hasn't evaporated, it's just that (IMO) the less-AI-enthusiastic are stepping back a bit to let the energy spend itself. And "LLMs can't become superintelligence" doesn't imply "self-improving machine intelligence is impossible", for that matter.
What happens if the momentum of us AI-addled outlives the tribal knowledge on the sidelines?
I genuinely fear your scenario -- "AI was good enough for long enough to let the artisans with the knowledge die off" -- now it's idiocracy. :)
Although is this different than people who know how to build and operate a steam engine? I am not certain.
bm3719 · · focus · HN ↗
Why say this? Well, of the plausible futures from here, the worst one is where AI plateus just good enough to make us all useless (or just destroy the pipeline of a critical mass of capable humans for assisted RSI), but at a point dumb enough that we don't get human-superior AI. This might happen or not, but there's no way to predict when something that doesn't exist will, so it's one of those horrors that can keep you up at night. Some of those skeptical about RSI have likened it to the Great Filter.
cyanydeez · · focus · HN ↗
They are not. Our problems are akin to cellular automata, and just the same. We can project a few steps forward but we cannot say this configuration will evolve to that configuration.
Eugenics is similar; the belief that we can measure physical chatacteristics and selectbased on the same for some desirable future configuration is just a self delusion.
Orboids in a swarm the whole does not direct the sum.
These are all basics that humans struggle against. AI will just head in adirection, and itll be as meaningless as an automata swarm. Youll recognize payterns but they wont make you sure of the next step.
_dwt · · focus · HN ↗
schmookeeg · · focus · HN ↗
I genuinely fear your scenario -- "AI was good enough for long enough to let the artisans with the knowledge die off" -- now it's idiocracy. :)
Although is this different than people who know how to build and operate a steam engine? I am not certain.