I agree with this article and the idea of RSI has never made sense to me.
Intelligence requires being able to solve problems in the real world; things like code and math are just useful proxies.
The real world cannot give instantaneous feedback like a compiler or proof-checker can. Suppose a model wanted to develop a cure to cancer. Wouldn’t it take years to synthesize and go through clinical trials for every iteration, creating a massive bottleneck on how quickly it can improve?
And that’s something that can eventually be verified. What about things that are difficult or impossible to systematically verify (knowing how/where to look for new ideas, or even just “common sense”)? How would a self-improving model even know it’s going wrong?
The evidence provided in the article is also compelling. If even frontier models only have 80% success on short AI research tasks, what happens if the model doesn’t realize its mistake and builds off of the work it fucked up? Wouldn’t the error rate compound, making even small error rates hugely problematic? (And these error rates OpenAI are showing are not small.)
It seems to be a given at this point that any clearly defined and quickly verifiable task, AI can do. But “make yourself smarter” is not such a task, and I don’t see a world where letting agents loop on that forever would lead to an explosion (other than in cost).
I’d love to hear what others think about this, since I feel like I must be missing something if all the top researchers seem to strongly believe in RSI as a possibility.
demibabs · · focus · HN ↗
Intelligence requires being able to solve problems in the real world; things like code and math are just useful proxies.
The real world cannot give instantaneous feedback like a compiler or proof-checker can. Suppose a model wanted to develop a cure to cancer. Wouldn’t it take years to synthesize and go through clinical trials for every iteration, creating a massive bottleneck on how quickly it can improve?
And that’s something that can eventually be verified. What about things that are difficult or impossible to systematically verify (knowing how/where to look for new ideas, or even just “common sense”)? How would a self-improving model even know it’s going wrong?
The evidence provided in the article is also compelling. If even frontier models only have 80% success on short AI research tasks, what happens if the model doesn’t realize its mistake and builds off of the work it fucked up? Wouldn’t the error rate compound, making even small error rates hugely problematic? (And these error rates OpenAI are showing are not small.)
It seems to be a given at this point that any clearly defined and quickly verifiable task, AI can do. But “make yourself smarter” is not such a task, and I don’t see a world where letting agents loop on that forever would lead to an explosion (other than in cost).
I’d love to hear what others think about this, since I feel like I must be missing something if all the top researchers seem to strongly believe in RSI as a possibility.