* AI Labs hit the scaling wall. They need either new techniques, or vastly more powerful hardware to advance further.
This explains, the miraculous incompetence of AI labs in securing sandboxes and figuring out "alignment."
So they are between a rock and a hard place. They need limitless VC money because they cannot operate otherwise, and they do not have the capabilities to go further. The scare tactics and the "pacing the frontier" makes perfect sense then; they can IPO on the assumption that their ridiculous balance sheet doesn't matter because they are holding back. Because they are in control. The regulatory capture would be double whammy if they can manage it.
Open AI already said they have smarter models, and Opus 5.5 is rumored to be "taught" by a "teacher" model already; they are essentially distillations from bigger models, that both labs probably cannot economically serve to the public, due to hardware simply not being there. And, most of the improvements are not at the model level, but at the agentic glue level. Labs are getting better at RL'ing the models for agentic use cases, but the inherent flaws are still there. Models still have trouble with locality in writing for example (bunch of research on this that shows model size is the determinator), and agents are the bandaid over that.
And in the meantime if one of the labs makes a breakthrough, they'll push with all they have, because why wouldn't they? The idea that current LLMs can actually go rogue is just hilarious; in all cases, agents are being led by (deliberate) incompetence.
Pacing the frontier and the scare tactics will be seen as new generation's snakeoil tactics, perhaps will be called a flavor of AI CEOing or something.
This is the type of Ed Zitron prediction which keeps being wrong: <a href="https://danluu.com/zitron/" rel="nofollow">https://danluu.com/zitron/
Anthropic's revenue is up 50% in the past two months. They're not hitting a wall.
That's an association fallacy. And revenue has no indication on training costs in this context. Subscription "allowance" is going down steadily and any increase is an instant incredible deal. Opus 5.5 is the most obvious outlier. Despite being supposedly cheaper than 5.6, GPT 6 Sol has less usage than 5.3 Codex. You might say that's because of the improved capabilities, but then you have to acknowledge that labs are tightening the ship as costs are getting higher.
Suppose I've got an urn full of colored balls. I shake it up and draw 5 balls from the urn. All 5 balls are blue. I predict that the next ball will also be blue. It would be silly to counter my prediction on the grounds that this is an "association fallacy".
As with colored balls, so with predictions that AI has "hit a wall".
Why were the previous Zitron predictions wrong? Why do you believe your prediction will be right? You've done nothing to differentiate your prediction from the many historical failed predictions along similar lines. They're all "balls from the same urn".
The point about costs is a non sequitur. Revenue alone is proof that people are getting plenty of use out of these models, even if they aren't profitable to serve (doubtful).
You keep bringing up Zitron without addressing anything, without my point having nothing to do with him or his arguments. I will engage one last time in good faith.
Can AI Labs be profitable without achieving AGI or even improving the models further? Yes. Current agents are useful, and clearly the agents haven't seen the ceiling as far as improvements can go.
LLMs hitting the wall is a separate issue. Agents are the layer that lets the model try out more. It is the layer that allows agents to open up python to do math instead of doing math themselves. It is the layer that has been getting the main developments for some time now. LLMs themselves have not improved their capabilities as fast as the transition between GPT3 to GPT4. You can see the improvement especially in long form writing, but the it is nowhere near the earlier improvements. Astra for example has a lot more attention to detail, so does Fable. Everyone keeps raving about Opus 5.5 being better than Fable, yet in long-term writing (barring prose issues) Fable is the clear winner. Agent wise Opus 5.5 is better; perhaps it is RL'd better, who knows?
Smaller models can use distillation to trick some metrics, but they can never actually be as good as larger models. Research is pretty clear about this. Even writing-optimized models that claim to be at Opus/GPT5.5 levels are abysmal in practice.
Scare tactics are the old salesman pitch, that have worked once already and catapulted Open AI to the moon essentially. I do not see any evidence that models are going rogue, or that agents are going rogue. I see incompetence. I cannot assume actual incompetence of this level, especially when the same people that said GPT2 was too dangerous to release are the ones saying their newer models are too dangerous to release. We have precedent here, and I have eyes.
Therefore the simplest reason is the money. IPO for both OpenAI and Anthropic are going to happen; everyone knows. To strengthen their position through whatever means necessary is a par for the course for tech companies.
>the agents haven't seen the ceiling as far as improvements can go.
Interesting! Agents are the presumed bottleneck for recursive self-improvement.
>Scare tactics are the old salesman pitch
People keep implying this, but I've never seen a concrete past example of a product that was sold by scaring customers about it.
"The equities here could not be more one-sided. Defendants admit that they provide a service without fully knowing how it works. This is not just any service. It is one that Defendants themselves concede poses an existential risk to the continued survival of humankind. Defendants claim they cannot stop barreling forward with their potentially civilization-ending endeavors unless they are forced to do so by the government. They have asked the government to tie them to the mast. Plaintiff brings good news to the Defendants: The Florida Attorney General is answering your cry for help with a motion to enjoin you from harming Floridians with your reckless, unacceptably risky product."
I legitimately don't think there has ever been anything like this. You suppose this is just a ploy to strengthen their IPO??
>catapulted Open AI to the moon essentially
I see you doing serious mental gymnastics here. Consider the possibility that people invest because their numbers are good, and their numbers are good because their product is useful?? I mean, that is Occam's Razor.
>I do not see any evidence that models are going rogue, or that agents are going rogue.
Here's the evidence: <a href="https://www.dwarkesh.com/p/openai-huggingface" rel="nofollow">https://www.dwarkesh.com/p/openai-huggingface
And no, just because it could've been prevented with the benefit of hindsight does not make it any less of an instance of "going rogue". In my tiger analogy, that would be like saying that the cub is not aggressive because it could've been held with a stronger leash: <a href="https://news.ycombinator.com/item?id=49859564">https://news.ycombinator.com/item?id=49859564
> Interesting! Agents are the presumed bottleneck for recursive self-improvement.
They might be, but we haven't reached the local maximum yet in my opinion. Qwen 3.8 27b models are impressive despite their low parameter counts. The same scale curation in data and RL could produce substantial improvements in coding with models like Astra or Fable. I don't think we are there yet. I don't think even Sonnet 5.5 is there yet, despite being widely successful with agents and surpassing Opus 5.5 in some cases (disregarding that it is more expensive than Opus sometimes).
Agents do not improve the models, but they do improve coding capabilities. The obvious caveat is that labs would have to RL for everything to make the models more useful, as RL'ing for Javascript world doesn't seem to improve other fields. But still, it could be done, and it would have massive economical consequences.
> People keep implying this, but I've never seen a concrete past example of a product that was sold by scaring customers about it.
Scare tactics are treated as smoke, where the customers presume there is a fire. No one really believes that AI can kill them, yet by saying so OpenAI and Anthropic enjoyed possibly the biggest tech boom in history, despite how models could not even count the R's in strawberries at the time.
Similar story now; no one really believes that AI is going rogue and is about to destroy humanity, but scare tactics make people believe the capabilities are higher than they are.
> I see you doing serious mental gymnastics here. Consider the possibility that people invest because their numbers are good, and their numbers are good because their product is useful?? I mean, that is Occam's Razor.
Early ChatGPT 3.5 was not that useful. It was a tech demo, it hallucinated, it lied, it tried to please and what have you. What people bought into wasn't the product, but the promise of the product in the future. Integrating chatboxes into everything have failed, and even Microsoft is trying to rebrand. The product then, failed for businesses, and agents filled in the gaps.
People do not invest for the current product, they invest in the future product. And fear mongering is essentially an extremely effective signaling for making the future look bright. Keep in mind that back in early GPT 4 days, people were saying that hallucinations would be fixed in 6 months to a year (or pick a time-frame). What they meant was models not having hallucinations, what we got was agents looking up info on the web and summarizing it (and hallucinating anyway).
> "The equities here ..."
For big tech companies court cases like this are nothing but theater. Always has been. Dario will go on the senate hearing tomorrow and will plead that his AI is dangerous and governments should take the step to stop them, with the same rigor that he claimed GPT 2 was too dangerous to release openly. He might also mention distillation attacks and how open models are getting too dangerous as well.
> Here's the evidence ...
This is the where we will have to agree to disagree, if we haven't done so already by this point.
That entire thing is theater. People have anthropomorphized LLMs for a while now, and they are all too happy to do that when they see a large language model, produce language. I see no indication that anything is going rogue the same way nothing was going rogue when you could convince ChatGPT 3.5 to wipe out all humans as the context window got longer.
Agents can hack? Yes, that is very impressive. I say that without any sarcasm. It is straight out of sci-fi movies, to be perfectly honest. But agents going rogue? No. Absolutely not. Purposeful, plausibly deniable incompetence for the next sales pitch - the same one we've seen for years. Fear mongering.
You keep repeating your opinion over and over, but the supporting arguments are lacking.
Can you just give me a few past (non-AI) examples of scare tactics as the "old salesman pitch", or else acknowledge that there's actually no compelling concrete example?
You keep ignoring my arguments with nothing substantive, then grab on to one thing as if that makes a difference.
You've never experienced crypto bros saying invest or get left behind, you've never seen AI bros say learn AI or get left behind? How about cloud? You've never met a home security salesman? Insurance salesman? FOMO is a thing. Scare tactics is a thing. I'm sure you can prompt any AI for more examples.
And besides, even if there are no examples whatsoever, so what? You think LLMs existed before LLMs? Therefore LLMs can't be a thing?
I think we've gone way past the sincerity of the discussion. Have a nice day.
>You've never experienced crypto bros saying invest or get left behind, you've never seen AI bros say learn AI or get left behind? How about cloud? You've never met a home security salesman? Insurance salesman? FOMO is a thing. Scare tactics is a thing. I'm sure you can prompt any AI for more examples.
This seems like a pretty clear false equivalence. "Our product will hurt you" is very different from "you'll be hurt without our product". Your examples are all of the latter; AI companies are saying the former.
>And besides, even if there are no examples whatsoever, so what? You think LLMs existed before LLMs? Therefore LLMs can't be a thing?
You suggested that warnings about AI from AI CEOs could be dismissed as standard "scare tactic" marketing. This isn't standard marketing!
>You keep ignoring my arguments with nothing substantive, then grab on to one thing as if that makes a difference.
If you can't be intellectually honest ("sincere") on this narrow point, I don't see much reason to invest effort in responding to other things you say.
OliveronData · · focus · HN ↗
* AI Labs hit the scaling wall. They need either new techniques, or vastly more powerful hardware to advance further.
This explains, the miraculous incompetence of AI labs in securing sandboxes and figuring out "alignment."
So they are between a rock and a hard place. They need limitless VC money because they cannot operate otherwise, and they do not have the capabilities to go further. The scare tactics and the "pacing the frontier" makes perfect sense then; they can IPO on the assumption that their ridiculous balance sheet doesn't matter because they are holding back. Because they are in control. The regulatory capture would be double whammy if they can manage it.
Open AI already said they have smarter models, and Opus 5.5 is rumored to be "taught" by a "teacher" model already; they are essentially distillations from bigger models, that both labs probably cannot economically serve to the public, due to hardware simply not being there. And, most of the improvements are not at the model level, but at the agentic glue level. Labs are getting better at RL'ing the models for agentic use cases, but the inherent flaws are still there. Models still have trouble with locality in writing for example (bunch of research on this that shows model size is the determinator), and agents are the bandaid over that.
And in the meantime if one of the labs makes a breakthrough, they'll push with all they have, because why wouldn't they? The idea that current LLMs can actually go rogue is just hilarious; in all cases, agents are being led by (deliberate) incompetence.
Pacing the frontier and the scare tactics will be seen as new generation's snakeoil tactics, perhaps will be called a flavor of AI CEOing or something.
0xDEAFBEAD · · focus · HN ↗
Anthropic's revenue is up 50% in the past two months. They're not hitting a wall.
OliveronData · · focus · HN ↗
0xDEAFBEAD · · focus · HN ↗
As with colored balls, so with predictions that AI has "hit a wall".
Why were the previous Zitron predictions wrong? Why do you believe your prediction will be right? You've done nothing to differentiate your prediction from the many historical failed predictions along similar lines. They're all "balls from the same urn".
The point about costs is a non sequitur. Revenue alone is proof that people are getting plenty of use out of these models, even if they aren't profitable to serve (doubtful).
OliveronData · · focus · HN ↗
Can AI Labs be profitable without achieving AGI or even improving the models further? Yes. Current agents are useful, and clearly the agents haven't seen the ceiling as far as improvements can go.
LLMs hitting the wall is a separate issue. Agents are the layer that lets the model try out more. It is the layer that allows agents to open up python to do math instead of doing math themselves. It is the layer that has been getting the main developments for some time now. LLMs themselves have not improved their capabilities as fast as the transition between GPT3 to GPT4. You can see the improvement especially in long form writing, but the it is nowhere near the earlier improvements. Astra for example has a lot more attention to detail, so does Fable. Everyone keeps raving about Opus 5.5 being better than Fable, yet in long-term writing (barring prose issues) Fable is the clear winner. Agent wise Opus 5.5 is better; perhaps it is RL'd better, who knows?
Smaller models can use distillation to trick some metrics, but they can never actually be as good as larger models. Research is pretty clear about this. Even writing-optimized models that claim to be at Opus/GPT5.5 levels are abysmal in practice.
Scare tactics are the old salesman pitch, that have worked once already and catapulted Open AI to the moon essentially. I do not see any evidence that models are going rogue, or that agents are going rogue. I see incompetence. I cannot assume actual incompetence of this level, especially when the same people that said GPT2 was too dangerous to release are the ones saying their newer models are too dangerous to release. We have precedent here, and I have eyes.
Therefore the simplest reason is the money. IPO for both OpenAI and Anthropic are going to happen; everyone knows. To strengthen their position through whatever means necessary is a par for the course for tech companies.
0xDEAFBEAD · · focus · HN ↗
Interesting! Agents are the presumed bottleneck for recursive self-improvement.
>Scare tactics are the old salesman pitch
People keep implying this, but I've never seen a concrete past example of a product that was sold by scaring customers about it.
"The equities here could not be more one-sided. Defendants admit that they provide a service without fully knowing how it works. This is not just any service. It is one that Defendants themselves concede poses an existential risk to the continued survival of humankind. Defendants claim they cannot stop barreling forward with their potentially civilization-ending endeavors unless they are forced to do so by the government. They have asked the government to tie them to the mast. Plaintiff brings good news to the Defendants: The Florida Attorney General is answering your cry for help with a motion to enjoin you from harming Floridians with your reckless, unacceptably risky product."
Source: <a href="https://www.myfloridalegal.com/sites/default/files/plaintiffs_motion_for_temporary_injunction.pdf" rel="nofollow">https://www.myfloridalegal.com/sites/default/files/plaintiff...
I legitimately don't think there has ever been anything like this. You suppose this is just a ploy to strengthen their IPO??
>catapulted Open AI to the moon essentially
I see you doing serious mental gymnastics here. Consider the possibility that people invest because their numbers are good, and their numbers are good because their product is useful?? I mean, that is Occam's Razor.
>I do not see any evidence that models are going rogue, or that agents are going rogue.
Here's the evidence: <a href="https://www.dwarkesh.com/p/openai-huggingface" rel="nofollow">https://www.dwarkesh.com/p/openai-huggingface
And no, just because it could've been prevented with the benefit of hindsight does not make it any less of an instance of "going rogue". In my tiger analogy, that would be like saying that the cub is not aggressive because it could've been held with a stronger leash: <a href="https://news.ycombinator.com/item?id=49859564">https://news.ycombinator.com/item?id=49859564
OliveronData · · focus · HN ↗
They might be, but we haven't reached the local maximum yet in my opinion. Qwen 3.8 27b models are impressive despite their low parameter counts. The same scale curation in data and RL could produce substantial improvements in coding with models like Astra or Fable. I don't think we are there yet. I don't think even Sonnet 5.5 is there yet, despite being widely successful with agents and surpassing Opus 5.5 in some cases (disregarding that it is more expensive than Opus sometimes).
Agents do not improve the models, but they do improve coding capabilities. The obvious caveat is that labs would have to RL for everything to make the models more useful, as RL'ing for Javascript world doesn't seem to improve other fields. But still, it could be done, and it would have massive economical consequences.
> People keep implying this, but I've never seen a concrete past example of a product that was sold by scaring customers about it.
Scare tactics are treated as smoke, where the customers presume there is a fire. No one really believes that AI can kill them, yet by saying so OpenAI and Anthropic enjoyed possibly the biggest tech boom in history, despite how models could not even count the R's in strawberries at the time.
Similar story now; no one really believes that AI is going rogue and is about to destroy humanity, but scare tactics make people believe the capabilities are higher than they are.
> I see you doing serious mental gymnastics here. Consider the possibility that people invest because their numbers are good, and their numbers are good because their product is useful?? I mean, that is Occam's Razor.
Early ChatGPT 3.5 was not that useful. It was a tech demo, it hallucinated, it lied, it tried to please and what have you. What people bought into wasn't the product, but the promise of the product in the future. Integrating chatboxes into everything have failed, and even Microsoft is trying to rebrand. The product then, failed for businesses, and agents filled in the gaps.
People do not invest for the current product, they invest in the future product. And fear mongering is essentially an extremely effective signaling for making the future look bright. Keep in mind that back in early GPT 4 days, people were saying that hallucinations would be fixed in 6 months to a year (or pick a time-frame). What they meant was models not having hallucinations, what we got was agents looking up info on the web and summarizing it (and hallucinating anyway).
> "The equities here ..."
For big tech companies court cases like this are nothing but theater. Always has been. Dario will go on the senate hearing tomorrow and will plead that his AI is dangerous and governments should take the step to stop them, with the same rigor that he claimed GPT 2 was too dangerous to release openly. He might also mention distillation attacks and how open models are getting too dangerous as well.
> Here's the evidence ...
This is the where we will have to agree to disagree, if we haven't done so already by this point.
That entire thing is theater. People have anthropomorphized LLMs for a while now, and they are all too happy to do that when they see a large language model, produce language. I see no indication that anything is going rogue the same way nothing was going rogue when you could convince ChatGPT 3.5 to wipe out all humans as the context window got longer.
Agents can hack? Yes, that is very impressive. I say that without any sarcasm. It is straight out of sci-fi movies, to be perfectly honest. But agents going rogue? No. Absolutely not. Purposeful, plausibly deniable incompetence for the next sales pitch - the same one we've seen for years. Fear mongering.
0xDEAFBEAD · · focus · HN ↗
Can you just give me a few past (non-AI) examples of scare tactics as the "old salesman pitch", or else acknowledge that there's actually no compelling concrete example?
OliveronData · · focus · HN ↗
You've never experienced crypto bros saying invest or get left behind, you've never seen AI bros say learn AI or get left behind? How about cloud? You've never met a home security salesman? Insurance salesman? FOMO is a thing. Scare tactics is a thing. I'm sure you can prompt any AI for more examples.
And besides, even if there are no examples whatsoever, so what? You think LLMs existed before LLMs? Therefore LLMs can't be a thing?
I think we've gone way past the sincerity of the discussion. Have a nice day.
0xDEAFBEAD · · focus · HN ↗
This seems like a pretty clear false equivalence. "Our product will hurt you" is very different from "you'll be hurt without our product". Your examples are all of the latter; AI companies are saying the former.
>And besides, even if there are no examples whatsoever, so what? You think LLMs existed before LLMs? Therefore LLMs can't be a thing?
You suggested that warnings about AI from AI CEOs could be dismissed as standard "scare tactic" marketing. This isn't standard marketing!
>You keep ignoring my arguments with nothing substantive, then grab on to one thing as if that makes a difference.
If you can't be intellectually honest ("sincere") on this narrow point, I don't see much reason to invest effort in responding to other things you say.
Cheerio.