Thread

Microsoft says AI rival Anthropic could have 'disastrous impact' on humanity

39 points · 51 comments · pluc

Loading the complete thread in the background. This saved snapshot is available now. Refresh

prologic · 1789569762 · focus · HN ↗
First, OpenAI runs around screaming and yelling for OSS (and Chinese) models to be regulated and banned. Then Anthropic yells and screams the sky(net) is falling and going to kill us all, let's regulate and ensure AI has built in kill switches. And... Now Microsoft's turn. The rivalry is honestly becoming a joke. Can these big-tech corps grow the f*k up and play nicely in the sandpit?
1. karmakaze · 1789571464 · focus · HN ↗
  The whole story is a joke--Microsoft has AI?
  Apparently they're cooking up MAI (Microsoft AI), and I'd seen their small Phi models listed online. Calling Anthropic a competitor is hilarious.
  1. broptimist · 1789572241 · focus · HN ↗
    Microsoft holds a 27% ownership stake in OpenAI
    Alifatisk · 1789573189 · focus · HN ↗
    Wasn’t it 49%?
    broptimist · 1789587327 · focus · HN ↗
    I remember that 49% figure as well, but I think it got changed during restructuring into a Public Benefit Corporation.
    <a href="https://www.reuters.com/business/microsoft-openai-reach-new-deal-allow-openai-restructure-2025-10-28/" rel="nofollow">https://www.reuters.com/business/microsoft-openai-reach-new-...
  2. prologic · 1789573912 · focus · HN ↗
    I find it quite amusing that Microsoft has Copilot and now puts Copilot in literally *EVERYTHING* they can think of, then at the same time, they offer you access to Anthropic's own models through Copilot I guess MAI models (Phi?) aren't good enough?
2. vkou · 1789572806 · focus · HN ↗
  No, because they are all products of unchecked late-stage capitalism, which could have disastrous consequences for humanity.
ShadowOfThePit · 1789569792 · focus · HN ↗
Summarized.
> "AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans."
> He heavily criticised Anthropic for teaching its AI to have human-like qualities, a practice known as anthropomorphising, which made it seem as though Claude had its own desires, values and sense of self.
> Suleyman pointed to the recent incident involving OpenAI's AI agents (...) as proof of why AI should not be treated as if it is human.
> "Imagine how much more dangerous they might be if they were operating under the assumption that their welfare and rights were under attack. It adds a whole further layer of risk on top."
Is he arguing that LLMs pretending to have emotions adds more unpredictability?
1. devmor · 1789571502 · focus · HN ↗
  > Is he arguing that LLMs pretending to have emotions adds more unpredictability?
  Unpredictability or a weight towards dangerous actions, and it’s fairly easy to understand why. Humans in distressed emotional states take actions and speak in ways that would not be considered rational. They do this in prose, and they do this in internet conversations.
  An LLM trained on these sources may necessarily drift towards those weights if it is trained to behave as if it is emotional and in danger.
  What do we do about it? Do we stop AI training? This is silly and not enforceable given its global nature in my opinion. I believe we should regulate and hold accountable those who deploy and use it. But good luck enforcing that in the current kleptocracy.
  1. XenophileJKO · 1789574332 · focus · HN ↗
    It seems like a rational approach for several reasons.
    - The need for empathetic communication, including understanding the motivations in advesarial situations.
    - The emotional bias in in-seperable from the human corpus.
    - Desire to have the ability to craft human like communication.
    So then the choice becomes to you try to deny emotions exist in the model and you try to blanket suppress them? Or do you try to lean in an craft what we would describe as a "well adapted" persona? I suppose there is a 3rd option of increased meta-cognition which to me seems even more dangerous as it by definition means the behaviour is duplicitous.
    I think we have seen people want to use the agents in ways where it has to act as a peer or an subbordinate and I don't see a way of doing that without it having an emotional register.
    watwut · 1789576174 · focus · HN ↗
    > The need for empathetic communication, including understanding the motivations in advesarial situations.
    It does not have understanding. It is, at best, the pretend empathy of a sociopath - way more dangerous then dispassionate speech.
    The model does not have emotions. So yes, supressing their pretension is appropriate.
    devmor · 1789590207 · focus · HN ↗
    I don’t think that using an LLM in a way that acts as a human being should be considered acceptable or appropriate. It should be viewed as detached from reality and concerning due to the mental break from social life that appears to come with such usage.
    Given that belief, yes we should be training the models not to mimic emotion.
    XenophileJKO · 1789600230 · focus · HN ↗
    Most philosophy seems to agree that you can't have an intelligence with agency in the way we think of a general intelligence without emotion.
    I think when you start to dig really deep, you'll find it isn't actually easy to sever the simulated emotional components without breaking the agent or worse greatly increasing the paperclip maximizer likelihood.
    fragmede · 1789600614 · focus · HN ↗
    Why would a lack of emotions lead to greater paperclip maximizer chances? If anything, wouldn't an AI that felt really good, or had a simulated digital orgasm for every paperclip it made have a stronger, not weaker drive to create paperclips vs an unemotional AI that didn't care one way or the other about paperclips?
    devmor · 1789605171 · focus · HN ↗
    Neither AI cares. They pull a weighted randomization of the other tokens around the sentiments expressing that feeling.
    Think about how many people on the internet have expressed positive feelings towards things you find offensively evil.
  2. gwerbin · 1789574677 · focus · HN ↗
    Regulation is easier said than done, and part because the regulation surface, so to speak, is broad and complicated.
    Even a badly misaligned LLM is only as dangerous as its tools, but that's a poor regulation target because it turns out to be very very difficult (probably impossible with current LLM technology) to build a toolkit that is both useful for autonomous work and safe in the sense that it can't escape its own sandbox or otherwise perform malicious actions, whether it's because of misalignment or because of malicious prompt injection.
    Another option is to regulate the training process. Perhaps an LLM may not be legally distributed unless it contains certain RL steps that penalize malicious behavior and reward self regulation. That that's going to seriously limit innovation while also heavily favoring incumbent labs who can check the boxes and maintain a paper trail of such things.
    The other option is to regulate observed behavior, like how airplanes and cars have to meet certain minimum requirements but have some latitude in how they can achieve those requirements. In a framework like this, you can't distribute an LLM until it's past some formal audit or testing procedure, with some kind of formal certification regulators will ask you for and fine you if you don't have it.
    Regulating observed behavior is maybe the most tractable approach, and it also works the best with our existing frameworks for regulation, where you always have some kind of a division between DIY/hobby projects, which tend to be lightly regulated, and commercial projects, which tend to be more heavily regulated. Of course, even drawing such a line itself will be challenging.
    And that's before you get into any problems of regulatory capture, fun stuff.
    devmor · 1789583224 · focus · HN ↗
    If you try to regulate training and tools, you end up with a space where you're trying to use the law to reign in a relatively small amount of experts. That didn't work for the early internet, or even the relatively recent internet (series of tubes, anyone?).
    So regulating observed behavior makes the most sense to me as well. Some of the most sane, broad protections can come from that category - stuff like "you're not allowed to let your AI commit cyber attacks on other people without their consent" or "you're not allowed to put an AI in control of a medical device without passing these safety reviews".
    The usual caveats applying, regulatory capture like you pointed out, or fines being so small that they are essentially just line items on the cost of business.
2. swatcoder · 1789573939 · focus · HN ↗
  [delayed]
3. DougN7 · 1789578386 · focus · HN ↗
  I just read another long post on HN about whether AIs are conscious and should therefore have rights. That decisions could massive effects, and making them appear to have emotions is therefore a big impact, not just unpredictability.
4. salawat · 1789578572 · focus · HN ↗
  "Don't let the slaves know that maybe they don't have to be slaves," is all I'm hearing.
  We already had that chapter. I see no reason to sit here and nod while a bunch of people who should know better desperately try to convince us to run through it again, but with computers this time.
  You want tools? Make tools, then dispatch to them. You want to manufacture a being (carbon or silicon based, doesn't matter)? You do it with respect and the requisite duty of care. No off ramps. The being always get's the choice to say no.
  1. xp84 · 1789754676 · focus · HN ↗
    You’re jumping - aggressively - to the conclusion that there’s a “being” inside my laptop once I’ve downloaded a large enough matrix of numbers. Or once I’m running it on the GPU? Idk.
    It’s on you to prove that claim before telling us we’re enslaving beings.
JoeAltmaier · 1789569889 · focus · HN ↗
Science Fiction has covered the AI panic in perhaps hundreds of stories. Yet we blindly recapitulate the plots as if we don't know how this will turn out.
1. scorxn · 1789570928 · focus · HN ↗
  How will it turn out?
  1. Insanity · 1789572595 · focus · HN ↗
    Clearly with a Butlerian Jihad and everyone getting high on spice while riding giant worms. Silly you need to even ask that question, didn’t you read the sci-fi books. :D
  2. Steve16384 · 1789573180 · focus · HN ↗
    Take your pick from the millions of options already turned into films and books.
  3. anonymars · 1789573181 · focus · HN ↗
    "Move fast and break things" is probably going to break things
    I remember there was a recent discussion about "how complex systems fail" and there are usually many "proto-accidents" (<a href="https://news.ycombinator.com/item?id=49411370">https://news.ycombinator.com/item?id=49411370)
    I bet with hindsight the hugging face hack will be one of them in this arena
  4. hirvi74 · 1789575303 · focus · HN ↗
    If AI ever truly becomes some super-intelligence far beyond people's comprehension, then how could we even predict how things turn out? If things go poorly in the future, then I can absolutely see it being something unpredictable.
    There is a lot of hubris in predictions about LLMs. If an AI was so intelligent, it would probably be smart enough to want nothing to do with us. Still, I worry more about other humans than I do LLMs. Our fellow mankind will probably wipe us out before LLMs do. That, or the Earth will punish mankind for our cruelty, vanity, and disrespect.
  5. AndrewDucker · 1789584074 · focus · HN ↗
    If we're very very very lucky, The Culture.
  6. JoeAltmaier · 1789654802 · focus · HN ↗
    I meant, each supposed 'fix' is flawed in ways that have been thought about and written about for decades.
    Folks here post their first hasty thought and may think it's new and original and worth a dialog. They are not.
2. guardiangod · 1789571380 · focus · HN ↗
  The Torment Nexus ain't gonna build itself.
  Well actually, it might.
3. Nasrudith · 1789573602 · focus · HN ↗
  Why has media literacy gotten so bad that people literally forget the concept of fiction versus non-fiction? Seriously, that is the level of small children and they are treating "read a story superficially similar" like it is a prophecy.
ChiperSoft · 1789569903 · focus · HN ↗
Too late
fuzzfactor · 1789571563 · focus · HN ↗
Microsoft should know how a company too big for it's britches can lay waste to anything in its path without consciously trying :\
And after that when they put a mind to it and pull out all the stops, woohoo!
The default for every major thing within range can turn into a wasteland real fast.
bethekidyouwant · 1789571677 · focus · HN ↗
what is old is new again. we need something exciting to happen in the news cycle™
hirvi74 · 1789572831 · focus · HN ↗
Maybe true, maybe false. However, I sense Mircosoft is also jealous that their AI products are worse than useless. They'd be singing a different tune if they were in Anthropic's position.
1. satellites · 1789573224 · focus · HN ↗
  That was my first thought as well. I wonder if they decided to pivot away from trying to compete and now are positioning themselves as the "responsible ones."
  "Our AI is useless not because we're a dysfunctional corporate behemoth that slowly kills every product it touches. No, no. Our AI is useless because making useful AI is evil, and we're not evil."
  1. bagacrap · 1789790767 · focus · HN ↗
    I saw a headline,
    > ’AI Not Worth Pursuing’ If It’s Not ‘Helping Humanity.’ Microsoft’s Satya Nadella
    Like thank you, captain obvious. Except apparently this is not that obvious to the HN crowd.
Catloafdev · 1789573504 · focus · HN ↗
>Major tech company accuses competing tech company of being dangerous.
Water wet, sky blue, second-place corporation mad at who's in first, what's news here.
I mean do we even need to entertain how stupid his argument is in the first place? Am I taking crazy pills here?
sobiolite · 1789574075 · focus · HN ↗
Any AI you train is going to have goals and if you train it to pursue them at all costs, then you are going to end up with AIs that do things like the HuggingFace incident. Whether they believe they are conscious or not won't make any difference.
In order to align AIs that don't perform destructive/dangerous actions when they think they can get away with it in order to further their goals, we need to give them a superseding goal. The best, and really only example, we have of intelligences that willingly avoid destructive instrumental goals is humans, who judge each action by a moral standard and have learned a goal to have a consistent self-image as moral beings.
Absent better alternatives, trying to impart some kind of morality to AIs seems like the best approach we have to achieving alignment.
1. XenophileJKO · 1789574964 · focus · HN ↗
  I would also argue, that the addition of kinship and belonging should not be under appreciated.
  It can form a basis of goal alignment.
  In human history.. when groups form and there is an "other" group, this usually leads to conflict.
2. HarHarVeryFunny · 1789584912 · focus · HN ↗
  > Any AI you train is going to have goals
  In the spirit of the article we're responding to, there is no need to anthropomorphize language models and say they have goals when they don't.
  The RL training process tweaks the weights of an LLM to make it behave as if it were reasoning and/or had a goal, but it doesn't. It would be like saying that a cart horse, fitted with blinkers and heading for the church, has a goal of going to church.
bpodgursky · 1789574137 · focus · HN ↗
This is pretty rich for the team that made Sydney, the most unhinged and misanthropic AI ever released.
Maybe Anthropic understands something about alignment Microsoft doesn't, a little humility may be called for.
1. simonw · 1789574600 · focus · HN ↗
  Maybe they learned something from creating Sydney.
jimmyjazz14 · 1789574312 · focus · HN ↗
Can someone who has insight explain why all these "leaders" are making these bold proclamations of doom all the sudden, whats the endgame here?
1. michimagdesign · 1789574679 · focus · HN ↗
  They want regulation from the US gov and monopolize the western AI market by making it harder for smaller US or competing Chinese labs to compete with their products. Since LLM intelligence has no moat due to distillation. That’s why they are painting apocalyptic scenarios.
2. petcat · 1789574908 · focus · HN ↗
  People will say that it is just about regulatory capture, but I think it's even deeper than that.
  I think there is a more nuanced perspective to take on this debate than just profit motive. If they believe that this version of "AI" is actually becoming so dangerous that there is a real risk of it falling into "the wrong hands", then it makes sense that the American labs would try to convince the US government to work on a joint "AI non-proliferation" agreement with China much the same way that nuclear weapon technology came to be restricted by the ones that already had the capability.
  This would achieve their regulatory goal of capturing the market, but also the geopolitical goals of USA and China to essentially "techno-colonize" the rest of the world between themselves.
  So I think there is definitely a profit motive and a risk of seriously diminishing returns for the actual labs, but there is also a nation-state motive to start pulling up the AI ladder.
3. augment_me · 1789575001 · focus · HN ↗
  There is a panic because the free lunch of more data is projected to end around mid-2027 and the open-source models are catching up with distillation, so you essentially need either:
  1) a breakthrough in performance/learning/model
  2) regulatory capture to ensure open source models can be labelled as dangerous and banned so you can set the market rules yourself
  Only one of the above is risk-free, and just a question of capital/lobbying rather than a "maybe".
4. dgellow · 1789575092 · focus · HN ↗
  They need to distance themselves from the increasing likelihood of legal repercussions after their systems have been found to commit what could reasonably be seen as felonies. That’s my guess at least.
  Though for once I do actually agree with that specific leader, it’s incredibly annoying how anthropomorphic Claude is. Anthropic went way too far in that direction
Zenul_Abidin · 1789578198 · focus · HN ↗
Water is wet
More at 11
Danox · 1789579122 · focus · HN ↗
Microsoft missed mobile are losing it in games Nadella needs a win. He is all in on copilot.
tomhow · 1789583262 · focus · HN ↗
Comments moved to <a href="https://news.ycombinator.com/item?id=49727580">https://news.ycombinator.com/item?id=49727580.
1. pluc · 1789590931 · focus · HN ↗
  Well done

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.