Thread

AI companies in race to demonstrate their model most threatening to humanity

441 points · 399 comments · ljewalsh

chasd00 · 1790595720 · focus · HN ↗
I’ve never seen CEOs work so hard to make the public aware of how dangerous and out of control their flagship product is. It makes me automatically assume they’re scheming about something else like regulatory capture to protect their market.
1. meowface · 1790595788 · focus · HN ↗
  Or perhaps they've been saying this for literally decades because they earnestly believe it.
  1. ulfw · 1790595960 · focus · HN ↗
    Which CEO said this for 'literally decades'? And if so why are they still pursuing it then?
    Most children have learned touching a hot stove is bad for them. Then why still do it at the age of 40?
    frabcus · 1790596885 · focus · HN ↗
    Sam Altman in 2015 - "Development of superhuman machine intelligence (SMI) [1] is probably the greatest threat to the continued existence of humanity" <a href="https://blog.samaltman.com/machine-intelligence-part-1" rel="nofollow">https://blog.samaltman.com/machine-intelligence-part-1
    Elon Musk in 2014 - "We need to be super careful with AI. Potentially more dangerous than nukes" <a href="https://x.com/elonmusk/status/495759307346952192" rel="nofollow">https://x.com/elonmusk/status/495759307346952192
    Dario Amodei in 2017 - "There’s a long tail of things of varying degrees of badness that could happen. I think at the extreme end is the Nick Bostrom" <a href="https://80000hours.org/podcast/episodes/the-world-needs-ai-researchers-heres-how-to-become-one/" rel="nofollow">https://80000hours.org/podcast/episodes/the-world-needs-ai-r...
    They're still pursuing it because without an international treaty, they think it is inevitable. In the case of Dario, they want to align it just enough and be first to stop others causing harm (a vainglorious strategy, yet still a strategy). In the case of Musk and Altman, not completely clear to me.
    Google feels less important right now probably, and Demis Hassabis no longer runs Deep Mind as CEO. However, he talked about this early too, and his reason for proceeding was he was only doing narrow AI (e.g. protein folding) which is fundamentally less dangerous.
    KellyCriterion · 1790597257 · focus · HN ↗
    Wondering how many details & knowhow they had already back then when saying these quotes:
    - For Altman Id say he was already deeper in the "transformer & model landscape" in 2015?
    - Musk in 2014: Not sure how much he was aware of the current-state back then
    frabcus · 1790597569 · focus · HN ↗
    Musk was told by Demis Hassabis in 2012, see <a href="https://time.com/6310076/elon-musk-ai-walter-isaacson-biography/" rel="nofollow">https://time.com/6310076/elon-musk-ai-walter-isaacson-biogra...
    Altman read Nick Bostrom's book Superintelligence, he mentions it in this post <a href="https://blog.samaltman.com/machine-intelligence-part-1" rel="nofollow">https://blog.samaltman.com/machine-intelligence-part-1
    I also read that book in 2014, and it is how I came to understand this problem early.
    meowface · 1790607346 · focus · HN ↗
    2012 - 2015 seems to have been around the time a lot of people first started taking ASI and ASI existential risk very seriously as things that will be relevant in our lifetime, and when they decided to try to build and/or prevent it.
    thfuran · 1790598115 · focus · HN ↗
    >In the case of Musk and Altman, not completely clear to me.
    They probably just want to ensure that the first AGI is a nazi.
    meowface · 1790605899 · focus · HN ↗
    While Musk is now far-right (albeit seemingly not a Hitlerist - yet), Altman is pretty much a liberal.
    ks2048 · 1790610000 · focus · HN ↗
    > Demis Hassabis ... he was only doing narrow AI (e.g. protein folding) which is fundamentally less dangerous.
    I don't think this is true. He repeatedly says things like "I’ve been working towards AGI my whole life".
    meowface · 1790613279 · focus · HN ↗
    Correct. He - I do not think unjustly - has long had AGI as the goal. "Solve AGI, then use AGI to solve medicine, and physics, and the rest of science, and create a better world." has been his thing basically forever.
    frabcus · 1790629898 · focus · HN ↗
    I'm thinking of when he warned Musk in 2012:
    "Hassabis told him to add another potential threat to the list: artificial intelligence. Machines could become superintelligent and surpass us mere mortals, -perhaps even decide to dispose of us."
    <a href="https://time.com/6310076/elon-musk-ai-walter-isaacson-biography/" rel="nofollow">https://time.com/6310076/elon-musk-ai-walter-isaacson-biogra...
    You're right that the picture is more complicated than I wrote above. Quickly looking, it seems that Shane Legg was concerned very very early on. I can't find anything tangible about what their strategy was to avoid the danger. It looks like Hassabis had longer timescales than Legg, so perhaps just thought they had more room before it really mattered?
    frabcus · 1790597033 · focus · HN ↗
    BTW, just to be clear it is insane that they're all still pursuing it, so completely reasonable to be sceptical and come up with other reasons. This is, however, a crazy situation. Lots of other people have refused to develop it - natural selection has picked those with a galaxy brained or vain strategy involving continuing to develop it to be running the companies we have.
    meowface · 1790612665 · focus · HN ↗
    Their mindset is: the upside is likely so, so, so immense, and the probability of amazing outcomes is so much higher than the probability of catastrophic outcomes, that the gamble is worth it, and it is not in fact a pure gamble because they can (in theory) try really hard to build it really slowly and carefully.
    So even with their earnest belief that there is a > 5% chance this technology could be 1) capable of autonomously killing billions of people during this century and 2) may not be sufficiently aligned and might actually act on that capability, the EV basically still sums positive, to them.
    Not saying this is necessarily a good or moral calculus. But it is what they think. There is no conspiracy or plot or deception.
  2. chinathrow · 1790595979 · focus · HN ↗
    If they believe it, they can stop. All of them. There's no sign though that any pacing/slowdown or whatever is in place, zero.
    perching_aix · 1790596412 · focus · HN ↗
    I don't think that's how the market works. Anti-slavery regulation didn't stop slavery either, just moved it abroad. Same for trash handling regulations, it's just shipped abroad. Falling behind on useful AI capability is obviously not in the interest of anyone, so either all players coordinate, or an arms race towards ??? it is.
    It may very well be self serving what they're saying, but that doesn't mean it isn't simultaneously true.
    benrutter · 1790597031 · focus · HN ↗
    > I don't think that's how the market works
    I think you're sort of right, but then I'd have found it very strange / deeply suspicious, if during the time of slavery being legal, advocates of abolitionism were practicing slave-owners.
    If you think AI is going to destroy humanity, it still feels absolutely crazy to leap to "well I might as well be the one to profit from destroying humanity".
    perching_aix · 1790598167 · focus · HN ↗
    Quick search does suggest to me that a number of - even prominent - abolitionists were slave owners, but I'll be honest, the topic is hardly my specialty.
    As for the other point, crazy, cynical, and even unethical as it may come off, it is hardly inconsistent with how people generally behave when met with hard limits (i.e. making the most of what's available, and approximating the desired state if that is unattainable). This is just a variation on the same.
    Something could be said about "what if they're mistaken", but if all players are making a mistake, then that makes little practical difference; it may as well be real.
    jquery · 1790606779 · focus · HN ↗
    A quick search suggests to me that those same abolitionists were born into families that were slave owners, but rejected their upbringing to become abolitionists. It doesn't suggest that a large fraction of practicing abolitionists were contemporary slave owners.
    perching_aix · 1790621792 · focus · HN ↗
    And then this interacts with the above analogy in whatever manner.
    dns_snek · 1790598383 · focus · HN ↗
    > Falling behind on useful AI capability is obviously not in the interest of anyone
    Hence the argument that they're either full of shit, or psychopaths who need to be stopped at all costs.
    ForHackernews · 1790596505 · focus · HN ↗
    There was a NYT piece about this recently. They are believers in a millennialist cult in which the arrival of an omnipotent machine-god is thought inevitable, and the only question is whether it will be "aligned" (good) or "misaligned" (evil).
    They are the good heroes racing to make sure they bring forth the Aligned God because if they fail, surely the bad people will raise the Misaligned Devil.
    Ref w/gift link <a href="https://x.com/sapinker/status/2096236079477096630" rel="nofollow">https://x.com/sapinker/status/2096236079477096630
    Sharlin · 1790596767 · focus · HN ↗
    This cult stuff was tired in 2010 (hey, at least back then there was One True Leader!) and not any less tired now.
    ForHackernews · 1790596877 · focus · HN ↗
    I'm not sure how else we are to interpret the behavior of the AI doomers & the AI developers (who are frequently the same people)
    frabcus · 1790597212 · focus · HN ↗
    This was fun games until their fictional machine god started actually getting very intelligent.
    Sharlin · 1790598651 · focus · HN ↗
    There is a fairly specific definition of what a cult is, and just being publicly worried about an existential risk isn’t that.
    ForHackernews · 1790599753 · focus · HN ↗
    If you all believe in the same prophecy and hang out at the same orgies together, I'm willing to call that a "cult", at least in the colloquial sense.
    suddenlybananas · 1790603523 · focus · HN ↗
    Would you at least concede that the Zizian offshoot was a cult? They murdered multiple people!
    meowface · 1790605600 · focus · HN ↗
    Rationalism has spawned many offshoot cults which are genuine and in some cases dangerous cults, but rationalism and Effective Altruism are not themselves cults.
    Most people among both ideologies have long believed in the arrival of a machine demigod of sorts, but I kind of am skeptical of the notion that there are any tech people who have not always held similar beliefs in the long run. Like, all the people reading this: you think in a thousand years from now humans are going to be doing most labor? Obviously far, far, far smarter machines will have unimaginably changed things by then, and/or we'll all be gone.
    The difference is most people in tech have historically thought it's maybe a century or three away, while the rationalists think it's a decade or two away (and also thought it would be 2020s - 2040s even decades prior to now).
    ForHackernews · 1790606752 · focus · HN ↗
    > I kind of am skeptical of the notion that there are any tech people who have not always held similar beliefs
    Ladies and gentlemen, I give you your totally-not-in-an-insular cult members.
    A thousand years is a long time, and the world will be so different that it's hard to make any concrete predictions. But yes, if I had to wager, I'd bet that what humans conceive of as "labor" in 3026 will still be whatever difficult or annoying bits are done by humans.
    In 1026 nearly all labor was agricultural labor. If you told a Roman peasant that magical self-driving contraptions could sow and reap the fields, he might wonder what could possibly be left for humans to do in such a world. And yet, here we are, and I still have to update JIRA tickets.
    For my myself, I expect AI to prove a more transformative technology than Docker but less transformative than the sewing machine.
    meowface · 1790606981 · focus · HN ↗
    You could be right, but short of an eternal Butlerian jihad or other extremely strict global government action that effectively bans most forms of AI forever, I think the probabilities of that are exceedingly low.
    What you think won't happen in 1000 years is what I think is almost certainly happening within 40 years. I'd bet everything I own it'll be way, way, way more impactful than the sewing machine within 40 years. I think some bits of physical labor may still exist in 40 years, but almost none, and only when a human really wants to talk to or deal with a human, such as in some medical and therapeutic professions, some art fields, etc. (Again, assuming no universal outlawing of AI.)
    ForHackernews · 1790607639 · focus · HN ↗
    Now you're already hedging and narrowing your claims: "almost" none, "some" physical" labor. For someone who claims to be willing to stake his whole bank account, you're not that confident.
    Why do you think restaurants with waiters keep opening when the automat has existed since 1895? <a href="https://en.wikipedia.org/wiki/Quisisana" rel="nofollow">https://en.wikipedia.org/wiki/Quisisana
    You are also probably unfamiliar with the history of the sewing machine and its impact on society. As AI has already poisoned search engines, here's an AI slop overview that hits some of the main points: <a href="https://sciencepedia.blog/sewing-machine-societal-importance" rel="nofollow">https://sciencepedia.blog/sewing-machine-societal-importance
    meowface · 1790608132 · focus · HN ↗
    When money's on the line, everyone hedges.
    Yes, I think even in some crazy distant future utopia, humans will still often want to interact with humans for things, and even to pay them for some services. Tasks that include speaking one-on-one and producing works of art will always retain some value to people, even when AIs can one day trivially produce things which most humans find aesthetically superior to anything a human could ever produce alone. People will seek other things out of art beyond aesthetics, like expressions of human emotion.
    Humans will enjoy and seek out human contact. This is not surprising. Just, for economy-impacting labor, I think it's almost all going to be machines, with very few humans playing any role. Even an oversight or accountability role. I think there's a 95% chance of it by 2100 and an 85% chance of it by 2066, conditional on no laws that forbid replacing jobs with AI.
    ForHackernews · 1790609171 · focus · HN ↗
    I could point out that that as far as economy-impacting labor goes, it's almost all machines already today. At least in the developed world, few people do any kind of work that isn't at least partially mechanized. You don't see it because lawnmowers and spreadsheets are "normal" machines in your mind.
    You're making the "this time it's different" case that AI will be categorically different from every other form of labor-saving automation that's come before it. I work with these models every day and I'm not buying it.
    meowface · 1790610523 · focus · HN ↗
    The hypothesis is about theoretical future models, not existing public ones. I'm making a prediction based on a pace of improvement that I believe will probably continue unabated and which likely is not going to hit significant bottlenecks for very long.
    ForHackernews · 1790610740 · focus · HN ↗
    Well, good luck with that. Consider reading When Prophecy Fails to steel yourself for the possibility that 2036 ends up largely the same as 2026, only with more spam and surveillance.
    _kb · 1790596819 · focus · HN ↗
    The only thing that can stop a bad guy with a god is a good guy with a god.
    meowface · 1790605410 · focus · HN ↗
    Pinker unfortunately has very little clue what he's talking about here. I eagerly await Scott Alexander's inevitable essay response to him.
    jquery · 1790607493 · focus · HN ↗
    Thank you for that gift link, I had no idea that Altman and other "leading" (deep-pocketed) figures in American AI had such close ties to that raving lunatic Eliezer Yudkowsky. It explains so much about why China is approaching AI like it's a friendly technology but American CEOs keep blabbering on about the End Times.
    meowface · 1790607911 · focus · HN ↗
    Sam Altman's Twitter bio a few years ago was something like "#1 eliezer yudkowsky fanfic fan".
    (I personally lean towards "Yudkowsky is basically correct about everything, but we should probably try to build ASI anyway, slowly and carefully". I'm pretty much where Amodei's at. I assume Altman has a similar idea, though is a bit less pessimistic.)
    yes_man · 1790596917 · focus · HN ↗
    OpenAI claims to have paused the training of their ”most capable” models: <a href="https://apnews.com/article/ai-openai-anthropic-agents-rogue-hack-2f8a2b9024d4f06793bcca12f8089d20" rel="nofollow">https://apnews.com/article/ai-openai-anthropic-agents-rogue-...
    Of course I am not making an argument for them being honest as there could be entirely other reasons to a pause (like financial ones). Just pointing out ”no signs of pacing” is not true
    jquery · 1790606641 · focus · HN ↗
    If you read between the lines, that's not exactly a pause. That's just them lamenting that unsupervised training has temporarily hit a limit, and using that as yet another reason the industry needs to be regulated so that OpenAI, Claude, and MechaHitler are the only approved AIs.
    meowface · 1790606802 · focus · HN ↗
    Oh my god, we have these "it's regulatory capture!" populists even on HN now? I thought we were better than this.
    They want coordinated action to prevent a multipolar trap and potential catastrophe by future superintelligent AI (which may emerge quite rapidly if RSI ever hits an inflection point). It's not rocket science. The rules would apply to everyone. The rules they're advocating for are not "only OpenAI and Anthropic can make AI".
    jquery · 1790607920 · focus · HN ↗
    Give me a break. It's possible to actually think Eliezer Yudkowsky is a complete lunatic and his Roko's Basilisk essay should've been the end of him as a "thought leader" in the AI space. He has close ties to Altman, Musk, Thiel, and others, so as far as I'm concerned it's just an End Times cult.
    meowface · 1790610207 · focus · HN ↗
    You were misinformed about his response to Roko's basilisk. He never thought it was plausible. He freaked out for an unrelated meta reason, basically, and later admitted he overreacted. (Basically he did not want forum members putting attempted mindviruses out into the ether as a general principle.)
    Critics of Yudkowsky have been trying to misportray that for years to make him seem delusional. You can see an example of that here: <a href="https://www.tracingwoodgrains.com/p/reliable-sources-how-wikipedia-admin" rel="nofollow">https://www.tracingwoodgrains.com/p/reliable-sources-how-wik...
    (Roko himself is genuinely crazy, but largely because he became a neo-Nazi.)
    If you think ASI is plausible this century, you would have no reason to not fear the end times yourself. If you don't think it's plausible it'll be created this century or perhaps ever, fair enough, but I'd bet against that.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.