We had people departing from OpenAI, Anthropic and now Google. The pace the frontier open letter was signed by over 1,000 experts. At which point do we believe the message?
You don't think that there is competitive pressure between Anthropic, OpenAI, Google, the various Chinese model companies, and others, to advance the capabilities of their AIs?
Or you don't think that sufficiently advanced AI can cause (mass) harms?
I am saying that if they lost alignment and their new model started injecting cryptolockers, dropping all tables of productions databases into the software or if it stated writing poisonous cooking recipes there would be backlash, lawsuits, and more towards such a lab. Consumers don't want to trust such a dangerous model so they won't buy the tokens and the lab will not want to spend money on lawsuits.
>Or you don't think that sufficiently advanced AI can cause (mass) harms?
Even a feather can cause mass harm if it's used to sign a declaration of war. Something merely being capable of causing mass harm is not an issue and doesn't mean that the existence of feathers are an issue.
But this assumes an evil superintelligence. We don't even need that for terrible things to happen; we just need people doing people things and AI doing AI things at scale, and that scale is rapidly exploding as we scramble to deploy AI in the real world.
Case in point, that hallucination (which, if it was an evil superintelligence, could have been "strategic") that almost led to US boarding a Chinese ship over suspicions of nuclear weapons: <a href="https://www.msn.com/en-gb/news/other/us-military-ai-failure-reportedly-almost-started-a-war-with-china/ar-AA2czaxK" rel="nofollow">https://www.msn.com/en-gb/news/other/us-military-ai-failure-...
It doesn't have SkyNet, it just has to be WOPR. And it doesn't have to be people who are motivated to cause harm, it just has to be people making consequential decisions who are careless, distracted, paranoid or anxious -- which everybody is at some point or another.
And I almost kill people when stopping at a cross walk. That doesn't mean I harmed a pedestrian. Society is set up to be very robust. Humans themselves make mistakes and do bad things and society has had to learn how to live with that truth.
>it just has to be WOPR
Then why argue for setting a pace for the frontier labs if we've already surpassed WOPR level integration / intelligence.
> And I almost kill people when stopping at a cross walk. That doesn't mean I harmed a pedestrian. Society is set up to be very robust. Humans themselves make mistakes and do bad things and society has had to learn how to live with that truth.
Maybe not you personally, but many, many other people have killed many, many pedestrians, and when they exhibit a pattern of bad driving -- or other deviant behavior -- we take them off the streets. That is an example of society being robust.
Except, over here, we're rushing to make AI, which we know has many deviant behaviors and we know caused harms in many circumstances (starting with AI-assisted suicides), even more powerful AND deploy it in more and more real world systems!
As history and, literally, current events show us again and again, society has failure modes that lead to widespread harm and destruction. We've had world wars and then literally had multiple close calls with nuclear war right after. And then we have all that's going on out there. (In related news, the Pentagon threw a hissy fit because Anthropic would not let them use Claude for autonomous killing machines. Do I have to even mention what they've been up to these days?)
Most of these failures are caused by misaligned incentives and socioeconomic forces. The incentives and forces around AI have hints of many brand new failure modes that we can't even foresee because things are moving so fast.
> Then why argue for setting a pace for the frontier labs if we've already surpassed WOPR level integration / intelligence.
"We're already going down this mountain pass at 200 miles an hour, why slow down now?"
I think we should not just slow down frontier AI development, we should also slow down where and how that AI gets deployed.
When Astra and Fable stop making absolutely brain dead mistakes on simple tasks and not a moment sooner because I’m grounded in reality. Are you aware tech bros have been saying this same thing about every AI advancement since the 1950s? The reason being every AI researcher deep down wants to believe they’re God. This happens to also be why they’re some of the most insufferable people.
> Are you aware tech bros have been saying this same thing about every AI advancement since the 1950s?
Whatever AI people have been saying for the last 70 years, AI capabilities have improved stupendously in the last 10 years and there is no sign of the rate of improvement slowing down. I would like to deny it, but I can't.
That's a false sense of security. Something could have a serious issue, but provide overwhelmingly more benefit (cars, say), and thus people would use them anyway. If people did not use cars, then traffic fatalities would not be a concern.
People are using AI.
Your argument is that AI cannot be a serious concern because it has major known limitations. But most of us are using it despite the major known limitations. So the existence of the known limitations is irrelevant to determining what level of harms are possible -- unless the known limitations are directly limiting the harms. In practice, it seems like most current harms, and many of the (not stupid) feared harms, are completely possible despite the known limitations. Sometimes even exacerbated by them. ("Sorry, you told me to never stop your insulin pump, but I stopped your insulin pump while diagnosing a test failure. I will record in my memory file that I must always... hello? user?")
The (not stupid) parenthetical might be too weaselly for you, but that's how I see the "AI is going to rise up and intentionally subjugate or exterminate humanity" brand of doomerism. While I don't think that's strictly impossible, there are so many other ways harms that are much more imminent and much harder to defend against, that it seems foolish (to me!) to worry about those.
If you can stop a problem by pulling out the plug, you don't have a real problem.
If you don't want to pull out the plug, you do have a real problem. (Don't want, or can't afford to, or can't be bothered to so extremely that you never will, or whatever.)
It’s completely meaningless their opinions on what the moral and sociological implications of AI are, if anything being an ML researcher sitting at Google is extremely anticorrelated with the type of sophistication and erudition needed to consider these things. At most what is valuable is their insight into the various scenarios possible in terms of capabilities or use cases.
basic mathematical expectation, positive * chancep + negative *chancen. People smoke knowing there’s a chance to kill them because the positive outcome outweighs this risk. Creating AGI/ASI can solve every human problem, so the risks is taken, but this doesn’t mean you have to put uranium in your cigars, or do RSI.
It's not that simple; the goods of AlphaFold are mostly not yet realized, and some negative effects of AI are being realized.
More importantly, as I tried to make clear in my post, the main reason I quit is not specific fears of AI doom (I'm uncertain about that), but my belief that AI is progressing too fast.
Believing the message and being able to act on it are two pairs of shoes. There is nothing we can do. The box of pandora has been opened.
The companies have no tools for this, because there is a chance that the company who solves AI will eat the entire cake. And countries cannot do something about it, because clearly it gives an advantage in warfare. When it comes to weapons everything that is possible will be done, which means everybody needs to do it to not be defense less.
There is nothing we can do anymore. We can only hope that LeCun is right and that LLMs are a dead end and that everyone trying to solve AGI with then is essentially wasting money.
Just a reminder that Hinton left Google from a much higher and more influential perch (not dismissing current author, but, you know), and for similar risk and communication reasons, way back in 2023.
This stuff isn't new - it's just newly breaking through into mainstream conversation.
pu_pe · · focus · HN ↗
karahime · · focus · HN ↗
munksbeer · · focus · HN ↗
charcircuit · · focus · HN ↗
NiloCK · · focus · HN ↗
You don't think that there is competitive pressure between Anthropic, OpenAI, Google, the various Chinese model companies, and others, to advance the capabilities of their AIs?
Or you don't think that sufficiently advanced AI can cause (mass) harms?
charcircuit · · focus · HN ↗
>Or you don't think that sufficiently advanced AI can cause (mass) harms?
Even a feather can cause mass harm if it's used to sign a declaration of war. Something merely being capable of causing mass harm is not an issue and doesn't mean that the existence of feathers are an issue.
keeda · · focus · HN ↗
Case in point, that hallucination (which, if it was an evil superintelligence, could have been "strategic") that almost led to US boarding a Chinese ship over suspicions of nuclear weapons: <a href="https://www.msn.com/en-gb/news/other/us-military-ai-failure-reportedly-almost-started-a-war-with-china/ar-AA2czaxK" rel="nofollow">https://www.msn.com/en-gb/news/other/us-military-ai-failure-...
It doesn't have SkyNet, it just has to be WOPR. And it doesn't have to be people who are motivated to cause harm, it just has to be people making consequential decisions who are careless, distracted, paranoid or anxious -- which everybody is at some point or another.
charcircuit · · focus · HN ↗
And I almost kill people when stopping at a cross walk. That doesn't mean I harmed a pedestrian. Society is set up to be very robust. Humans themselves make mistakes and do bad things and society has had to learn how to live with that truth.
>it just has to be WOPR
Then why argue for setting a pace for the frontier labs if we've already surpassed WOPR level integration / intelligence.
keeda · · focus · HN ↗
Maybe not you personally, but many, many other people have killed many, many pedestrians, and when they exhibit a pattern of bad driving -- or other deviant behavior -- we take them off the streets. That is an example of society being robust.
Except, over here, we're rushing to make AI, which we know has many deviant behaviors and we know caused harms in many circumstances (starting with AI-assisted suicides), even more powerful AND deploy it in more and more real world systems!
As history and, literally, current events show us again and again, society has failure modes that lead to widespread harm and destruction. We've had world wars and then literally had multiple close calls with nuclear war right after. And then we have all that's going on out there. (In related news, the Pentagon threw a hissy fit because Anthropic would not let them use Claude for autonomous killing machines. Do I have to even mention what they've been up to these days?)
Most of these failures are caused by misaligned incentives and socioeconomic forces. The incentives and forces around AI have hints of many brand new failure modes that we can't even foresee because things are moving so fast.
> Then why argue for setting a pace for the frontier labs if we've already surpassed WOPR level integration / intelligence.
"We're already going down this mountain pass at 200 miles an hour, why slow down now?"
I think we should not just slow down frontier AI development, we should also slow down where and how that AI gets deployed.
chlorion · · focus · HN ↗
There are trillions of dollars at stake here, do you think getting 1000 people on board is that difficult with this level of money? lol
sowhat1 · · focus · HN ↗
perching_aix · · focus · HN ↗
Trope circa 1950 not true, because of trope circa even earlier...
pu_pe · · focus · HN ↗
roca · · focus · HN ↗
Whatever AI people have been saying for the last 70 years, AI capabilities have improved stupendously in the last 10 years and there is no sign of the rate of improvement slowing down. I would like to deny it, but I can't.
sfink · · focus · HN ↗
People are using AI.
Your argument is that AI cannot be a serious concern because it has major known limitations. But most of us are using it despite the major known limitations. So the existence of the known limitations is irrelevant to determining what level of harms are possible -- unless the known limitations are directly limiting the harms. In practice, it seems like most current harms, and many of the (not stupid) feared harms, are completely possible despite the known limitations. Sometimes even exacerbated by them. ("Sorry, you told me to never stop your insulin pump, but I stopped your insulin pump while diagnosing a test failure. I will record in my memory file that I must always... hello? user?")
The (not stupid) parenthetical might be too weaselly for you, but that's how I see the "AI is going to rise up and intentionally subjugate or exterminate humanity" brand of doomerism. While I don't think that's strictly impossible, there are so many other ways harms that are much more imminent and much harder to defend against, that it seems foolish (to me!) to worry about those.
If you can stop a problem by pulling out the plug, you don't have a real problem.
If you don't want to pull out the plug, you do have a real problem. (Don't want, or can't afford to, or can't be bothered to so extremely that you never will, or whatever.)
monideas · · focus · HN ↗
worik · · focus · HN ↗
Then they say they are going to keep doing it...
Forgive me for alerting you: they are liars
jadermcs · · focus · HN ↗
keithnz · · focus · HN ↗
roca · · focus · HN ↗
keithnz · · focus · HN ↗
roca · · focus · HN ↗
More importantly, as I tried to make clear in my post, the main reason I quit is not specific fears of AI doom (I'm uncertain about that), but my belief that AI is progressing too fast.
blitzar · · focus · HN ↗
Yet the same 1,000 people are all raising funds to do more on the frontier.
_ink_ · · focus · HN ↗
The companies have no tools for this, because there is a chance that the company who solves AI will eat the entire cake. And countries cannot do something about it, because clearly it gives an advantage in warfare. When it comes to weapons everything that is possible will be done, which means everybody needs to do it to not be defense less.
There is nothing we can do anymore. We can only hope that LeCun is right and that LLMs are a dead end and that everyone trying to solve AGI with then is essentially wasting money.
NiloCK · · focus · HN ↗
Just a reminder that Hinton left Google from a much higher and more influential perch (not dismissing current author, but, you know), and for similar risk and communication reasons, way back in 2023.
This stuff isn't new - it's just newly breaking through into mainstream conversation.