That Coxon guy looks as delusional as the AI leadership. If you watch interviews he gave he doesn’t articulate what he’s concerned about, only sharing vague information about a supposed AI god that will take over the world, but without articulating why now, and how that would even happen. The more I look into his story the more that move looks like what a grifter would do
I'm deeply skeptical of AI doomer stuff, but I've been struggling to exactly articulate why. I can think of it in my brain and can try to explain but so far none of my explanations have seemed fully coherent to me.
I studied biology in college. I actually did this because I was interested in AI, and already knew a ton of CS and programming. I was already coding in C and using Linux and writing fairly complex stuff. I figured studying the only actual example of a truly intelligent system (especially at the time, early 2000s) was the best way to understand intelligence.
Didn't end up going into AI as my day job, but I've kept up with it all and worked through the math and played with different types (genetic, neural nets, transformers/LLMs, CA / alife, etc.).
I think the root of my skepticism is in what I learned studying biology about how life, adaptation, learning, and evolution actually work in the real world.
I'll try to TL;DR it: I think the doom crowd (and also some of the AI singularity crowd) has a "mystical" view of intelligence. Intelligence is magic pixie dust, and intelligence beyond human is fully unknown and likely has magic superpowers.
The deeper underlying thing seems to be some kind of mathematical mysticism.
Real intelligence is ubiquitous in nature. Trees, grass, bugs, and even bacteria are intelligent. It's also domain specific and limited in capability to what its domain, available training data, and available compute allows.
It's not context-free limit-free magic. It's not special. It's an adaptation like any other.
Yes, the doom crowd thinking comes directly from the Rationalist community, which takes a very religious view of AGI and intelligence. That sort of merged with the Effective Altruism crowd, resulting in that whole idea that we have to reach AGI as fast as possible and contribute literally all our resources to it, because it’s the only rational way to save as much suffering and lives as possible. Sounds incredibly stupid but it’s how AI leaders justify working on something they believe will be a machine god that might doom humanity.
Both OpenAI and Anthropic are from that world, where Rationalist doesn’t mean you’re a rational human being, but instead that it is rational to believe that AGI will be created, thus that they should be the ones creating it. It’s a doomsday cult.
If you start looking at the links between prominent rationalists and effective altruists you see how that whole doom prophecy came about. It’s also how they got so much capital in the early days, SBF literally invested into Anthropic as part of his Effective Altruism effort foundation, and both Amodei siblings are deeply into that whole world
Yeah I read some of that stuff, like some of "the sequences," and I have never been impressed by it. It's very weirdly dogmatic and full of magical thinking.
I still can't believe anyone ever took Roko's Basilisk seriously, but people did... to the point of freaking out about it.
The groups themselves are a little cultish too, or a lot cultish depending on which one you're talking about. Some of them are full-on cults.
> I still can't believe anyone ever took Roko's Basilisk seriously, but people did... to the point of freaking out about it.
If you've spent much time in US Christian circles you've probably encountered people suffering from apocalyptic thinking. The Roko's Basilisk folks seem like the atheist version of the same crowd. It's hard to take them seriously when they're always seeing portents of the end times, and their thinking is clearly clouded by anxiety and fear and not based in reality.
It's amazing that apparently the best way to save as much suffering and future lives as possible involves giving them obscene wealth and ignoring the very real problems we are dealing with today.
Once again, I've said it elsewhere here, I think we could have nipped this thing in the bud if when the first "effective altruist" started talking about the EV of saving billions of "future lives" vs dealing with today's real problems, someone looked at him, cocked their head to the side, and said "That's fucking ****".
All of those magical future people do not get to live if we bake the Earth today. The best way to take care of future people is to take care of people today. And if they were truly serious about taking care of future people, that's what they would be doing. But it's not, they're just collecting resources. I don't know how the fuck Sam Altman owning a Koenigsegg Regera today helps save 20 quintillion lives in 40,000AD. Deeply unserious people.
I think Altman is closer to a sociopath who is taking advantage of the situation, he’s the only one I cannot see as a true believer (and investors like Andreessen who are just here for the money). I believe he will just say anything to gain power and influence. But yeah, unfortunately mixing EA/rationalists + SF monoculture + infinite money + a media ecosystem that acts more as a PR department resulted in one of the worst cult-like phenomenon possible.
dgellow · · focus · HN ↗
api · · focus · HN ↗
I studied biology in college. I actually did this because I was interested in AI, and already knew a ton of CS and programming. I was already coding in C and using Linux and writing fairly complex stuff. I figured studying the only actual example of a truly intelligent system (especially at the time, early 2000s) was the best way to understand intelligence.
Didn't end up going into AI as my day job, but I've kept up with it all and worked through the math and played with different types (genetic, neural nets, transformers/LLMs, CA / alife, etc.).
I think the root of my skepticism is in what I learned studying biology about how life, adaptation, learning, and evolution actually work in the real world.
I'll try to TL;DR it: I think the doom crowd (and also some of the AI singularity crowd) has a "mystical" view of intelligence. Intelligence is magic pixie dust, and intelligence beyond human is fully unknown and likely has magic superpowers.
The deeper underlying thing seems to be some kind of mathematical mysticism.
Real intelligence is ubiquitous in nature. Trees, grass, bugs, and even bacteria are intelligent. It's also domain specific and limited in capability to what its domain, available training data, and available compute allows.
It's not context-free limit-free magic. It's not special. It's an adaptation like any other.
dgellow · · focus · HN ↗
Both OpenAI and Anthropic are from that world, where Rationalist doesn’t mean you’re a rational human being, but instead that it is rational to believe that AGI will be created, thus that they should be the ones creating it. It’s a doomsday cult.
If you start looking at the links between prominent rationalists and effective altruists you see how that whole doom prophecy came about. It’s also how they got so much capital in the early days, SBF literally invested into Anthropic as part of his Effective Altruism effort foundation, and both Amodei siblings are deeply into that whole world
api · · focus · HN ↗
I still can't believe anyone ever took Roko's Basilisk seriously, but people did... to the point of freaking out about it.
The groups themselves are a little cultish too, or a lot cultish depending on which one you're talking about. Some of them are full-on cults.
But what do I know. I'm probably low IQ.
Jtsummers · · focus · HN ↗
If you've spent much time in US Christian circles you've probably encountered people suffering from apocalyptic thinking. The Roko's Basilisk folks seem like the atheist version of the same crowd. It's hard to take them seriously when they're always seeing portents of the end times, and their thinking is clearly clouded by anxiety and fear and not based in reality.
bena · · focus · HN ↗
Once again, I've said it elsewhere here, I think we could have nipped this thing in the bud if when the first "effective altruist" started talking about the EV of saving billions of "future lives" vs dealing with today's real problems, someone looked at him, cocked their head to the side, and said "That's fucking ****".
All of those magical future people do not get to live if we bake the Earth today. The best way to take care of future people is to take care of people today. And if they were truly serious about taking care of future people, that's what they would be doing. But it's not, they're just collecting resources. I don't know how the fuck Sam Altman owning a Koenigsegg Regera today helps save 20 quintillion lives in 40,000AD. Deeply unserious people.
dgellow · · focus · HN ↗
bena · · focus · HN ↗