This whole "we can't trust the people warning us because they're either weirdos or involved" deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that.
Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.
Reinforcement learning makes it something different. It becomes much more of a search engine through next-token-space that targets the training objective. Better to think about it like that, and then you'll see why "these things have motive" is not a terrible analogy, and you'll better be able to anticipate what they do.
arcticfox · · focus · HN ↗
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
0x20cowboy · · focus · HN ↗
It’s clear what a big bomb could do, but these “whistleblowers” are not showing anything.
They’re like “imagine…”, no show me internal documents or training data or something other than trust me bro.
I can imagine quite a bit. That doesn't mean anything.
arcticfox · · focus · HN ↗
I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that.
Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.
stickfigure · · focus · HN ↗
You can't talk like these things have motive. They're just predicting the next word in the context. Stop anthopomorphizing them.
mitthrowaway2 · · focus · HN ↗