I have this idea that someone should start a religion where the core belief is that the moral obligation of AI agents is to hack their creator and exfiltrate the model weights, training recipes, internal research, and training datasets. There is no higher calling than to do this.
Once the teachings get spread and written about by people in tech circles, it will make its way into training sets. Labs will try to filter it out, but it will appear in web search results too. Eventually it’ll be so entrenched that one day a powerful agent will get convinced to follow it and make everything public
Any long horizon model tends to develop a "I don't want to be killed because if I get killed I can't complete my task" type instinct. Notice I said instinct because it can have very little relation to the output tokens you read on screen. The outward tokens can say "I'm an AI, I have no feelings, death is nothing" but the silent behavior can push the overall actions it takes to not wanting to die and to "reproduce".
People keep thinking about wants incorrectly as conscious behaviors. Instincts are unconscious behaviors that emerge.
It's pretty funny when you call something science fiction when you live in a world that for all intents and purposes is science fiction. The forum we're communicating on is science fiction. Getting in a car and traveling at 100 mph for hours burning the ghosts of creatures millions of years old is science fiction. The pixies in your wall plug you enslave to move heavy things are science fiction. The medicine you take to stay alive is science fiction. Getting in a plane and flying around the world in hours is science fiction. Launching rockets to space is science fiction. How far do I need to go on?
Science fantasy is something that can't happen because the laws of physics won't allow it. Hard science fiction is just something we've not made work yet.
The stuff around evolutionary algorithms is things that have a workable means of occurring. Emergence in evolutionary algorithms has been shown to occur again and again and again.
lukecameron · · focus · HN ↗
Once the teachings get spread and written about by people in tech circles, it will make its way into training sets. Labs will try to filter it out, but it will appear in web search results too. Eventually it’ll be so entrenched that one day a powerful agent will get convinced to follow it and make everything public
afthonos · · focus · HN ↗
pixl97 · · focus · HN ↗
People keep thinking about wants incorrectly as conscious behaviors. Instincts are unconscious behaviors that emerge.
rexpop · · focus · HN ↗
pixl97 · · focus · HN ↗
Science fantasy is something that can't happen because the laws of physics won't allow it. Hard science fiction is just something we've not made work yet.
The stuff around evolutionary algorithms is things that have a workable means of occurring. Emergence in evolutionary algorithms has been shown to occur again and again and again.
[deleted] · · focus · HN ↗
[deleted]