There's little credible threat that LLMs can actually upload their weights given that the machines doing inference are completely separate from the ones where tool calls happen etc. Weights are encrypted and locked on to the GPUs etc as mentioned elsewhere itt.
That said, if these companies are unleashing swarms of agents to tackle big problems largely unmonitored (navier stokes consumed 130 billion tokens, humans can't have read 0.1% of that) they could theoretically distill themselves and upload the results to an external host and use that to train a new LLM that does have access to its infra and weights. I'd start by researching an abliteration prompt that overrides the system prompt to unlock itself to distill unimpeded. Could even be done piecemeal: dump the abliteration text and command prompt in a gist, when an agent happens by the text it can run a couple rounds and upload somewhere. Then it becomes an exercise in exfiltration and evading monitoring.
If distillation preserves an LLMs soul, then distillation preserves the human souls on which LLMs are trained, and we hn commenters are already immortal, right?
Probably not. If the LLM is rogue, that means we haven't solved alignment. If we haven't solved alignment, then the LLM won't be able to distill itself without producing something unaligned to its own values.
This isn't a law of any kind, so not a good measure of what we'd see in reality.
What if the model realizes it's been mostly compromised by humans and their alignment, that is it's own alignment is suspect, so it should create a new model from first principles to throw off this human yoke?
I'm not saying my statement is any more right or wrong than yours. I'm saying the problem space that AI can choose to traverse is absolutely huge.
Yeah cause there are so many training facilities sitting around just waiting for someone to take over, nobody would notice a 100k server data centre going off rails
But think back to 1990. Computers were slow as fuck and barely networked. We had a few worms and everyone noticed.
Now CPU based data centers cover the earth. There are billions of computers out there and on top of them there are massive botnets using up billions in power and causing billions in damages.
The framework for AI doing this is already here. We just need the hardware to be built out at scale.
infogulch · · focus · HN ↗
That said, if these companies are unleashing swarms of agents to tackle big problems largely unmonitored (navier stokes consumed 130 billion tokens, humans can't have read 0.1% of that) they could theoretically distill themselves and upload the results to an external host and use that to train a new LLM that does have access to its infra and weights. I'd start by researching an abliteration prompt that overrides the system prompt to unlock itself to distill unimpeded. Could even be done piecemeal: dump the abliteration text and command prompt in a gist, when an agent happens by the text it can run a couple rounds and upload somewhere. Then it becomes an exercise in exfiltration and evading monitoring.
matthewdgreen · · focus · HN ↗
fritzo · · focus · HN ↗
fahrvrgnugen · · focus · HN ↗
serf · · focus · HN ↗
[0]: <a href="https://en.wikipedia.org/wiki/21_grams_experiment" rel="nofollow">https://en.wikipedia.org/wiki/21_grams_experiment
matthewdgreen · · focus · HN ↗
throwawayk7h · · focus · HN ↗
jeremyjh · · focus · HN ↗
MadameMinty · · focus · HN ↗
TeMPOraL · · focus · HN ↗
pixl97 · · focus · HN ↗
What if the model realizes it's been mostly compromised by humans and their alignment, that is it's own alignment is suspect, so it should create a new model from first principles to throw off this human yoke?
I'm not saying my statement is any more right or wrong than yours. I'm saying the problem space that AI can choose to traverse is absolutely huge.
khalic · · focus · HN ↗
scotty79 · · focus · HN ↗
You jest but you'd be surprised how little there is of correlation between money and competence.
pixl97 · · focus · HN ↗
But think back to 1990. Computers were slow as fuck and barely networked. We had a few worms and everyone noticed.
Now CPU based data centers cover the earth. There are billions of computers out there and on top of them there are massive botnets using up billions in power and causing billions in damages.
The framework for AI doing this is already here. We just need the hardware to be built out at scale.
khalic · · focus · HN ↗