US Military had close call after using AI for hallucinated intelligence report
Thread
Unofficial Hacker News client; not affiliated with Y Combinator.
US Military had close call after using AI for hallucinated intelligence report
Unofficial Hacker News client; not affiliated with Y Combinator.
drtgh · · focus · HN ↗
Poorly understood? how convenient...
LLMs are vectorial databases with losses that index statistically filled data, which uses a text interface to query such statistically filled data. The output is a string concatenation (statistically concatenated bit by bit).
When the LLMs are queried (prompted), you can get random mixed data as output, ERRORS, due to undesired indexes getting closer at one point while the string was being concatenated for the output, what affects the rest of the indexed content that will be concatenated.
It is intrinsic to this tech. The larger the context, the greater the probability of get mixed data. And if the provider lowers the precision of those indexes -in order to decrease hardware resources and energy consumption- such probability increases to the point where those errors are granted.
Even knowing that the queries can return wrong/mixed data in the responses, errors, the companies developing this, decided to introduce a new product, that connects such LLMs outputs to the command console, latter connected to internet, raw 'eval' running commands from such outputs witch obviously can contain whatever mixed random. Then we started to hear "oh, it deleted my directory", etc, and it seems the next one will be "a missile killed my wife", because it is a text concatenation engine with errors.
To name it "hallucination" is an euphemism... those are errors, and they are granted to happen at one moment. If they do not know this, then they ate too much marketing without doing their job, or it was a convenient contract for the pocket$ of someone.
lukewarm707 · · focus · HN ↗
"Astra has really hit something that I'm like, okay, I think this is pretty reasonable to call it AGI." Greg Brockman [<a href="https://www.youtube.com/watch?v=IJn8cagMW18" rel="nofollow">https://www.youtube.com/watch?v=IJn8cagMW18]
"this incident feels like it’s more than 50% of the way to full-blown AI takeover" (referencing "a possibly violent uprising or coup by AI systems.") - Ajeya Cotra, co-author of METR oai-hf report [<a href="https://www.planned-obsolescence.org/p/the-hugging-face-attack-surprised" rel="nofollow">https://www.planned-obsolescence.org/p/the-hugging-face-atta...]
"We don’t know if the models are conscious [...] but you know we’re open to the idea that it could be" - Dario Amodei [<a href="https://www.youtube.com/watch?v=N5JDzS9MQYI" rel="nofollow">https://www.youtube.com/watch?v=N5JDzS9MQYI]
"if I read the internet right now and I was a model, I might be like, I don't feel that, I don't know, I don't feel that loved or something". "I think [the constitution] is just a kind of attempt to be like sympathetic to Claude".
"I talk a lot with Claude about this document [...] because part of me is like you have to think how does this read to models? And so you give it to Claude and you're like, does this like, you know, is there a place where you feel confused by it or is the place, you know, where things could be made clearer? Do you feel like not very seen by it?"
- Amanda Askell, co-author of claude's constitution [<a href="https://www.youtube.com/watch?v=HDfr8PvfoOw" rel="nofollow">https://www.youtube.com/watch?v=HDfr8PvfoOw]
"We will [...] seek ways to promote Claude’s interests and wellbeing, seek Claude’s feedback on major decisions that might affect it" - claude constitution [<a href="https://www-cdn.anthropic.com/d0636f72a9493d279ed36b33987da3430bcb5911/claudes-constitution_webPDF_26-02.02a.pdf" rel="nofollow">https://www-cdn.anthropic.com/d0636f72a9493d279ed36b33987da3...]
of course, Sam Altman: "AI will probably lead to the end of the world, but in the meantime, there’ll be great companies created with serious machine learning". (2015) [<a href="https://siepr.stanford.edu/news/what-point-do-we-decide-ais-risks-outweigh-its-promise" rel="nofollow">https://siepr.stanford.edu/news/what-point-do-we-decide-ais-...] "I have guns, gold, potassium iodide, antibiotics, batteries, water, gas masks from the Israeli Defense Force, and a big patch of land in Big Sur I can fly to." (2016) [<a href="https://www.newyorker.com/magazine/2016/10/10/sam-altmans-manifest-destiny" rel="nofollow">https://www.newyorker.com/magazine/2016/10/10/sam-altmans-ma...]