Thread

US Military had close call after using AI for hallucinated intelligence report

519 points · 396 comments · realsarm

drtgh · 1789758284 · focus · HN ↗
> relatively poorly understood technology
Poorly understood? how convenient...
LLMs are vectorial databases with losses that index statistically filled data, which uses a text interface to query such statistically filled data. The output is a string concatenation (statistically concatenated bit by bit).
When the LLMs are queried (prompted), you can get random mixed data as output, ERRORS, due to undesired indexes getting closer at one point while the string was being concatenated for the output, what affects the rest of the indexed content that will be concatenated.
It is intrinsic to this tech. The larger the context, the greater the probability of get mixed data. And if the provider lowers the precision of those indexes -in order to decrease hardware resources and energy consumption- such probability increases to the point where those errors are granted.
Even knowing that the queries can return wrong/mixed data in the responses, errors, the companies developing this, decided to introduce a new product, that connects such LLMs outputs to the command console, latter connected to internet, raw 'eval' running commands from such outputs witch obviously can contain whatever mixed random. Then we started to hear "oh, it deleted my directory", etc, and it seems the next one will be "a missile killed my wife", because it is a text concatenation engine with errors.
To name it "hallucination" is an euphemism... those are errors, and they are granted to happen at one moment. If they do not know this, then they ate too much marketing without doing their job, or it was a convenient contract for the pocket$ of someone.
1. margalabargala · 1789758879 · focus · HN ↗
  I agree with most of your comment, but...
  > To name it "hallucination" is an euphemism... those are errors
  I find this and other "don't anthropomorphize the computer" statements incredibly unconvincing.
  People develop terms for things and language has always contained overloaded or "literally inaccurate" terms.
  An LLM can have "hallucinations" in the same way a modern computer program can have "bugs".
  1. usernomdeguerre · 1789759667 · focus · HN ↗
    I disagree, I think 'Hallucination' is a risk-shedding weasel-word. It's meant to shift blame away from the technology and its creator (multibillion dollar AI companies etc) in a way that doesn't hold those actors accountable or responsible for the outcomes.
    In any other software it would be an error, regression, bug. And in a human process it would be at ~least something someone would call 'bullshit'.
    margalabargala · 1789763596 · focus · HN ↗
    > In any other software it would be an error, regression, bug.
    How is "bug", literally an organism with a will of its own that you cannot control, any less of a weasel word?
    usernomdeguerre · 1789764846 · focus · HN ↗
    <a href="https://en.wikipedia.org/wiki/Bug_(engineering)#History" rel="nofollow">https://en.wikipedia.org/wiki/Bug_(engineering)#History
    But on reflection I don't disagree it was probably made for similar effect in the era of human software development. That sounds like it strengthens my point?
    margalabargala · 1789765323 · focus · HN ↗
    On the contrary I think it weakens your point.
    People don't consider "bug" a weasel word, to the point that you yourself held it up as an example of not being a weasel word, despite it being a willful, uncontrollable organism.
    I see no reason why "hallucination" won't become a similar piece of neutral jargon. It already is for many people, even if you're not (yet?) among them.
    usernomdeguerre · 1789767006 · focus · HN ↗
    It becomes a neutral term because we are practicioners (presumably?) who benefit from it and have thus let it become habitual. I already admitted it was weasely upon reflection.
    If you want this class of LLM error to also become habitual and neutral then fine, I don't, and I think many others don't.
    elzbardico · 1789871367 · focus · HN ↗
    You need to consult the etymology of the word bug in the context of computing. In the pre-historical times of vacuum-tube computers, with exposed conductors with potentials sometimes over 200v, a coachroach in the wrong place can definitely flip a few bits in the most benign scenario.
    margalabargala · 1789879562 · focus · HN ↗
    Right, yes, that's my point...you need to consult the comment you replied to.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.