Thread

US Military had close call after using AI for hallucinated intelligence report

519 points · 396 comments · realsarm

drtgh · 1789758284 · focus · HN ↗
> relatively poorly understood technology
Poorly understood? how convenient...
LLMs are vectorial databases with losses that index statistically filled data, which uses a text interface to query such statistically filled data. The output is a string concatenation (statistically concatenated bit by bit).
When the LLMs are queried (prompted), you can get random mixed data as output, ERRORS, due to undesired indexes getting closer at one point while the string was being concatenated for the output, what affects the rest of the indexed content that will be concatenated.
It is intrinsic to this tech. The larger the context, the greater the probability of get mixed data. And if the provider lowers the precision of those indexes -in order to decrease hardware resources and energy consumption- such probability increases to the point where those errors are granted.
Even knowing that the queries can return wrong/mixed data in the responses, errors, the companies developing this, decided to introduce a new product, that connects such LLMs outputs to the command console, latter connected to internet, raw 'eval' running commands from such outputs witch obviously can contain whatever mixed random. Then we started to hear "oh, it deleted my directory", etc, and it seems the next one will be "a missile killed my wife", because it is a text concatenation engine with errors.
To name it "hallucination" is an euphemism... those are errors, and they are granted to happen at one moment. If they do not know this, then they ate too much marketing without doing their job, or it was a convenient contract for the pocket$ of someone.
1. margalabargala · 1789758879 · focus · HN ↗
  I agree with most of your comment, but...
  > To name it "hallucination" is an euphemism... those are errors
  I find this and other "don't anthropomorphize the computer" statements incredibly unconvincing.
  People develop terms for things and language has always contained overloaded or "literally inaccurate" terms.
  An LLM can have "hallucinations" in the same way a modern computer program can have "bugs".
  1. john_strinlai · 1789759239 · focus · HN ↗
    >language has always contained overloaded or "literally inaccurate" terms.
    "literally" is a great example of this, because it can also mean "not literally, but with emphasis".
  2. rrr_oh_man · 1789759239 · focus · HN ↗
    Every output an LLM creates is a hallucination.
  3. bix6 · 1789759332 · focus · HN ↗
    Knowingly causing errors is not forgivable whereas hallucinations sounds esoteric and moves blame away from the people who are knowingly causing errors. It’s marketing speak.
  4. piker · 1789759363 · focus · HN ↗
    I also agree with the parent, and I would also suggest "hallucination" is better than "error" which might imply an available deterministic correction. Hallucination makes it clear we're dealing with something different than an "error" or "bug".
    orwin · 1789767367 · focus · HN ↗
    I disagree, for me "error" is way, way more accurate than "hallucination", but i did take applied statistics in college and that might have influenced my vocabulary. Maybe that for the general public, "hallucination" is a better description, i might have biases in this case. But "error" is _definitely_ more accurate.
    If people want to call "drisse", "aussière", "balancine" and "ecoute" all as "boat ropes", they are correct. In english, i would certainly call them all "boat ropes" in any case, as i never needed to translate their names. It isn't the most accurate in my opinion, but as long as you're not working on them (or manning a boat in my analogy), who cares.
    narnarpapadaddy · 1789775051 · focus · HN ↗
    For humans hallucinations are a particular class of error, so I find hallucination more descriptive than either error or bug.
    I also think it’s relevant because a hallucinator often doesn’t recognize that the hallucination isn’t real. That’s more accurate for the LLM than either lie or confabulation, IMO. They algorithm is trained to produce strings of text that have semantic meaning based on some statistical likelihood of tokens appearing next to each other. The LLM algorithm is working as intended.
    Hallucinations are also often emergent from a particular state or situation, which reflects the generative aspect of LLMs.
    Hallucinations are sometimes resolved in humans by grounding exercises. “Touching grass.” The same is true for LLM hallucinations. Inaccuracies are found by cross-checking the output against an internet search or another LLM.
  5. Slow_Hand · 1789759413 · focus · HN ↗
    I prefer “confabulation”. It seems truer to what is happening:
    The LLM isn’t seeing something that’s not there, but deliberately making up _something_ so that it can return a response.
  6. [deleted] · 1789759510 · focus · HN ↗
    [deleted]
  7. usernomdeguerre · 1789759667 · focus · HN ↗
    I disagree, I think 'Hallucination' is a risk-shedding weasel-word. It's meant to shift blame away from the technology and its creator (multibillion dollar AI companies etc) in a way that doesn't hold those actors accountable or responsible for the outcomes.
    In any other software it would be an error, regression, bug. And in a human process it would be at ~least something someone would call 'bullshit'.
    vorticalbox · 1789760756 · focus · HN ↗
    I’m not sure either would is particularly good at describing what is happening.
    Error in implies something broke, which nothing broke the LLM did exactly what they where designed to do generate text based on a statistically likely bases.
    Hallucination Does really fit here either. It implies it’s experiencing something that is not there which it isn’t experiencing anything.
    Towaway69 · 1789761735 · focus · HN ↗
    Howabout: lied. The LLM lied indirectly (perhaps) but it made a claim that was false. Which is a lie.
    Humans lie and LLMs “hallucinate”? What gives. It’s an untruth that the LLM is selling for a truth, that’s lying in my books.
    And since we don’t know how or why the LLM works, we can’t even judge whether it explicitly lied or only because it didn’t know better.
    vorticalbox · 1789829187 · focus · HN ↗
    Lie implies it knows what it is saying to be false.
    t-3 · 1789762025 · focus · HN ↗
    Unexpected Result is perhaps a more accurate description.
    segsegsgsg · 1789760824 · focus · HN ↗
    error, regression, bug, bullshit are not weasel words, hallucination is a weasel word because why exactly? your argument is a weasel argument.
    usernomdeguerre · 1789762238 · focus · HN ↗
    I explained why, if you want me to engage i'll try but you're asking me to restate my position.
    s1artibartfast · 1789774682 · focus · HN ↗
    You made a lot of clams about how it intentionally shifts blame, but none of them are supported.
    Do you really think accountability would be meaningfully different had they been called bugs?
    Why are you so confident it is intentional? My understanding of the history is that it was a technical term among researchers long before it had any public mind share. It's popular because it's and intuitive for most people, not because there was a concerted effort hooked up by some PR and legal team.
    e5yyey · 1789777604 · focus · HN ↗
    I disagree, I think 'Bug' is a risk-shedding weasel-word. It's meant to shift blame away from the technology and its creator (multibillion dollar AI companies etc) in a way that doesn't hold those actors accountable or responsible for the outcomes. In any other software it would be an error, regression, hallucination. And in a human process it would be at ~least something someone would call 'bullshit'.
    leonidasrup · 1789761978 · focus · HN ↗
    Using the term "Hallucination" makes it sound less problematic, less impactfull for user.
    They should have used the term "error". For example in statistics, there many kinds of errors, discretization error, prediction error, sampling error, ...
    <a href="https://www.statisticshowto.com/errors-in-statistics/" rel="nofollow">https://www.statisticshowto.com/errors-in-statistics/
    lukan · 1789762303 · focus · HN ↗
    I don't know, but all humans make errors, but if some humans are known to hallucinate, you don't let them do important things unsupervised. So error sounds actually less problematic to me.
    verdverm · 1789763311 · focus · HN ↗
    We do a surprising amount of "hallucinations" without the extreme version of hallucinations. We assemble things we sort of remember into incorrect statements all the time. I'm sure every one of us has been corrected for misremembering something or stating something based on misremembered facts (plague of clickbait headlines).
    This is more or less how I see the LLM output, but as a path finding exercise over next-token probability graphs. This is (i.e.) why they are trained to use phrases like "wait but" or "actually", these words even out the probability of different paths, giving them their ability to "consider" different solutions.
    leonidasrup · 1789763372 · focus · HN ↗
    Because so few humans hallucinate, many imagine hallucinations like dreams and dreams are mostly harmless.
    plant-ian · 1789762160 · focus · HN ↗
    Totally weasel words in this time frame. I think in 10 years after everyone has a better understanding of what we are dealing with these weasel words would maybe make sense. Right now it seems more sensible to deem this at best a false positive, or glitch, or if it must be anthropomorphized a screw up or a f' up. I don't think the llms are dehydrated. Although that's funny on another level. Edited: to be less abrasive
    margalabargala · 1789763596 · focus · HN ↗
    > In any other software it would be an error, regression, bug.
    How is "bug", literally an organism with a will of its own that you cannot control, any less of a weasel word?
    usernomdeguerre · 1789764846 · focus · HN ↗
    <a href="https://en.wikipedia.org/wiki/Bug_(engineering)#History" rel="nofollow">https://en.wikipedia.org/wiki/Bug_(engineering)#History
    But on reflection I don't disagree it was probably made for similar effect in the era of human software development. That sounds like it strengthens my point?
    margalabargala · 1789765323 · focus · HN ↗
    On the contrary I think it weakens your point.
    People don't consider "bug" a weasel word, to the point that you yourself held it up as an example of not being a weasel word, despite it being a willful, uncontrollable organism.
    I see no reason why "hallucination" won't become a similar piece of neutral jargon. It already is for many people, even if you're not (yet?) among them.
    usernomdeguerre · 1789767006 · focus · HN ↗
    It becomes a neutral term because we are practicioners (presumably?) who benefit from it and have thus let it become habitual. I already admitted it was weasely upon reflection.
    If you want this class of LLM error to also become habitual and neutral then fine, I don't, and I think many others don't.
    elzbardico · 1789871367 · focus · HN ↗
    You need to consult the etymology of the word bug in the context of computing. In the pre-historical times of vacuum-tube computers, with exposed conductors with potentials sometimes over 200v, a coachroach in the wrong place can definitely flip a few bits in the most benign scenario.
    margalabargala · 1789879562 · focus · HN ↗
    Right, yes, that's my point...you need to consult the comment you replied to.
  8. Rebuff5007 · 1789759675 · focus · HN ↗
    Note that "bug" came from an actual moth in a computer: <a href="https://www.computerhistory.org/tdih/september/9/" rel="nofollow">https://www.computerhistory.org/tdih/september/9/
    john_strinlai · 1789759998 · focus · HN ↗
    neat part of history, but i dont think that's what that says.
    the last sentence starts with "Originating with Thomas Edison in the 1800s, the term “bug” is still used [...]", and there would be no reason to use the word "actual" in the sentence "First _actual_ case of bug being found" if it was the origin of the term.
    my clanker found this: <a href="https://spectrum.ieee.org/did-you-know-edison-coined-the-term-bug" rel="nofollow">https://spectrum.ieee.org/did-you-know-edison-coined-the-ter...
    "The use of “bug” to describe a flaw in the design or operation of a technical system dates back to Thomas Edison. He coined the phrase 140 years ago to describe technical problems during the process of innovation."
    the moth seems to be a popular misconception, though, given that the article starts with "Ask someone to identify the first computer bug, and he or she might mention computer programmer Grace Hopper and the dead moth found in a relay of Harvard University’s Mark II electromechanical computer in 1947"
    Sharlin · 1789760059 · focus · HN ↗
    I believe the word was already in use to denote a malfunction of any sort of machine or device. As such this was a bug (insect) that caused a bug (glitch); it was punny already in 1947.
  9. [deleted] · 1789759776 · focus · HN ↗
    [deleted]
  10. cmiles74 · 1789759985 · focus · HN ↗
    Anthropomorphizing the tool led directly to this problem, where we nearly started a war with China.
  11. antonvs · 1789760078 · focus · HN ↗
    The term “hallucination” is a projection of inappropriate expectations onto a program. We know that LLMs are not “truth machines,” but we really want them to be. So when they produce a result that happens not to match external reality - which, it should be noted, LLMs don’t generally have access to - we call it an hallucination.
    “Bugs” are completely different. With bugs, we have a clear specification and we have a program that’s supposed to meet that specification. If it doesn’t, we say the program has bugs, and if it’s important enough we can change the program to eliminate the bugs.
    You can try to apply similar logic to LLMs, but you’d be making a category error, and you’ll fail to get the results you want in general. It’s not the same thing at all.
    If anything, the concept of an LLM hallucination is a bug in human understanding of LLMs.
    jyounker · 1789760277 · focus · HN ↗
    The word "confabulation" is much more precise and appropriate than "hallucination". We should use it instead.
    orwin · 1789767567 · focus · HN ↗
    Yes, but when a statistical model give you an erroneous result, you call the output an error, not a bug. I think error is more appropriate here. The error can be a sampling error, an inference error, or yes, a software error (or bug)
  12. jyounker · 1789760119 · focus · HN ↗
    From the point of view of the system, this is an error. It is incorrect information.
    The term "hallucination" feels much more like anthropomorphizing. The word hallucination implies an aberrant condition. A much better term would be "confabulation".
    You don't trust things or individuals that confabulate.
    ChrisLTD · 1789760340 · focus · HN ↗
    a filling in of gaps in memory through the creation of false memories by an individual who is affected with a memory disorder (as Korsakoff syndrome) and is unaware that the fabricated memories are inaccurate and false
    vs.
    a sensory perception (such as a visual image or a sound) that occurs in the absence of an actual external stimulus and usually arises from neurological disturbance (such as that associated with delirium tremens, schizophrenia, Parkinson's disease, or narcolepsy) or in response to drugs (such as LSD or phencyclidine)
    pocksuppet · 1789760988 · focus · HN ↗
    So call them confabulations
    gizajob · 1789761221 · focus · HN ↗
    Confabulation is also a symptom very prevalent in forms of narcissism and psychopathy. Gaps in understanding or perception are back-filled by confabulating so as to not risk the omnipotence of the confabulator.
    Up to the reader to decide whether this phenomenon is found in the statements of AI leadership or not.
    t-3 · 1789762904 · focus · HN ↗
    It's also something people tend to do when thinking, daydreaming, trying to solve problems, etc. We just usually don't fall for our own bullshit.
    LLMs don't either. They just give output in response to input. If the output is wrong that's because the model is wrong, not because the LLM is doing anything it's not supposed to be. It just wasn't built well enough to produce the expected result.
    reichstein · 1789762071 · focus · HN ↗
    > From the point of view of the system, this is an error. It is incorrect information.
    Which system?
    The LLM has no _concept_ of "correct". It emits output, based on its input and internal state.
    If that output happens to be correlated with reality, then it's useful. If it doesn't, and this is not a creative exercise, it's not useful.
    Everything an LLM emits is equal to it. It's all confabulation - this it says that is not based on facts, because it also has no concept of fact. Value judgements you make about the output is all you.
    "Confabulation" is no less anthropomorphizing than "hallucination".
  13. order-matters · 1789760840 · focus · HN ↗
    hallucination is common language for these models at this point which describes a particular type of error where the models make shit up.
    it is noticeable that the form of this particular error holds a similar shape to what is casually described as hallucinations, in that there is a generated content that often appears to blend naturally into the rest of the output but is false.
    the term hallucination often invokes a caution that this particular type of error may be influential and believable and is particularly dangerous
  14. nonethewiser · 1789761086 · focus · HN ↗
    Sure… but being wrong doesnt necessarily make it a hallucination:
    >It was only just before the planned operation that officials dug deeper into the report put together by a special operations command analyst and found it had been generated with the help of artificial intelligence (AI) — and that a chatbot the analyst had used inaccurately identified the material the ship was carrying. CNN was not able to learn what the misidentified cargo was.
  15. 0x20cowboy · 1789762431 · focus · HN ↗
    It’s not an error or a hallucinations it works correctly every time, and statistically picks the next token for the sequence.
    Retuning inf or crashing would be an error.
    If you want to ascribe some kind of meaning to the tokens, then maybe the training data was insufficient to predict the token in the sequence you wanted, but it doesn’t predict the next “fact”, and it doesn’t “think” it predicts the next token.
    margalabargala · 1789765744 · focus · HN ↗
    LLMs are useful because (and inasmuch as) their output generally reflects coherent reality.
    And their output does, usually, reflect coherent reality.
    The problem class of "properly operating program emits output incompatible with coherent reality" is something that is reasonable to put under its own term, considering it's a new class of problem.
    In other words, I think you misunderstand the language others are using. "Hallucination" doesn't refer to an "error" in the sense that crashing is an error, it refers to a situation in the problem class above, which is compatible with it working correctly every time.
    > it doesn’t “think” it predicts the next token.
    I never said it did. And I agree that LLMs don't "think". That said I am fully willing to go to bat arguing "thinking tokens" is a perfectly fine piece of jargon. Metaphors are completely acceptable parts of language, and contextual meaning is something grasped by everyone including the pedants who pretend not to.
    0x20cowboy · 1789845460 · focus · HN ↗
    > I think you misunderstand the language others are using. "Hallucination" doesn't refer to an "error" in the sense that crashing is an error, it refers to a situation in the problem class above, which is compatible with it working correctly every time.
    I do not misunderstand, I think maybe you do. You think there is a proper next word selection based on logic or meaning and there for the model selected the wrong one - it hallucinated.
    I am saying the model has no concept if anything other than the probability of select a token which is not based in any logic so it is working properly- it only works on numbers.
    It is random chance that it is ever correct, not that it is correct often and messed up this one time.
  16. jacquesm · 1789763460 · focus · HN ↗
    OpenAI calls them 'mistakes'. But that's just a fig leaf.
    Google does it too: "AI responses may include mistakes."
    Mistakes have an air of innocence. But these are not mistakes, they are purposefully releasing stuff that they know is broken, they just don't know when it is broken...
    s1artibartfast · 1789775267 · focus · HN ↗
    Broken is a little hyperbolic.
    Lots of totally viable essential or everyday products are not perfectly reliable.
    Medicine is not 100% reliable. My car isn't 100% reliable. Hell, my phone and cellular network are not 100% reliable.
    They are all still extremely useful tools. I might want them to be even better, but that's a cost versus quality question.
  17. DanHulton · 1789781539 · focus · HN ↗
    I’ll even go one step further - I don’t even like saying “Artificial Intelligence”. I think even that anthropomorphizes the machine too much. I prefer “Simulated Intelligence”, and I feel like that describes what is going on much better.
    We are, through this process, simulating intelligence. These models aren’t intelligent, but they can simulate it. Every simulation has a degree of fidelity, and we’re not at 100%, not even with the top models. When you think about it in those terms, I find it becomes a lot easier to keep their limitations in mind. Additionally, it becomes easier to remember that this is an algorithm that you are running, and are responsible for, not another being that you can ascribe blame to.
    hardbass · 1790159633 · focus · HN ↗
    What is the difference between simulating eg factoring a cryptographic number that is the product of two large primes and "actually" factoring that number?
    DanHulton · 1790289821 · focus · HN ↗
    That is just one thing an intelligent actor can do. That is not intelligence.
  18. elzbardico · 1789871213 · focus · HN ↗
    They are neither hallucinations or errors. They are just generations that happen to not be grounded in facts from the real world.
    margalabargala · 1789879644 · focus · HN ↗
    > They are just generations that happen to not be grounded in facts from the real world.
    Right, yes, and "hallucination" is the term that a critical mass of people have chosen to use.as a shorthand so that we don't have to write out "generations that happen to not be grounded in facts from the real world" every time it happens.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.