Thread

US Military had close call after using AI for hallucinated intelligence report

519 points · 396 comments · realsarm

drtgh · 1789758284 · focus · HN ↗
> relatively poorly understood technology
Poorly understood? how convenient...
LLMs are vectorial databases with losses that index statistically filled data, which uses a text interface to query such statistically filled data. The output is a string concatenation (statistically concatenated bit by bit).
When the LLMs are queried (prompted), you can get random mixed data as output, ERRORS, due to undesired indexes getting closer at one point while the string was being concatenated for the output, what affects the rest of the indexed content that will be concatenated.
It is intrinsic to this tech. The larger the context, the greater the probability of get mixed data. And if the provider lowers the precision of those indexes -in order to decrease hardware resources and energy consumption- such probability increases to the point where those errors are granted.
Even knowing that the queries can return wrong/mixed data in the responses, errors, the companies developing this, decided to introduce a new product, that connects such LLMs outputs to the command console, latter connected to internet, raw 'eval' running commands from such outputs witch obviously can contain whatever mixed random. Then we started to hear "oh, it deleted my directory", etc, and it seems the next one will be "a missile killed my wife", because it is a text concatenation engine with errors.
To name it "hallucination" is an euphemism... those are errors, and they are granted to happen at one moment. If they do not know this, then they ate too much marketing without doing their job, or it was a convenient contract for the pocket$ of someone.
1. theptip · 1789760419 · focus · HN ↗
  > LLMs are vectorial databases
  You use a bunch of technical-sounding words here to make it sound like you understand. But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.
  Almost nothing is understood about the actual representations used for nontrivial concepts, decision algorithms, etc.
  If you look at the field of mechanistic interpretability, compared to “GOFAI” like learned decision trees, an LLM is completely opaque.
  1. Betelbuddy · 1789761798 · focus · HN ↗
    >> an LLM is completely opaque
    And despite that, although they are not like that in practice as there are too many uncontrolled variables, with temperature at zero, for the same input they produce always the same reply.
    chrisjj · 1789765795 · focus · HN ↗
    > with temperature at zero, for the same input they produce always the same reply.
    Nonsense.
    <a href="https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/" rel="nofollow">https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
    coldtea · 1789768512 · focus · HN ↗
    BS. Run them sequentially on a single core, and without fancy speedups enabled, and they do. The algorithm is determinstic. Any non-determinism present with 0 temperature it's not some mysterious LLM-inherent property, but something that can be seen in any large program taking advantage of multi-core, floating point, and other CPU-based parallelism optimization.
    chrisjj · 1789823231 · focus · HN ↗
    > Any non-determinism present with 0 temperature it's not some mysterious LLM-inherent property,
    Of course it's not mysterious. It is well understood by all who are aware of the fundamental unreliability of all major LLMs in general use today.
    coldtea · 1789866011 · focus · HN ↗
    The phrasing still makes it sound like it's due to the LLM. It's not. It's do to compiler and CPU, and OS optimizations, and a non-LLM program could suffer the same just as well.
    chrisjj · 1789894630 · focus · HN ↗
    This unreliability is entirely due to the implentation of the LLM. Yes, any other program implemented equally carelessly could suffer the same. But you'd be hard pressed to find any as unreliable as a typical LLM.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.