Thread

1. drtgh · 1789758284 · focus · HN ↗

   > relatively poorly understood technology

   Poorly understood? how convenient...

   LLMs are vectorial databases with losses that index statistically filled data, which uses a text interface to query such statistically filled data. The output is a string concatenation (statistically concatenated bit by bit).

   When the LLMs are queried (prompted), you can get random mixed data as output, ERRORS, due to undesired indexes getting closer at one point while the string was being concatenated for the output, what affects the rest of the indexed content that will be concatenated.

   It is intrinsic to this tech. The larger the context, the greater the probability of get mixed data. And if the provider lowers the precision of those indexes -in order to decrease hardware resources and energy consumption- such probability increases to the point where those errors are granted.

   Even knowing that the queries can return wrong/mixed data in the responses, errors, the companies developing this, decided to introduce a new product, that connects such LLMs outputs to the command console, latter connected to internet, raw 'eval' running commands from such outputs witch obviously can contain whatever mixed random. Then we started to hear "oh, it deleted my directory", etc, and it seems the next one will be "a missile killed my wife", because it is a text concatenation engine with errors.

   To name it "hallucination" is an euphemism... those are errors, and they are granted to happen at one moment. If they do not know this, then they ate too much marketing without doing their job, or it was a convenient contract for the pocket$ of someone.

   1. pftburger · 1789759822 · focus · HN ↗

      It's not the first time we are encountering this issue. We've seen it in other autonomous systems. Trains are an older one, cars are a newer one. As you move out of the lower levels, the operator has a tendency to assume the system is increasingly more capable than it is. In trains, its so bad that they generate fake signals that the operator needs to respond to within a timeframe. I'd love to see this with implementations of other critical autonomous systems like this. Occasionally inject known errors into the system and expect the operator to catch them. If they don't, well... If it was a train driver I think we would fire them. If its an intelligence operative ordering a strike? :shrugs wearliy:

      1. pocksuppet · 1789761059 · focus · HN ↗

         note the airline industry has moved past firing pilots who make mistakes, since that turned out to be a recipe for more plane crashes, not less. Instead, they find out why the mistake happened, and fix it. In some cases, this involves firing the pilot. They do not do that by default.

         1. pftburger · 1789763405 · focus · HN ↗

            ACK on the going too draconian. 100% on the find the problem and fix it instead of blaming someone or something as a cheap solution

Unofficial Hacker News client; not affiliated with Y Combinator.