Thread

1. 2001zhaozhao · 1789596999 · focus · HN ↗

   Engineer: "HELP, our production DB is frozen on this query that worked fine before!"

   Infra: "Hmm, let's check... Well would you look at that, it seems like your LLM query planner usually works and produces fast queries, but this time when you changed a variable name to trigger query rebuild, it happened to hallucinate and miss an index, would you mind re-running the LLM a few times until you get a faster query?"

   1. malisper · 1789599271 · focus · HN ↗

      Funnily enough, you could replace "LLM query planner" with just "query planner" and this comment would still hold true

      1. Tanjreeve · 1789619599 · focus · HN ↗

         That bug is fixable and verifiable. The LLM you cross your fingers till the next time the same thing happens.

         1. egeozcan · 1789621811 · focus · HN ↗

            > fixable and verifiable

            By people with a specific skill set. LLMs generation can also be fixed and verified by people with a certain skill set, and non-deterministic computing doesn't automatically mean unpredictable. When people say that the LLMs are a black box, it means unpredictability in unknown situations.

            You do structured output, input validation, output validation, lower temperature, limit decisions, RL, etc. to increase predictability to near certainty. It's just statistics after all. Or you can as well generate the code to do the job.

            It's just that the required skill set is a different one to do those things, and unusual in the context of DB administration.

            1. tempfile · 1789639068 · focus · HN ↗

               None of the things you mention are guaranteed to increase the probability of correctness. You can run the LLM output through as many deterministic programs as you like, but "the query plan runs in acceptable time" is not something you can verify with such a tool. Nobody knows how the LLM does it, so they cannot know how to make the LLM do it better.

               1. egeozcan · 1789659427 · focus · HN ↗

                  > Nobody knows how the LLM does it

                  From a completely technical perspective, we have a rough idea how the LLMs work, and improving a system requires measuring outcomes and you don't necessarily need to understand the mechanism.

                  EXPLAIN ANALYZE against data that's similar in size to prod checks a query written by an LLM as good as anything we can write... but, yes, you're right, we still didn't solve the halting problem - neither the LLMs.

Unofficial Hacker News client; not affiliated with Y Combinator.