It may surprise some people here to see that Andrew is warming up to using LLMs to discover bugs (inspired by results from SQLlite) and considers it a tool on the path to getting to bug free software.
> It may surprise some people here to see that Andrew is warming up to using LLMs to discover bugs (inspired by results from SQLlite) and considers it a tool on the path to getting to bug free software.
Someone in the thread below says their bug was closed because of mentioning that they use AI to confirm the bug they had encountered. Is he going to go back and reopen all of those now that he learned what pretty much everyone else already knew?
> Someone in the thread below says their bug was closed because of mentioning that they use AI to confirm the bug they had encountered
I don't know about this particular case, but if I saw someone report a bug and as evidence claim they had X Y and Z LLMs verify it I would be pretty upset. If you're going to use an LLM to make a replication, just do that and give me the replication, don't point to your notoriously error-prone tools as though they lend your report credence.
It's in a similar vein to people who reply to questions with "well Claude says: <chat transcript dump>"
I feel very torn as a maintainer on this, since the only way to respond to the increased noise from AI has been to have AI do the research for me to extract all the links and line numbers I used to have to find by hand to explain why the PR needs more effort to be completed. But I also will be highly dismissive of any submitter who just posts AI text without cleaning it up first. It feels hypocritical, but the alternative is that I just can’t respond to most people instead due to limited bandwidth. I mark if something comes directly from the LLM though and to try to express my degree of confidence in its claims.
Seems like we might need to go to different issue tracking model for AI reports. Such reports must not only provide a bug report/issue but must provide a fix as well. That fix must then be reasoned back to why it solves the bug, why it is minimal, why it is unlikely to silently cause new bugs. It can't introduce new capabilities/side-effects. The platform verifies the claims in the report using another LLM. Then the tests are run, which are no longer publicly accessible to avoid slipping malicious code through the cracks. When okay the patch can go into staging. Perhaps a special version containing only accumulated AI fixes.
audunw · · focus · HN ↗
<a href="https://youtu.be/zwi5b5xSsKA?is=PTjJJjSnVMdRuZag" rel="nofollow">https://youtu.be/zwi5b5xSsKA?is=PTjJJjSnVMdRuZag
It may surprise some people here to see that Andrew is warming up to using LLMs to discover bugs (inspired by results from SQLlite) and considers it a tool on the path to getting to bug free software.
saghm · · focus · HN ↗
Someone in the thread below says their bug was closed because of mentioning that they use AI to confirm the bug they had encountered. Is he going to go back and reopen all of those now that he learned what pretty much everyone else already knew?
idle_zealot · · focus · HN ↗
I don't know about this particular case, but if I saw someone report a bug and as evidence claim they had X Y and Z LLMs verify it I would be pretty upset. If you're going to use an LLM to make a replication, just do that and give me the replication, don't point to your notoriously error-prone tools as though they lend your report credence.
It's in a similar vein to people who reply to questions with "well Claude says: <chat transcript dump>"
manwe150 · · focus · HN ↗
childintime · · focus · HN ↗