FLAWED's Flaws and What This Means for Industry Research
Thread
Unofficial Hacker News client; not affiliated with Y Combinator.
FLAWED's Flaws and What This Means for Industry Research
Unofficial Hacker News client; not affiliated with Y Combinator.
darkamaul · · focus · HN ↗
- Aug 6: Off By 1, the security lab of 1Password publishes their blog post (current title: Why AI-generated vulnerability patches still require expert human review)
This article gets some specialized and mainstream press coverage
- September 15 : Trail of Bits publishes a rebuttal on their blog 1Password's AI patching benchmark is misleading
- September 22: Suha publishes the current blog post
The main idea in the rebuttal is that the headline number (26 percent only of the AI generated patches are flawless) is misleading because the experiments were not well designed.