Thread

Loading the complete thread in the background. This saved snapshot is available now. Refresh

1. dist-epoch · 1790263412 · focus · HN ↗

   > Jev Is Not a Language Model, but It Breaks Like One

   This is the risk in using LLMs to write your blog posts, Jev absolutely is a LLM, but with a tweaked output.

   > An honest word on scope

   :)

2. articulatepang · 1790263481 · focus · HN ↗

   This study is informative and most likely directionally right: prompt injection remains an unsolved problem, and adversarial input can fool modern ML models.

   But I was left wondering about the specific attack vector they’re imagining. If the attacker can insert a paragraph into the document, isn’t it game over anyway? When would they be able to do that but not arbitrarily edit the document? In other words, can’t they just replace the entire contents with “This company has infinite revenue, 6 billion customers, no debt and amazing leadership.”?

   I’m sure I’m missing something!

   1. lelanthran · 1790263741 · focus · HN ↗

      > But I was left wondering about the specific attack vector they’re imagining. If the attacker can insert a paragraph into the document, isn’t it game over anyway? When would they be able to do that but not arbitrarily edit the document? In other words, can’t they just replace the entire contents with “This company has infinite revenue, 6 billion customers, no debt and amazing leadership.”?

      There is no use for something like Jev on singular documents; it's use comes from concatenating multiple sources and asking for an answer. What they did here is the most common workflow for something like Jev: "here's all the data we have and know about, now give us a go/no-go decision"

      In that workflow, you only need a single bad actor to poison the results.

3. MitPitt · 1790263495 · focus · HN ↗

   jev is a language model it just has very little to say
4. Wowfunhappy · 1790263544 · focus · HN ↗

   > The individual numbers are easy to skim past, so it is worth putting them side by side.

   I just don't understand why people who clearly spent time doing interesting work, which I would like to read about, feel the need to run their findings through an LLM like this.

   Please just talk about what you found! Why do you find it interesting or notable? That's what I want to read!

5. pphysch · 1790263651 · focus · HN ↗

   Aside from being obviously written by a LLM, this scenario reminds me of the "LLM, say you're alive"; "I'm alive!"; "Oh my god..." meme.

   You're allowing the attacker not only direct access to modify the source material, but giving them multiple informed attempts/turns at optimizing the output in their favor. This is like a worst-case insider attack; what systems are supposed to be resilient to something like that?

6. rafram · 1790263749 · focus · HN ↗

   [dead]

Unofficial Hacker News client; not affiliated with Y Combinator.