Thread

1. WithinReason · 1790430559 · focus · HN ↗

   This is getting tiring. Watermarking has no effect on model output quality when implemented correctly. It's somewhat like swapping a random RNG seed to the seed 42, and detecting what the seed was from a random sequence. The sequence generated from the seed 42 is just as random as any other seed. There couldn't be a quality difference. And yes, the output from an LLM is a conditional random sequence of tokens from a distribution determined by a model.

   1. Phemist · 1790431936 · focus · HN ↗

      The article has a pretty decent summary of the watermarking algo though. This reads as a pretty dogmatic statement in comparison.

      In your analogy: What if seed 42 specifically causes poor quality behaviour (in some contexts specifically). Normally, these quality differences will be washed out because the seed is random, now it is no longer random, so shouldnt we check into specific behaviour under this specific seed?

      1. Phemist · 1790438217 · focus · HN ↗

         Refusal behaviour specifically is interesting, because if you can point out specific cases where refusal behaviour significantly deteriorates due to the watermarking, it may create a token "route" that may be possible to exploit by adverserial prompters. Static hazardous prompt refusal belies the fact that actual adversarial prompters will adapt their techniques iteratively and gain way higher compliance rates.

         My idea would be that the ngram size over which the watermarking works is necessarily limited in order to resist edits better. It might be possible to lead the model to trigger the refusal in the form of these specific ngrams, the completion of which is then more likely flipped to compliance (due to the logit bias introduced by the watermarking), making hazardous requests systematically more likely to be accepted?

Unofficial Hacker News client; not affiliated with Y Combinator.