Thread

1. willmadden · 1790430334 · focus · HN ↗

   Watermarking sounds like a good idea, but it's not. Token drift from watermarking will degrade the quality of outputs and could allow clever people to circumvent guardrails.

   You should assume all text is AI generated. If you want to "test" someone at school or during an interview, have them write with a pencil and paper.

   1. skybrian · 1790432451 · focus · HN ↗

      Changing a random seed could either improve or degrade the output. In theory, better and worse outputs should be equally probable, depending on your luck.

      1. riskable · 1790435663 · focus · HN ↗

         They tested this in the research: They found that the statistical likelihood of bad tool calling was higher with watermarking compared to random seeds. At least, that was my takeaway (I read the whole thing).

         This is because SynthID and similar watermarking methods for LLMs don't just change the random seed. They take additional steps (that I have yet to read about) in order to detect when someone changes a few words of the output, trying to remove the watermark.

         The other takeaway is that just by knowing a watermark is being applied gives an attacker an advantage in working around safety features because then they know the output isn't based on true randomness and can take advantage of that in a similar fashion to how breaking cryptography becomes easier when the RNG isn't truly random.

Unofficial Hacker News client; not affiliated with Y Combinator.