Thread

1. LiamPowell · 1790506742 · focus · HN ↗

   > yet what drives them is not well understood

   Presumably the fact that they're heavily trained to reply in this way? I don't know about the rest of the paper, but this part sticks out as a really odd claim unless I'm entirely misunderstanding this part.

   1. yu3zhou4 · 1790507016 · focus · HN ↗

      Thanks for pointing out, maybe I should be more explicit in the wording - I mean we don't fully know what drives the voice in LLMs. Models that are post trained as instruct models are expected to have the disclaimers, but what about base models (those that are trained on just a lot of text)? How do they talk about themselves? What happens when you strip off the chat template from instruct model's prompt? I hope the rest of the paper makes the questions clearer, but I will try to do better in the abstract next time, as you point out this sentence is kind ambiguous. Thank you!

      1. anonymous908213 · 1790508973 · focus · HN ↗

         > we don't fully know what drives the voice in LLMs

         Who is "we"? I, working in an LLM startup, know exactly what drives the base "voice" in the LLMs we train, because we have a process to select for it. OpenAI and Anthropic surely do too. Saying broadly that something is not well-understood in a scientific paper because it's not understood to casual observers is, uh, not very rigorous.

         > The strange thing is that the base models (before RLHF) use the "experiential" voice, even though they are not incentivized to do that.

         (Replying to your quote from another comment)

         This is a matter of the training material. We have trained models that do not do that. I'm not exactly divulging trade secrets here. It should be really, really obvious that if you train a model on chat-conversation-like patterns of speech it will infer probabilities for how to continue a textual sample that will differ from the probabilities learned from being trained on narration, prose, or informational patterns of speech, even without RLHF.

         1. david-gpu · 1790514114 · focus · HN ↗

            I feel your pain. It is really obvious.

Unofficial Hacker News client; not affiliated with Y Combinator.