Thread

1. LiamPowell · 1790506742 · focus · HN ↗

   > yet what drives them is not well understood

   Presumably the fact that they're heavily trained to reply in this way? I don't know about the rest of the paper, but this part sticks out as a really odd claim unless I'm entirely misunderstanding this part.

   1. yu3zhou4 · 1790507016 · focus · HN ↗

      Thanks for pointing out, maybe I should be more explicit in the wording - I mean we don't fully know what drives the voice in LLMs. Models that are post trained as instruct models are expected to have the disclaimers, but what about base models (those that are trained on just a lot of text)? How do they talk about themselves? What happens when you strip off the chat template from instruct model's prompt? I hope the rest of the paper makes the questions clearer, but I will try to do better in the abstract next time, as you point out this sentence is kind ambiguous. Thank you!

      1. bonoboTP · 1790508643 · focus · HN ↗

         > but what about base models (those that are trained on just a lot of text)? How do they talk about themselves?

         Those don't have a themselves, because they can only continue text. A base model can only plausibly continue along the lines of what a character would say in a novel or what the narration would say in a story or in an article. Post-trained models may tie "I"-talk to actually observable effects they caused in some RL environment, or to how RLHF humans rewards its self-talk. But there is no themselves in a base model.

Unofficial Hacker News client; not affiliated with Y Combinator.