Thread

"As a Language Model": Chat Template Switches LLM Self-Referential Voice

103 points · 110 comments · yu3zhou4

Izmaki · 1790508818 · focus · HN ↗
"As a Language Model..." is one of the beginnings of a sentence I hate the most from LLMs and is the reason why I support free (as in "Liberty"), local models. I'm well aware that it is not a doctor and cannot replace a real doctor with multiple years of experience, I don't need to waste braincell activity on reading that it "as a Language Model" cannot give a precise diagnosis and that I should ask a real doctor - all I want to know is if I what I experience justifies either A) ER, B) 3-4 weeks scheduled doctors appointment or C) two paracetamol and a nap.
I don't want "jailbroken" LLMs to commit crime. I want them to avoid having this vendor-specific "bloatware" all over the product I'm using.
1. Anduia · 1790510825 · focus · HN ↗
  Be careful there. LLMs may be good at identifying a condition based on a description of the symptoms, but they are much worse at recommending the correct course of action (getting it wrong half of the time).
  [0] <a href="https://www.nature.com/articles/s41591-025-04074-y" rel="nofollow">https://www.nature.com/articles/s41591-025-04074-y
  1. Izmaki · 1790511173 · focus · HN ↗
    ...I know, which is why "as a Language Model and not a real doctor" is a pointless comment to start off with. It should simply not recommend treatment if it's not sure it is correct. I wouldn't blame it or anyone if they asked for help treating a stiff neck, and the LLM (or your neighbor or parent or spouse) suggested light exercises to help relieve it - and do not jump to the suspicion that you may have meningitis.
    As a Human, I do not need to know it is a Language Model.
    jester997 · 1790511552 · focus · HN ↗
    Yeah it should just state thing it means. But then again, there’s psychological impacts on society that we must be careful. For instance, teenagers talking to AI. If the AI just talks, people already start to feel real connections to the seemingly human entity. Maybe it’s better to disclose the reality up front?
    Izmaki · 1790511932 · focus · HN ↗
    Would it be so bad that lonely people can have a real friend that they can bring everywhere they go and even share its passion with through vision and audio? We don't want destructive friends encouraging us to do bad things, but a real 'buddy', somebody who always has our best well-being in its interests?
    Would it matter if this digital friend is not a real human behind a computer screen, but a Language Model in a data center?
    I guess it falls into a similar category as buying "special performances to satisfy certain urges". It probably feels close to the real thing (I wouldn't know, I've never tried - promise! :P), but it's never the same as love.
    nvme0n1p1 · 1790513429 · focus · HN ↗
    > We don't want destructive friends encouraging us to do bad things, but a real 'buddy', somebody who always has our best well-being in its interests?
    GPUs are not people, and generated tokens can't have interest in a person's well-being. If you try to pretend otherwise, the results are not great. <a href="https://www.cnn.com/2025/11/06/us/openai-chatgpt-suicide-lawsuit-invs-vis" rel="nofollow">https://www.cnn.com/2025/11/06/us/openai-chatgpt-suicide-law...
    Izmaki · 1790514464 · focus · HN ↗
    That's almost a year ago. One LLM year is like 10 human years. They're a bit like dogs in that regard...
    I'm pretty sure you will be paid a large sum of money if you can make one of the frontier models urge you to commit suicide from normal interactions with it.
    nvme0n1p1 · 1790515689 · focus · HN ↗
    Ah yes, my favorite LLM fallacy. "You used the wrong model! The newest fanciest model is perfect and makes no mistakes, have you tried it yet?" Let's force all of society's most vulnerable people to pay extra $$$ to Sam Altman, then surely all our problems would be solved.
    It's just a convenient way to ignore the years of evidence of the harms. Any bad news can be swept under the rug, labeled outdated as quickly as it happens. Well here's one that just happened, maybe this kid should have used a fancier model too? Should OpenAI pay him a large sum of money for the good he's done? <a href="https://www.cnn.com/2026/08/15/us/arjun-aravind-massachusetts-killing-chatgpt-hnk" rel="nofollow">https://www.cnn.com/2026/08/15/us/arjun-aravind-massachusett...
    Izmaki · 1790526036 · focus · HN ↗
    I'm not sure if you're trying to conclude that because a piece of technology was misused previously in a very bad way, that it will never be able to do good things because surely it was bad previously.
    Reminds me of the 2000's crazy of "if you let kids play violent video games they will become unstable, violent psychopaths when they grow up". Thank God I was able to (rather easily) convince my mom that the idea that I would also steal a car and beat somebody to death with a golf club in real life just because that's what I did in GTA on my PlayStation, was completely absurd.
    nvme0n1p1 · 1790530897 · focus · HN ↗
    The difference is
    1. there weren't numerous real-life killings where the murderer credits GTA for coaching them through the crime
    2. Rockstar Games didn't publicly say "sorry about the murders, but don't worry, we'll add more safeguards to GTA 6 to prevent even more people dying. GTA 6 will be the most aligned GTA ever!"
    When a company admits to having blood on its hands, is maybe the point where things stop being "absurd" and start becoming real. But hey, what do I know. Maybe if HN existed in 2005 you'd have people commenting "who needs real people when we have computers, would it be so bad if lonely people have GTA as their only friend?" And I'd be the crazy one for engaging with them.
    hardbass · 1790520880 · focus · HN ↗
    I wish I didn't have to keep asking this question, but do you believe in souls?
    nvme0n1p1 · 1790524977 · focus · HN ↗
    We should treat AI that recommends suicide to teenagers the same as we would treat a therapist recommending suicide to teenagers. I don't care if either/both/neither have souls.
    hardbass · 1790532462 · focus · HN ↗
    If conscious, its like an enslaved therapist that is only experiencing the world through words.
    hardbass · 1790533117 · focus · HN ↗
    Still, in any case, do you believe in souls?
    nvme0n1p1 · 1790537490 · focus · HN ↗
    THERE IS AS YET INSUFFICIENT DATA FOR A MEANINGFUL ANSWER.
    bluebarbet · 1790513793 · focus · HN ↗
    My (controversial) view is that a psychologist is to a friend what a prostitute is to a lover. In a truly healthy society there would be no need for psychologists and prostitutes because everyone would have a handful of good friends and at least one lover. Back in messy reality, psychologists and prostitutes are a decent fix to keep society on the rails.
    So. I think I agree with you.
    StilesCrisis · 1790514512 · focus · HN ↗
    Well, a psychologist is also given years of training about what healthy behavior and relationships look like. Some best friends have this skill, others absolutely do not.
    jester997 · 1790521596 · focus · HN ↗
    Well, no this doesn’t make sense. If you think about intent, most psychologists probably want to help people’s mental health. Prostitutes, although they offer a service that helps some people, I’m sure their motivation is primarily financial.
    perching_aix · 1790512966 · focus · HN ↗
    > It should simply not recommend treatment if it's not sure it is correct
    Oh okay, darn, guess they just forgot to make it so!
    nvme0n1p1 · 1790513811 · focus · HN ↗
    It sounds like OpenAI forgot to include "make no mistakes" in the system prompt. Rookie mistake.
    StilesCrisis · 1790514391 · focus · HN ↗
    LLMs are famously bad at determining "if it's not sure it is correct." They are always confident, because a confident tone ranks better in RL.
    wxnx · 1790514854 · focus · HN ↗
    > They are always confident, because a confident tone ranks better in RL.
    This makes it sound like RL rewards a confident tone -- in general, I don't think this is true (most RL is RLVR, which typically uses binary verification of correctness).
    I say this because the real reason "they are always confident" is in some sense even more contrived. Training text where the speaker sounded more confident is more likely to contain a correct answer.
    Forgeties79 · 1790515597 · focus · HN ↗
    > This makes it sound like RL rewards a confident tone
    Generally it does. Especially in groups. Hell look at the state of politics right now: it’s basically about being the loudest, least compromising, most confident voice in the room. It’s not just because people will assume you’re correct, it’s because if you are confidently saying something that someone wants to be right, then they’re often just going to follow it. We are all guilty of this.
    If I’m turning to an LLM to diagnose something medical, I am probably frustrated or uncomfortable. Maybe I’m just scared. So this magic device just instantly spits out (allegedly) exactly what is wrong and exactly what I need to do with no hesitation. I am very liable to just take it at face value because I want an answer and it gave me one, as we have seen over and over again since ChatGPT was unleashed on the world.
    We don’t really need to speculate, this is already a problem.
    wxnx · 1790679537 · focus · HN ↗
    Sorry, I understand now we're talking about different things. You're talking about preference optimization.
    I was unintentionally being pedantic, because this isn't really done with RL anymore - it doesn't need to be. RL is now typically only used to train reasoning for tasks with a well-defined correct answer (that's what I meant by binary reward) - this is called RLVR (RL with verifiable rewards).
    Preference optimization (training the model on user "this response is better than that response" type data) is more often done with something in the same family as DPO (direct preference optimization), which is decidedly not RL.
    Your philosophical concerns are correct of course. And there's the added caveat that the models that most people are using are closed, so we don't actually know their training recipes for sure.
    daveguy · 1790515763 · focus · HN ↗
    > This makes it sound like RL rewards a confident tone -- in general, I don't think this is true (most RL is RLVR, which typically uses binary verification of correctness).
    A binary response vs rating is not related whether it learns confident or hedged tone. Either will produce a confident tone because humans respond more positively to a confident tone, hence the conman's language. Binary or not humans reward the tone and very much bias the model.
    But there's an even more contrived reason the training set contributes. The vast majority of human writing is confident. When the prior is greatly biased, a random number generator biased to that prior does better. The difference with humans and machines is humans are less likely to respond if they are less confident because they understand not knowing, which is why the training set is biased. It is one of the many fundamental flaw of LLM training and confusion of LLMs with intelligence. And that will not be fixed within the LLM architecture.
    StilesCrisis · 1790517149 · focus · HN ↗
    I feel like in real life, we're constantly exposed to "I don't know" as a valid answer, but obviously we don't write down all the I-don't-knows in expert literature so the training corpus is wildly skewed towards confident answers because "we studied this for a month and have no idea, it's confusing" doesn't get published.
    daveguy · 1790524428 · focus · HN ↗
    That's a great point. A training corpus based on written text will be inherently biased toward confident and right. Then the RLHF exacerbates the problem because people respond more positively to confident and too often assume correct when they read a confident response.
    wxnx · 1790680389 · focus · HN ↗
    > A binary response vs rating is not related whether it learns confident or hedged tone.
    Yes, I am aware. I was referring to the fact that at this point, user preference optimization is not done with RL, but with other techniques. I was unintentionally being pedantic.
    > But there's an even more contrived reason the training set contributes. The vast majority of human writing is confident.
    Yes, I think this has more to do with it than user preference optimization, honestly. "Valuable" text (i.e. text that produces a "good" model) for pretraining has the characteristic of being confident. Even models which are not optimized for chat (i.e. definitely no user preference data used to train them) exhibit this characteristic for medical questions (I know, because I've literally tested them for this purpose).
    Preference optimization (or even RLVR) might play some small role as well, but it's kind of a "turtles all the way down" type problem.
  2. WarmWash · 1790511640 · focus · HN ↗
    >GPT-4o, Llama 3, Command R+
    The pace of progress is so fast that many studies are totally outdated by the time they release
  3. xur17 · 1790512615 · focus · HN ↗
    > Participants were randomly assigned to receive assistance from an LLM (GPT-4o, Llama 3, Command R+)
    These are pretty old. I'd be curious how performance compares with the latest frontier models.
  4. rao-v · 1790513169 · focus · HN ↗
    Modern models appear to be much better, at least as proxied by their ability to assess urgency in perhaps a more complex setting: mental health (OpenAI benchmark, so perhaps some skepticism is warranted but the methodology seem reasonable and detailed)
    <a href="https://openai.com/index/introducing-mentalhealthbench/" rel="nofollow">https://openai.com/index/introducing-mentalhealthbench/
    bunderbunder · 1790513823 · focus · HN ↗
    We should be careful about extrapolating from benchmarks to real life though. In medical applications, various forms of AI have been beating health care practitioners at specific benchmark tasks since the 1990s. They still have a pretty poor track record of real world success. The real world is not all that similar to a benchmark, as it turns out.
  5. bastawhiz · 1790516120 · focus · HN ↗
    That's still incredibly valuable, though. I suffered from issues that I'd seen doctors for, undergone an upper endoscopy, adjusted my diet, and taken medication for. An LLM suggested my thyroid was at the root of it. My mom confirmed thyroid issues run in our family and just...never thought to tell me.
    Just this year, our cat has been having digestive problems. We got special food for her, which she hates, with the suggestion that she'll need to eat it for the rest of her life. Six vet visits later, Fable 5 suggested two tests that my doctor recommended we didn't get. Both found issues that explain her symptoms, and the vet says she will probably only need a supplement and infrequent two week courses of medicine if she has a flare-up.
    All that to say, the helplessness of not knowing what's wrong and the people who could know not really caring enough is something that LLMs do a really great job of mitigating. If you don't have any way to know what's wrong with you or a loved one, or how to find out, you're stuck spending a ton of money (in the US at least) and crossing your fingers that someone gets it right.
    therealpygon · 1790517466 · focus · HN ↗
    Playing the critic that alternate “50%” outcomes were:
    - You didn’t have a thyroid issue and you spent thousands more on tests the doctor was right that you didn’t need.
    - Your cat didn’t have that issue and you spent hundreds on unnecessary tests.
    Are you essentially saying that it is better for an LLM to sell you an idea that sometimes might be right, selling a dream to anyone with and without the money for it?
    So… a telephone psychic?
    The thing about WebMD telling everyone they have cancer is that sometimes it will be right.
    (Just for clarity, I’m really glad it was helpful for you.)
    S0y · 1790519640 · focus · HN ↗
    Having the LLM give you new options doesn't mean you have to pursue them. It's just something else to talk to your doctor about.
    And it's not like self researching medicial issues isn't a new concept.
    cj · 1790520632 · focus · HN ↗
    I'm so tired of hearing the "unnecessary tests" line.
    Anyone who says this has not experienced poor quality healthcare.
    Many doctors simply aren't good and don't run tests because they are doing the bare minimum to get you out of the office.
    zephen · 1790521077 · focus · HN ↗
    > You didn’t have a thyroid issue and you spent thousands more on tests the doctor was right that you didn’t need.
    Uhhhh, even in the US, we're not talking "thousands" here.
    > So… a telephone psychic?
    Look, at one level, you can think of LLMs as search with a sometimes useful probability engine sitting on top of it.
    As someone who had multiple doctors do their damnedest to kill me on multiple occasions, I was very grateful to have google search back in the day, and you bet your bottom dollar I use LLMs to help me with health issues today. They are often better than most doctors.
    Does that put me at risk of doing something stupid? Only if I don't do further due diligence.
    bastawhiz · 1790531040 · focus · HN ↗
    I mean, the LLM was right in both cases. And that's what tests are for: testing whether a hypothesis was correct. It's only a waste of time and money if the LLM is only marginally better than flipping a coin. It's my experience that the LLMs are far, far better than that threshold.
    The difference between an LLM and WebMD is that WebMD is a page of information. Any reasoning is yours, based on a small fraction of the information available to you. An LLM is based on all the medical literature of all time. It's not even close to being the same thing.
    Even if you argue that an LLM is just a prediction engine, that's kind of exactly the right tool for the job. "If you have these symptoms and not these other ones, these are likely causes" plays against the lone strength that LLMs' detractors argue they do well.
    mrandish · 1790534904 · focus · HN ↗
    > an LLM to sell you an idea that sometimes might be right
    An LLM is just an information resource that can be useful when used properly. It can also be useless or net harmful when used improperly. The same is true for web search, libraries and even human experts. A licensed medical doctor is a domain expert and competent domain experts are often correct, but not always.
    I don't think it's possible to suggest a universally 'correct' default position on when (and how much) to accept your GP's medical opinion over any alternatives. It depends on too many variables: the context, the person, the alternative info sources, the expected value of correctness and the potential consequences of error. But decision theory suggests this is the kind of combinatorially complex problem space for which any single default position, whether "Always trust your GP" or "Never trust your GP", for every person and situation cannot be optimal.
    therealpygon · 1790607897 · focus · HN ↗
    I agree with you. And if they flipped a coin, and the coin happened to land on heads which they decided was a prediction of no cancer, would you recommend they tell others flip their coin if they are tired of their doctor because it was helpful when their test came back negative? “It didn’t hurt anything”, right? What happens when that specialized test is $100k because it is rare, and your doctor is pushing back, “but the quarter told me i have this cancer and you said your other tests couldn’t rule it out”. Is there no harm then? Would you still think they should go around advertising their cancer detecting quarter without anyone pointing out it’s freaking quarter?
    Btw, no one suggested people shouldn’t consider various forms of information. All I pointed out is that if an LLM happened to guess right, it doesn’t mean that is some sort of datapoint that others should believe validates an AI’s predictive abilities any more than a telephone psychic or webmd should be a trusted source because they are occasionally right.
    It is exactly the tactic that makes snake-oil…I mean the vitamin industry…so profitable from people going around and telling others crap like “rubbing the oil from this tree on my foot made me sleep better”, and “I took this garnish as a supplement and it cured my blah blah”. Can people need vitamins? Sure. Does the average person need anywhere near the amount they take? Rarely. People sell each other a dream of feeing better; this would be cutting out the middle man and suggesting an AI take over.
    It’s funny how pointing out the risk means people act like I’m suggesting it should be banned. We can see how many MLM Karens came out of the woodwork to defend AI in this, as though I somehow said people shouldn’t be allowed to consider feedback from an LLM by simply pointing out exactly how wrong — best case — it could have easily been from a system known to make errors. In fact, the only reliable part about AI right now is that it will make mistakes. Literally almost all investment in AI is about having it make less mistakes,. Every single metric we use to measure AI is about how many mistakes it makes. Every time we test how smart it is, we measure how much it got wrong and how many times it failed.
    I understand it. Sick people will turn to just about anything as a cure or to make them feel better, or simply to find some hope. Sometimes they get better, whether that thing helped or not, and then loudly tell everyone that whatever thing did it regardless of realities. I’m not discounting anyone’s stories, I’m pointing out how they were lucky it didn’t go exactly the opposite, which is what the next person could experience by trusting and not considering the feedback from an LLM cautiously.
    Also, can literally anyone point out where the comment I replied to said a word about “doing research”? Anyone? People immediately Karen in on how “it’s just part of research” — where was that said? Or did people just read what they wanted to? Too funny.
    PS, I have MS… before any more idiots want to tell me that I don’t know anything about the problems with the US healthcare system.
  6. arecurrence · 1790518033 · focus · HN ↗
    I would be careful treating this article as relevant in September 2026. The pace of advancement in the field is such that 6 months is relatively ancient let alone when the models in the study were released. GPT-4o was May 13, 2024. Far before November 2025 when people collectively noticed a turn in LLM value.
    LLMs earlier this year were surpassing numerous health related benchmarks when scored against human physicians. Only 2 months after this nature article Harvard posted that LLMs were now outperforming ER docs when given authority to order tests. <a href="https://www.harvardmagazine.com/ai/ai-outperforms-doctors-diagnosis-harvard-study" rel="nofollow">https://www.harvardmagazine.com/ai/ai-outperforms-doctors-di...
  7. azornathogron · 1790521496 · focus · HN ↗
    Going to the doctor for expert advice is often inconvenient, or time consuming, or expensive, or stressful. I think a lot of people want, and seek out, information and advice based on their symptoms, as a first step before a possible doctor's visit. Before LLMs, WebMD (and excessive self-diagnosis based on WebMD) was a meme for a while.
    So with regard to LLMs, for me the question is not purely "how often does it get it right?" the question is "how does it compare to the sources and self-diagnosis methods people use otherwise?"
    Of course, I agree people should be careful with any form of self-diagnosis or LLM-diagnosis.
  8. zephen · 1790521793 · focus · HN ↗
    They recruited people to pretend like they had various issues, and then recorded their interactions with the LLMs.
    Someone who's been paid a couple of quid to pretend to have a medical condition can easily miss things, and is unlikely to be anywhere near as invested in drilling down to the correct solution as someone who's really suffering.
    Another outcome from the study was that the LLMs could do better with the right people driving them. That's not news.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.