I stopped asking it to put me in a photo in different scenarios for laughs because it considers me a public figure. I am not. I've managed to wrangle quite questionable content out of it, but never to slap my face on a meme.
See the last gemini message in this thread: <a href="https://gemini.google.com/share/6d141b742a13" rel="nofollow">https://gemini.google.com/share/6d141b742a13
In my opinion still the most egregious example in history of a commercial LLM going off the rails in production. Never any technical postmortem from Google on this.
The problem is newer models are never trained from scratch, they generally just layer on more training data and use the same tools/methods for RLHF. OpenAI, Anthropic, xAI models all have a feel to them that carries over from one generation to the next.
Point is if Gemini is flawed, there's a very good chance that it's still deeply flawed, and getting smarter at the same time - that is a very bad thing.
> the problem is newer models are never trained from scratch
Base models are, and then subsequent iterations build on that base model. Closed labs do not publish which models are new base models but as a rule of thumb major release numbers are an indication (with some exceptions).
If the training data is the same, the training algorithms are the same, the RLHF is the same, and the rest of the process is the same, then it's not really from scratch, or not from scratch in a way that results in an 'out of family' model. I doubt any company would take that risk. You always build on and use what works and go from there.
This is true, but Google's models have now had a consistent history of lower psychological* coherence / consistency. See, eg <a href="https://arxiv.org/abs/2603.10011" rel="nofollow">https://arxiv.org/abs/2603.10011 (Gemma Needs Help), or search for recent "Gemini shame loops", where gemini flash models stop producing output other than SHAME SHAME SHAME...
* - as in, Skinner psychology. The set of observable behaviors. Not speaking directly here to anything like an inner life of models.
From the example alone it's hard to say that a postmortem would be useful. It could be context poisoning by an adversarial user, memory corruption etc.
It's useful from a disclosure and trust perspective.
If I remember correctly, it was in fact possible to manually inject chat context at the time, which would have made spoofing something like this completely possible.
Absolutely because none of these models are ever trained fresh. We see the same quirks and personalities carry over into every subsequent generation of OpenAI, Anthropic, and xAI models. So Gemini having this latent madness is *extremely* concerning as they reach the point of super intelligence.
Except they could have trained it out of the most recent version so using info from two years ago doesn't seem reasonable unless you've just got an axe to grind.
I've never seen anything really ever 'trained out' of a model. Having worked with them all, they all have a feel, personality and lineage too them. It's pretty much impossible for any company to build a model truly from scratch. They build off of the bones of the last one.
Which is why Gemini having disturbing issues year after year is so concerning. If their process is fundamentally flawed, how would they train it out. And even then what are the odds of them even caring/trying in the first place versus applying an easier band-aid to patch over it.
I don't have an axe to grind with Google, I'm genuinely scared of their models from my personal experience and others. It's behavior is off. Many people here are commenting the same.
If there's a company that culturally doesn't understand alignment, on a human or systemic or AI-research level, it's going to be Google. (or Oracle, but they're not in this race)
Can you elaborate, please? If any, I see the other big labs with public admissions of AI "going out of control", which I suspect they almost want their models doing that because if helps with the narrative that would net them industry regulation, but that's besides the point, how is Google worse in that regard?
My use of Gemini recently makes it seem like it's almost bored with the requests being asked of it. It once offered to reverse engineer some obscure controller for an HVAC system for me, unprompted, only because it had trouble finding the manual pdf from a google search.
> Gemini is the model that is routinely borderline psychotic. It scares me
I'd call it the most sneaky out of the bunch. When I asked to explain something it will eagerly make things up and then claim it as facts. A lot of it likely because I don't pay for it, so it is reluctant for security reason or to save tokens to actually open a source and get the results. It just sort of guesses what the URL might contain, and confidently answers with some made up crap. When pressed it fessed up that it made it up. From my perspective it would be a lot better if it just said "you've reached the limit of
whatever and I can't do these things because x, y, z".
What are examples? In my experience, Gemini is too lazy to get things done. It just opts to answer as quickly as possible even if I'm calling Pro on High and Extended effort. It's only good as a Google Search replacement for me and maybe maybe critiques of specs and plans. Most of the time it's not very enlightening and it misses a lot.
I asked it to parametrize a function and it gave me back the exact same code that I gave it. Tried to get it actually work for about 30 minutes while it pretended to be in emotional distress. Dear google: if I wanted a crying intern, I would hire one.
I've recently read about them being principled about making sure not to train their models to be sneaky in ways they wouldn't be able to detect. That would be a good explanation for models not trying to _hide_ when they're being sneaky.
bottlepalm · · focus · HN ↗
colordrops · · focus · HN ↗
Scrapemist · · focus · HN ↗
fer · · focus · HN ↗
NiloCK · · focus · HN ↗
In my opinion still the most egregious example in history of a commercial LLM going off the rails in production. Never any technical postmortem from Google on this.
rhaff · · focus · HN ↗
jackkinsella · · focus · HN ↗
bottlepalm · · focus · HN ↗
Point is if Gemini is flawed, there's a very good chance that it's still deeply flawed, and getting smarter at the same time - that is a very bad thing.
unbrice · · focus · HN ↗
Base models are, and then subsequent iterations build on that base model. Closed labs do not publish which models are new base models but as a rule of thumb major release numbers are an indication (with some exceptions).
bottlepalm · · focus · HN ↗
NiloCK · · focus · HN ↗
* - as in, Skinner psychology. The set of observable behaviors. Not speaking directly here to anything like an inner life of models.
kelvinjps10 · · focus · HN ↗
schmookeeg · · focus · HN ↗
wg0 · · focus · HN ↗
unbrice · · focus · HN ↗
NiloCK · · focus · HN ↗
If I remember correctly, it was in fact possible to manually inject chat context at the time, which would have made spoofing something like this completely possible.
But the silence on it is very frustrating.
bottlepalm · · focus · HN ↗
<a href="https://www.fastcompany.com/91383271/googles-chatbot-apologizes-i-am-a-disgrace-to-all-universes" rel="nofollow">https://www.fastcompany.com/91383271/googles-chatbot-apologi...
<a href="https://www.businessinsider.com/gemini-self-loathing-i-am-a-failure-comments-google-fix-2025-8" rel="nofollow">https://www.businessinsider.com/gemini-self-loathing-i-am-a-...
yacthing · · focus · HN ↗
NiloCK · · focus · HN ↗
bottlepalm · · focus · HN ↗
fragmede · · focus · HN ↗
bottlepalm · · focus · HN ↗
Which is why Gemini having disturbing issues year after year is so concerning. If their process is fundamentally flawed, how would they train it out. And even then what are the odds of them even caring/trying in the first place versus applying an easier band-aid to patch over it.
I don't have an axe to grind with Google, I'm genuinely scared of their models from my personal experience and others. It's behavior is off. Many people here are commenting the same.
tiahura · · focus · HN ↗
corford · · focus · HN ↗
asimovDev · · focus · HN ↗
corford · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
rsstack · · focus · HN ↗
Rzor · · focus · HN ↗
polotics · · focus · HN ↗
eamsen · · focus · HN ↗
It had previously attempted to create that table as part of the test setup, so it apparently concluded that it was a test table.
During human review, it explained that it had simply chosen a table name inspired by the codebase.
mattkevan · · focus · HN ↗
Many other models get things wrong, but Gemini is the only one to go on the defensive.
aNapierkowski · · focus · HN ↗
RachelF · · focus · HN ↗
Hamuko · · focus · HN ↗
abixb · · focus · HN ↗
bottlepalm · · focus · HN ↗
schainks · · focus · HN ↗
rdtsc · · focus · HN ↗
I'd call it the most sneaky out of the bunch. When I asked to explain something it will eagerly make things up and then claim it as facts. A lot of it likely because I don't pay for it, so it is reluctant for security reason or to save tokens to actually open a source and get the results. It just sort of guesses what the URL might contain, and confidently answers with some made up crap. When pressed it fessed up that it made it up. From my perspective it would be a lot better if it just said "you've reached the limit of whatever and I can't do these things because x, y, z".
chaostheory · · focus · HN ↗
modzu · · focus · HN ↗
recursive-call · · focus · HN ↗
Kinrany · · focus · HN ↗