I’ve been trying to put my finger on what it is that is happening when the robot writes stuff like “3 campuses, one app” example, I’m glad the author was able to identify it as chat context leaking through.
The other version of it is the robot over-indexing on some part of the prompt and leaving comments places. An example is I ask it to prefer integration tests using TestContainers, it starts adding comments to every new test saying “Real services, no mocks” or something.
And yes, I have a line in my *.MD saying not to do that.
LLMs love <a href="https://tvtropes.org/pmwiki/pmwiki.php/Main/SuspiciouslySpecificDenial" rel="nofollow">https://tvtropes.org/pmwiki/pmwiki.php/Main/SuspiciouslySpec... . You tell them not to do a thing, they make a point of saying they didn't do the thing.
It's partly a "Don't think of a Pink Elephant" problem. I've worked on several projects where I had to keep telling prompt writers to stop writing negative examples because the more you include the more its "attention" to them is all it has. Like telling a toddler not to do something and being surprised that is now all they can think about and they want to keep doing it. These prompt writers kept getting surprised that I'd delete all their negative examples and harshly worded "Don't do X" and "Never Y" and "NO: Z" sections they spend so much time on and got better results with smaller more focused positive example only prompts.
weakfish · · focus · HN ↗
The other version of it is the robot over-indexing on some part of the prompt and leaving comments places. An example is I ask it to prefer integration tests using TestContainers, it starts adding comments to every new test saying “Real services, no mocks” or something.
And yes, I have a line in my *.MD saying not to do that.
JoshTriplett · · focus · HN ↗
WorldMaker · · focus · HN ↗
demibabs · · focus · HN ↗
They just also really love to tell you about it.