I’ve been trying to put my finger on what it is that is happening when the robot writes stuff like “3 campuses, one app” example, I’m glad the author was able to identify it as chat context leaking through.
The other version of it is the robot over-indexing on some part of the prompt and leaving comments places. An example is I ask it to prefer integration tests using TestContainers, it starts adding comments to every new test saying “Real services, no mocks” or something.
And yes, I have a line in my *.MD saying not to do that.
LLMs love <a href="https://tvtropes.org/pmwiki/pmwiki.php/Main/SuspiciouslySpecificDenial" rel="nofollow">https://tvtropes.org/pmwiki/pmwiki.php/Main/SuspiciouslySpec... . You tell them not to do a thing, they make a point of saying they didn't do the thing.
And btw, if you have slop comments in your PR, you'll get a review from an agent (my company pay for it). I won't bother to read if you didn't bother either.
Eh, I'd prefer a opus5.5 review with consequent approval over one by a human which I have to wait 5 days for.
Ymmv, I also hate it when people dont read their own PR first though. But I know I've previously identified comments as LLM written which the dev... On further discussion definitely wrote themselves. So you gotta keep the false positives in mind.
weakfish · · focus · HN ↗
The other version of it is the robot over-indexing on some part of the prompt and leaving comments places. An example is I ask it to prefer integration tests using TestContainers, it starts adding comments to every new test saying “Real services, no mocks” or something.
And yes, I have a line in my *.MD saying not to do that.
JoshTriplett · · focus · HN ↗
orwin · · focus · HN ↗
And btw, if you have slop comments in your PR, you'll get a review from an agent (my company pay for it). I won't bother to read if you didn't bother either.
ffsm8 · · focus · HN ↗
Ymmv, I also hate it when people dont read their own PR first though. But I know I've previously identified comments as LLM written which the dev... On further discussion definitely wrote themselves. So you gotta keep the false positives in mind.