You don't need documentation or the 3rd party memory systems. The code IS the documentation.
All this stuff is LLM rube goldberg machines. It just pollutes context.
I barely use AGENTS.md/CLAUDE.md these days. And where they remain, it's super basic high level stuff.
I'm honestly still kicking myself in the ass on many projects where I did something similar to this. I kept tons of markdown docs and decision docs. Now those things are just causing problems because they got stale. Even after having sessions of reconciling documentation, the LLM just gets confused.
It’s a weird thing isn’t it, the urge to save these artifacts? The worst is when the llm refers to the decision and design in code comments. In my opinion there’s one thing that is worth documenting; tricky architecture or implementation details that are some how counterintuitive to what would have normally been done. But again this can be documented in the code and tests.
I think it's going to be an incredibly common, maybe universal cycle people will go through working with AI until they realize it doesn't work long term.
The "why" should most often be self evident. If it's not, that's what commit messages are for. Not more markdown files and code comments.
Unfortunately, that is not how many software development projects work. A customer may request certain things, or the software might be part of a larger system.
Consider working on software for a coffee machine. Why is pin 42 (GRIND) activated every now and then? Would you really want to document that in Git commit messages?
Yeah, I'm going too far the other way. There are definitely legitimate reasons to comment in code and have documentation. I just feel like we're seeing a MASSIVE amount of documentation now and it's so much that it's mostly worthless.
I'm seeing decision files that are so big the LLM can't fit it all in context on some projects. Then the LLM makes decisions that revert previous ones and later sessions don't pick that up so it sticks to the original decision. Now in some sessions, every so often I have to remind the LLM, "no, we changed that later, we do it this way now"
And I'm seeing our knowledge base grow to a completely useless giant mess of stale, outdated, duplicated, or superfluous info. LLMs often pull unrelated info or confuse similar but different things or get old documentation for something that's been updated to new documentation in a different part of the knowledge base. And these are LLMs generating the docs. And we have LLMs and agents reconciling. But it doesn't seem to always get everything.
For code comments, it's terrible because the comments are starting to get larger than the code. A large chunk of the comment can be discerned from the code itself. Then the comment has details on why that maybe don't quite make much sense. It's like the LLMs start using words in a specific context that doesn't really apply to the word in normal spoken english. Then it will also often include a specific JIRA ticket id, you check the JIRA ticket, you see that yeah, the code was changed because of that JIRA ticket, but it's not really related to the ticket itself, it was just a blocker. But now the comment forever links it to THAT ticket (And then now sometimes the LLM pulls in that ticket with the atlassian mcp).
Haha, all the software devs who hate writing documentation are naturally finding their preexisting beliefs reinforced when the LLM is able to discern intent without docs. An LLM can be spooky impressive at reading minimized or obfuscated code, for example.
But this article argues that LLMs do better when the context is smaller — when it can understand the totality of the task with as little context as possible. And so having correct API-level docs is greatly advantageous. Anecdotally, this rings true to me — when the local context is good and clear, the LLM writes code matching my intent even when my prompt is sloppy and poorly specified.
Rejoice! The LLM will write the docs for you, relieving you of most of the work.
However without intervention, it will do too much and record absurdly verbose docs (similar to how an LLM will relentlessly refactor your code until you instruct it to move in minimal, incremental changesets). You will still need to edit down what the LLM generates.
100% agree. Most of the things that make development better for humans also make development better for agents… and I think docs are even more important with agents, because of some kind of multiplicative effect. The agents are coding faster, and the benefits of documentation are somewhat more pronounced because of the speed.
The docs end up stale or you waste a ton of time and tokens keeping them up to date. And you can't even trust the LLMs to keep the docs up to date, you will still need to review it yourself keep a bunch of shit out of them.
"docs and maps" are hardly specifying preferences. I DID say I still often keep a high level AGENTS.md or CLAUDE.md. Where I have them, they're super high-level. In places where we have well defined and followed standards that top level markdown file can be as simple as "check out this other repo for patterns" and where I don't, the markdown file is high level overview of what the app is and then a high level overview of architecture or some preferences. Always a shorter/smaller doc.
I don't mind SOME documentation. I'm growing extremely frustrated with the absolute MOUNTAIN of docs, comments, decision documents, etc being created. I'm so fucking tired of the LLM rube goldberg machines being built.
You guys show me a measurable difference on something with docs vs without docs and I'll believe it.
The docs just make you feel good. They're not worth anything. They just create more work if anything because now you don't only have to fight entropy in your codebase but also your docs.
Uh, you aren't providing numbers either, just appending the same assertion to the end of each branch of the discussion. The result is the same as when the LLM does that: geometric increase in what we have to fight through to reason about the problem.
With the LLM, though, I have some influence: I can instruct it to be more succinct. I don't have a specific top-level instruction about that at present in any global prefs file, but I've occasionally asked it to summarize and archive and it's done an okayish job of pruning.
Yes, you're right, I'm not providing numbers. But that's because I'm just using the tool as designed... the vanilla version. I'm using the baseline. I believe anthropic and openai publish plenty for my side of the argument already.
I'm not the one bolting things on claiming it increases productivity. Why would I ADD stuff without any proof or evidence that it works? I'm simply NOT adding these things.
"CLAUDE.md files solve this by giving Claude persistent context about your project. A well-configured CLAUDE.md transforms how Claude works with your specific project. The file serves multiple purposes: providing architectural context, establishing workflows, and connecting Claude to your development tools. Each addition should solve a real problem you have encountered, not theoretical concerns about what Claude might need."
Yeah. Basically IMO/IME the current best practice is to start the LLMs out with minimal/no skills/instructions/docs.
Notice the things they struggle with and the small problems (typically, environment issues IME) they repeatedly encounter and re-solve across multiple sessions. That is what your instructions should cover. When possible, move those instructions into skills, so they get loaded into context selectively instead of on every session. (Example: instructions for running specs, placed into a skill that only gets loaded into context when it’s time to run specs)
Again, this can be automated by the LLMs themselves: both Codex and Claude (and I’m assuming other major harnesses) know how to read their own transcripts and are good at looking for repeated friction and making concrete suggestions to reduce that friction in the future.
It takes a bit of a time investment on the user’s part, and every now and then you probably should throw it all out and start fresh so that the new batch of instructions can be appropriate for the current state of the repo and the capabilities of whatever model(s) you’re using.
Honestly, don't agree. I think even this is a waste of time and context.
The LLMs will always have these suggestions on things that will "reduce friction in the future" but I really think the LLM is a sycophant glazing you. Because even when you add those skills or prompts at some point the LLM ignores it or does whatever it wants to do anyways. Better to leave most of that out and deal with it when it arises.
Even your example, instructions for running specs... none of the current frontier models need this at all.
Nope. Opus and Sol for the most part. Haven't even used Astra or Fable.
At work it's all Anthropic. I spent a while on Sonnet models but have been pretty consistent about using Opus now. For personal projects I bounce between Sol/Terra and Gemini Flash.
Most of the models are good enough now. You don't need all these documents, memory, plugins, skills, etc. MCPs are good for hooking up to external sources of data (at work we have gitlab mcp, datadog mcp, and our knowledge base mcp... which the knowledge base is mostly worthless now because of all this AI generated documentation, but I digress). I DO still use beads on a lot of projects, but not always. My prompts these days are lazy as fuck: "review this repo, check out this other repo with architectural patterns we should follow and libraries/terraform/etc we should use, <basic desc of goal>. What do you think? Let's review everything and discuss before we start building"
Oh no, I might have to stop the llm agent on occasion, or review what they did and tell them to fix a couple things. Way better use of tokens and my time than creating some rube goldberg machine. Way better than keeping a giant decision file that goes stale and causes context corruption because the llm only saw the OLD decision and not the UPDATED decision.
I think they're using Skynet. They seem to be claiming that part of the reason for not giving the LLM directions is because the LLM won't follow them anyway.
That must be a remarkable model. Smart enough to flawlessly discover everything on its own in every session... but also it just straight up just doesn't obey direction.
I'm saying a lot of devs/engineers little rain dances aren't really bringing the rain.
Keep a basic CLAUDE.md/AGENTS.md that keeps high level details. Have a way to reference other projects/apps/codebases to use as reference architecture.
Don't keep tons of markdown files. Don't keep "decision" documents (these are probably the worst). Don't make skills for every little thing (superpowers are dead these days, stop using them). Oh yeah, don't have the LLM generate docs for reference later... if it can generate the docs it doesn't really need them.
I understand what you're railing against in general: engineers who make these big memory/doc systems that quickly go stale and are at best useless and at worst actively harmful. Even more insidiously, these engineers are blind to this fact because maybe some of this stuff was effective 6-12 months ago and they haven't reassessed their workflows. Now that's dogma. We really do agree on that.
However. At least in the projects I've worked on, it's trivial to read the transcripts and notice that without enough direction, there are certain things the agents will burn lots of tokens to rediscover in every session. That's absolutely not dogma.
I think there's a balance between the tokens burned up rediscovering things vs tokens being used keeping docs/decisions/skills/etc.
And I think a lot of places are burning way more on the latter than the former. I think the former is more cost effective due to caching as well. So I HEAVILY lean towards the former especially with how effective I think it is these days.
Don't agree. LLM generated docs are some of the worst because like I said, the LLM never trims, only amends. So we used to do X but now we've had an architectural change or some type of change where we should never do X. Instead of just removing the instructions to do X in the docs, it amends them, "we made the decision to no longer do X because of Y". Now it has X multiple times in context instead of just not having X in context at all.
I hesitate to make blanket statements, since this is quickly evolving, but in general I agree. LLM generated docs will quickly degrade to overly verbose, unreadable crap.
Not just that they get stale, but if you're using the LLM to generate the doc you don't need it.
I don't think I've ever seen an agent review a doc and then NOT also go look at the code. So skip the middleman, just have the agent look at the code.
Yup. Examples examples examples. All of the descriptive stuff is just nonsense that confuses the point. Makes perfect sense when you remember that these things are not intelligent, but truly just autocomplete on steroids.
>> You don't need documentation or the 3rd party memory systems. The code IS the documentation.
We have heard this nonsense from the "we don't need to write comments, code should be self-documenting" types for decades. It was wrong in that context, and it is wrong in this one.
Code tells you how a system works. It does not tell you why it works that way. That is what memory is for. It exists so that your AI does not keep undoing past decisions when it writes or refactors code.
I've had the LLMs undo past decisions MORE from the docs than from having no docs.
Old decisions always end up in the docs. If the LLM gets a whiff of an old decision, but doesn't get the update to that decision, well now you're doing things the old way again.
I liked this advice when humans wrote code. Though even then I'd urge people to write meaningful commit messages that capture the "why" of what they did, so no one tramples their intent by mistake.
But not sure it works in an age where most code is LLM-generated. Especially if that code is not even reviewed by humans (irresponsible or not, it's happening), and commit messages are also generated by AI. I think something is needed to separate "what did the human operator intend" from what the agent went and built.
I do agree that this gets way overengineered. My approach has been more or less what you stopped doing though - committing all our timestamped "plan/implementation docs" and "investigation docs" that document what the user wanted + empirical findings, and making all prior session transcripts searchable. It's seemed mostly helpful? For whatever reason I haven't run into many staleness problems so far.
I'm mainly aiming my frustrations at all the markdown files being committed, all the additions to knowledge bases, all the comments in the code (especially the ones referencing specific JIRA tickets). This stuff isn't helpful, it gets hella stale. I've had the LLM fuck up plenty due to these docs and comments.
Commit message ARE EXACTLY where architectural decisions or nuance should go. Not another fucking .md or more comments.
I did say a high-level CLAUDE/AGENTS.md is fine.
Make a high level .md, let it rip, iterate. You can give specifics in your original prompt.
I don't need to document the framework or the libraries etc. Pre-code, I tell it in the prompt one time. After it inits the project it's in the code.
I don't understand. Code doesn't capture the context in which decisions were taken: why is code the way it is? What is important? What is not? How can agents make correct decisions without knowing context that cannot be inferred from code?
Code as documentation works better when the code is declarative or a DSL. These capture intent and promises (as in promise theory) better.
When it is not, it has to be reasoned out and does not work well for documentation.
Other things that code and tests alone do not capture well:
- promises (as in Promise Theory) made to other parties. Claude already has PT in its training data.
- Constraints-inducing-properites, as in Roy Fielding / Christopher Alexander. While tests, and property testing can capture properties, there is no formal connection to the constraints that induces fhem. By constraints, I am not talking about business requirements and business value — those are better understood through Promise Theory. I am talking about things like at-least-once delivery or total ordering (from append-only constraint). Claude already has Fielding’s dissertation and Alexander’s works and ideas in its training data.
- grammars, as in pattern panguages (not just patterns) a la Alexander / Fielding are also not captured in code alone. These tell both humans ans AI how to extend a pattern, and how to identify anti-patterns (when they violate a constraint-inducing-property)
- LLMs are trained with many different worldviews and bounded contexts at the same time, and is very capable of translating across it. However, these need to be soelled out, otherwise it would talk in whatever it infers
Specifications written for the exact way components are wited together run into that stale doc problem. Although it takes much more human attention and token burn to describe things in terms of pattern language and promise theory, it becomes easier over time. The actual implementation plan tends to fall out more cleanly when all those other stuff are at least considered. This is where I have been spending most of my time.
Code can get much larger than the documentation that summarizes it. It also doesn't cover intent or rationale. Even if you inexplicable don't want documentation, at the very least use something like gitnexus to map out your code because relying on code alone isn't good enough
It's frequent for SWEs to make blanket statements with considering the vast space of issues other people face that they don't have experience with or awareness of. Anyway, I'm not going to tell you what you do or don't need, only what worked and didn't for me.
In my codebase it is difficult to get agreement on comments and documentation so rather than rely on it I adapted. One of the first things I did when I succumbed to agentic development was to point codex at the code and ask it to generate a high level description of where important files, such as our public API, reside, what the hierarchy is, what the code does, etc. In my case, this level of documentation is fairly static if I avoid implementation details. So now I have a handful of agent files in my tree and it seems to save quite a few tokens and improve my results. I frequently have other devs ask me how I get such good results when doing agentic reviews of their changes(always my first step now before I start my human review). I also include instructions in the agents files instructing the agent to maintain the agent files if any relevant changes are made. It seems to work quite well for me.
Idk I just can't agree. Code is the what but it doesn't tell you the why. There are so many times where at first glance the code seems suboptimal or bad or wrong, and it's only when you learn of some constraint somewhere else that it begins to make sense.
All code is written under constraints, and most constraints live outside the code.
Most often the "why" should be self-evident.
When it's not, there are commit messages.
Please for the love of god, quit generating markdown files (especially having the LLM generate the file, because if it could generate it, it doesn't need it), quit generating "decision" docs, and stop having more comments than code.
I partially agree, way too many people are cargo culting overly complex AI workflows with little empirical data. My framing is a bit different though, I consider code the spec and actually keep a decent amount of docs for higher level concepts. So far this is working well for me across Claud and Codex.
High level docs are fine and can be okay or I can see them as being helpful. I did say I'll keep an AGENTS.md/CLAUDE.md. There also are SOME docs and sometimes the LLM makes the docs anyways. I'm not an absolutist, but I'm REALLY fucking tired of seeing these giant .md files, "decision" documents, more comments than code, etc.
It's a LOT of cargo-culting overly complex AI workflows. It's devs/engineers making rube goldberg machines.
Everyone is doing their own little rain dance and when it rains they say "see, I told you it works"
In general, as LLMs get smarter, “less is more” becomes increasingly true. You’re polluting their context with dozens or hundreds of instructions, all of which the LLM tries to satisfy. However,
You don’t need documentation […or…] memory
…oh heck no. Easy to miss at Claude and Codex’s default detail level but if you read your actual session transcripts, you are almost certain to notice the LLM solving lots and lots of the same little problems over and over again.
The code IS the documentation
First, lots of things cannot be learned from the code.
Trivial example: LLMs struggled with QA on our app. They didn’t know how to find the seeded test accounts. They would create new ones and do it wrong, or find the seeded test accounts but not know their passwords because they were encrypted, so they’d change the passwords but not tell the other agents. Shitloads of tokens burned. It was a no-brainer to just give them the credentials in a skill that gets loaded when they do QA.
You could say that’s an environment issue, not a code issue. But as far as actual code IME at a minimum we have to tell the agents our general repo structure and architecture patterns.
The days of “you are the world’s greatest Python programmer, write good code” or whatever are over (if that style of prompting even worked in the first place) and as I said less is more. But, still….
> You could say that’s an environment issue, not a code issue
Yeah, that's exactly what I'd say. Or maybe you're just approaching the problem now.
You have to prompt these QA agents, correct? Why not give the instructions on where to get credentials in the original prompt?
> But as far as actual code IME at a minimum we have to tell the agents our general repo structure and architecture patterns.
I did say I'll keep an AGENTS.md or CLAUDE.md. I keep that SUPER high level. The most depth I'll give is info about example projects (That the llm can reach using an MCP) to follow for architectural patterns.
Why not give the instructions on where to get
credentials in the original prompt?
We don't push code unless it's been QA'd, and the LLMs need to know the credentials every time they do automated QA. It's very rare for me have a session in that repo where they wouldn't need that info. So it's a great candidate for AGENTS.md
Another concrete example would be codebase conventions. We prefer lean models. Cross-model concerns go into service objects. Without this direction LLMs tend to default to stuffing too much code into the models themselves, as is the de facto standard for most MVC apps and therefore this is how LLMs are trained.
Even if they could discover our large codebase's conventions flawlessly on their own in every session, this absolutely would require nontrivial work repeated in every session: quite a few turns grepping, conversing with LSPs, or whatever.
So yes... we could describe our conventions in every single session by typing it right into the prompts... right after typing the directions to find the test credentials... and the other ten or twelve things we'd be telling it every single time...
I'm assuming these QA agent are automated, probably in CI/CD. And I'm assuming the prompt that runs these QA agents is from a codebase in VC. Why would I put QA instructions into the codebase's repo as a top level markdown file and not in the pertinent part of my QA process?
Even with your documentation, the llm is gonna do a lot of that grepping and discovery. Unless you have such comprehensive documentation that it's basically code itself... in which case, it should just use the code as documentation.
The good news is that I don't think what you guys are doing is going to be really bad. It's just not near optimal and it's creating this weird dogma around all these "tools"
Do you not run your tests locally? Regardless of what's happening or not happening in CI/CD, I would think that most workflows involve running the tests locally. We do TDD more or less and for the browser-based tests, the LLMs need to know how to log into stuff.
No, these kind of tests aren't generally run locally. Not to get to prod at least. These are all deterministic tests to get to prod.
Locally? Our devs can do whatever they want. For anything I'm working on, for local deployments and testing, I generally build it out so it's set in Makefile/Taskfile. If you're running your determinstic tests locally it should really just be a single command that bootstraps everything and runs the tests, or updates a certain part of your app and runs the tests. The LLM shouldn't need credentials in that instance.
Regardless I'm deploying a local stack of some sort all the credentials and everything are probably stored locally or in an env var where the llm will have access. So if I was having the LLM drive a browser during active development, it can look at the files.
Sure, I guess we could put in the top level markdown more details about this... but why? It takes little time or context for it to figure it out. We have shit change so frequently that it's just something else we have to maintain.
Versioned documentation is inspectable, reviewable, and easier to correct than opaque recalled snippets. Documentation should describe current truth; append-only events can preserve history.
kaydub · · focus · HN ↗
All this stuff is LLM rube goldberg machines. It just pollutes context.
I barely use AGENTS.md/CLAUDE.md these days. And where they remain, it's super basic high level stuff.
I'm honestly still kicking myself in the ass on many projects where I did something similar to this. I kept tons of markdown docs and decision docs. Now those things are just causing problems because they got stale. Even after having sessions of reconciling documentation, the LLM just gets confused.
le-mark · · focus · HN ↗
kaydub · · focus · HN ↗
AppleBananaPie · · focus · HN ↗
I think it's going to be an incredibly common, maybe universal cycle people will go through working with AI until they realize it doesn't work long term.
smokel · · focus · HN ↗
Why something exists, and how it connects to the outside world may be documented in comments, but more often than not it isn't.
haukebri · · focus · HN ↗
[dead]
arcanemachiner · · focus · HN ↗
kaydub · · focus · HN ↗
smokel · · focus · HN ↗
Consider working on software for a coffee machine. Why is pin 42 (GRIND) activated every now and then? Would you really want to document that in Git commit messages?
kaydub · · focus · HN ↗
I'm seeing decision files that are so big the LLM can't fit it all in context on some projects. Then the LLM makes decisions that revert previous ones and later sessions don't pick that up so it sticks to the original decision. Now in some sessions, every so often I have to remind the LLM, "no, we changed that later, we do it this way now"
And I'm seeing our knowledge base grow to a completely useless giant mess of stale, outdated, duplicated, or superfluous info. LLMs often pull unrelated info or confuse similar but different things or get old documentation for something that's been updated to new documentation in a different part of the knowledge base. And these are LLMs generating the docs. And we have LLMs and agents reconciling. But it doesn't seem to always get everything.
For code comments, it's terrible because the comments are starting to get larger than the code. A large chunk of the comment can be discerned from the code itself. Then the comment has details on why that maybe don't quite make much sense. It's like the LLMs start using words in a specific context that doesn't really apply to the word in normal spoken english. Then it will also often include a specific JIRA ticket id, you check the JIRA ticket, you see that yeah, the code was changed because of that JIRA ticket, but it's not really related to the ticket itself, it was just a blocker. But now the comment forever links it to THAT ticket (And then now sometimes the LLM pulls in that ticket with the atlassian mcp).
haukebri · · focus · HN ↗
[dead]
rectang · · focus · HN ↗
But this article argues that LLMs do better when the context is smaller — when it can understand the totality of the task with as little context as possible. And so having correct API-level docs is greatly advantageous. Anecdotally, this rings true to me — when the local context is good and clear, the LLM writes code matching my intent even when my prompt is sloppy and poorly specified.
Rejoice! The LLM will write the docs for you, relieving you of most of the work.
However without intervention, it will do too much and record absurdly verbose docs (similar to how an LLM will relentlessly refactor your code until you instruct it to move in minimal, incremental changesets). You will still need to edit down what the LLM generates.
klodolph · · focus · HN ↗
trinsic2 · · focus · HN ↗
kaydub · · focus · HN ↗
The docs end up stale or you waste a ton of time and tokens keeping them up to date. And you can't even trust the LLMs to keep the docs up to date, you will still need to review it yourself keep a bunch of shit out of them.
joquarky · · focus · HN ↗
kaydub · · focus · HN ↗
I don't mind SOME documentation. I'm growing extremely frustrated with the absolute MOUNTAIN of docs, comments, decision documents, etc being created. I'm so fucking tired of the LLM rube goldberg machines being built.
kaydub · · focus · HN ↗
The docs just make you feel good. They're not worth anything. They just create more work if anything because now you don't only have to fight entropy in your codebase but also your docs.
rectang · · focus · HN ↗
With the LLM, though, I have some influence: I can instruct it to be more succinct. I don't have a specific top-level instruction about that at present in any global prefs file, but I've occasionally asked it to summarize and archive and it's done an okayish job of pruning.
kaydub · · focus · HN ↗
I'm not the one bolting things on claiming it increases productivity. Why would I ADD stuff without any proof or evidence that it works? I'm simply NOT adding these things.
frankacter · · focus · HN ↗
While arguing against using the tool as designed.
<a href="https://claude.com/blog/using-claude-md-files" rel="nofollow">https://claude.com/blog/using-claude-md-files
"CLAUDE.md files solve this by giving Claude persistent context about your project. A well-configured CLAUDE.md transforms how Claude works with your specific project. The file serves multiple purposes: providing architectural context, establishing workflows, and connecting Claude to your development tools. Each addition should solve a real problem you have encountered, not theoretical concerns about what Claude might need."
JohnBooty · · focus · HN ↗
Notice the things they struggle with and the small problems (typically, environment issues IME) they repeatedly encounter and re-solve across multiple sessions. That is what your instructions should cover. When possible, move those instructions into skills, so they get loaded into context selectively instead of on every session. (Example: instructions for running specs, placed into a skill that only gets loaded into context when it’s time to run specs)
Again, this can be automated by the LLMs themselves: both Codex and Claude (and I’m assuming other major harnesses) know how to read their own transcripts and are good at looking for repeated friction and making concrete suggestions to reduce that friction in the future.
It takes a bit of a time investment on the user’s part, and every now and then you probably should throw it all out and start fresh so that the new batch of instructions can be appropriate for the current state of the repo and the capabilities of whatever model(s) you’re using.
kaydub · · focus · HN ↗
The LLMs will always have these suggestions on things that will "reduce friction in the future" but I really think the LLM is a sycophant glazing you. Because even when you add those skills or prompts at some point the LLM ignores it or does whatever it wants to do anyways. Better to leave most of that out and deal with it when it arises.
Even your example, instructions for running specs... none of the current frontier models need this at all.
joquarky · · focus · HN ↗
kaydub · · focus · HN ↗
At work it's all Anthropic. I spent a while on Sonnet models but have been pretty consistent about using Opus now. For personal projects I bounce between Sol/Terra and Gemini Flash.
Most of the models are good enough now. You don't need all these documents, memory, plugins, skills, etc. MCPs are good for hooking up to external sources of data (at work we have gitlab mcp, datadog mcp, and our knowledge base mcp... which the knowledge base is mostly worthless now because of all this AI generated documentation, but I digress). I DO still use beads on a lot of projects, but not always. My prompts these days are lazy as fuck: "review this repo, check out this other repo with architectural patterns we should follow and libraries/terraform/etc we should use, <basic desc of goal>. What do you think? Let's review everything and discuss before we start building"
Oh no, I might have to stop the llm agent on occasion, or review what they did and tell them to fix a couple things. Way better use of tokens and my time than creating some rube goldberg machine. Way better than keeping a giant decision file that goes stale and causes context corruption because the llm only saw the OLD decision and not the UPDATED decision.
JohnBooty · · focus · HN ↗
That must be a remarkable model. Smart enough to flawlessly discover everything on its own in every session... but also it just straight up just doesn't obey direction.
Hmmm.
kaydub · · focus · HN ↗
I didn't say give no direction.
I'm saying a lot of devs/engineers little rain dances aren't really bringing the rain.
Keep a basic CLAUDE.md/AGENTS.md that keeps high level details. Have a way to reference other projects/apps/codebases to use as reference architecture.
Don't keep tons of markdown files. Don't keep "decision" documents (these are probably the worst). Don't make skills for every little thing (superpowers are dead these days, stop using them). Oh yeah, don't have the LLM generate docs for reference later... if it can generate the docs it doesn't really need them.
JohnBooty · · focus · HN ↗
kaydub · · focus · HN ↗
You're just stuck on your dogma.
JohnBooty · · focus · HN ↗
However. At least in the projects I've worked on, it's trivial to read the transcripts and notice that without enough direction, there are certain things the agents will burn lots of tokens to rediscover in every session. That's absolutely not dogma.
kaydub · · focus · HN ↗
And I think a lot of places are burning way more on the latter than the former. I think the former is more cost effective due to caching as well. So I HEAVILY lean towards the former especially with how effective I think it is these days.
kaydub · · focus · HN ↗
icedchai · · focus · HN ↗
kaydub · · focus · HN ↗
I don't think I've ever seen an agent review a doc and then NOT also go look at the code. So skip the middleman, just have the agent look at the code.
joquarky · · focus · HN ↗
kaydub · · focus · HN ↗
NBJack · · focus · HN ↗
ramesh31 · · focus · HN ↗
enraged_camel · · focus · HN ↗
We have heard this nonsense from the "we don't need to write comments, code should be self-documenting" types for decades. It was wrong in that context, and it is wrong in this one.
Code tells you how a system works. It does not tell you why it works that way. That is what memory is for. It exists so that your AI does not keep undoing past decisions when it writes or refactors code.
kaydub · · focus · HN ↗
Old decisions always end up in the docs. If the LLM gets a whiff of an old decision, but doesn't get the update to that decision, well now you're doing things the old way again.
dregitsky · · focus · HN ↗
I liked this advice when humans wrote code. Though even then I'd urge people to write meaningful commit messages that capture the "why" of what they did, so no one tramples their intent by mistake.
But not sure it works in an age where most code is LLM-generated. Especially if that code is not even reviewed by humans (irresponsible or not, it's happening), and commit messages are also generated by AI. I think something is needed to separate "what did the human operator intend" from what the agent went and built.
I do agree that this gets way overengineered. My approach has been more or less what you stopped doing though - committing all our timestamped "plan/implementation docs" and "investigation docs" that document what the user wanted + empirical findings, and making all prior session transcripts searchable. It's seemed mostly helpful? For whatever reason I haven't run into many staleness problems so far.
kaydub · · focus · HN ↗
I'm mainly aiming my frustrations at all the markdown files being committed, all the additions to knowledge bases, all the comments in the code (especially the ones referencing specific JIRA tickets). This stuff isn't helpful, it gets hella stale. I've had the LLM fuck up plenty due to these docs and comments.
Commit message ARE EXACTLY where architectural decisions or nuance should go. Not another fucking .md or more comments.
jjfoooo4 · · focus · HN ↗
kaydub · · focus · HN ↗
Make a high level .md, let it rip, iterate. You can give specifics in your original prompt.
I don't need to document the framework or the libraries etc. Pre-code, I tell it in the prompt one time. After it inits the project it's in the code.
mmcnl · · focus · HN ↗
kaydub · · focus · HN ↗
If it's not, that's what commit messages are for.
hosh · · focus · HN ↗
When it is not, it has to be reasoned out and does not work well for documentation.
Other things that code and tests alone do not capture well:
- promises (as in Promise Theory) made to other parties. Claude already has PT in its training data.
- Constraints-inducing-properites, as in Roy Fielding / Christopher Alexander. While tests, and property testing can capture properties, there is no formal connection to the constraints that induces fhem. By constraints, I am not talking about business requirements and business value — those are better understood through Promise Theory. I am talking about things like at-least-once delivery or total ordering (from append-only constraint). Claude already has Fielding’s dissertation and Alexander’s works and ideas in its training data.
- grammars, as in pattern panguages (not just patterns) a la Alexander / Fielding are also not captured in code alone. These tell both humans ans AI how to extend a pattern, and how to identify anti-patterns (when they violate a constraint-inducing-property)
- LLMs are trained with many different worldviews and bounded contexts at the same time, and is very capable of translating across it. However, these need to be soelled out, otherwise it would talk in whatever it infers
Specifications written for the exact way components are wited together run into that stale doc problem. Although it takes much more human attention and token burn to describe things in terms of pattern language and promise theory, it becomes easier over time. The actual implementation plan tends to fall out more cleanly when all those other stuff are at least considered. This is where I have been spending most of my time.
chaostheory · · focus · HN ↗
kaydub · · focus · HN ↗
chaostheory · · focus · HN ↗
01100011 · · focus · HN ↗
In my codebase it is difficult to get agreement on comments and documentation so rather than rely on it I adapted. One of the first things I did when I succumbed to agentic development was to point codex at the code and ask it to generate a high level description of where important files, such as our public API, reside, what the hierarchy is, what the code does, etc. In my case, this level of documentation is fairly static if I avoid implementation details. So now I have a handful of agent files in my tree and it seems to save quite a few tokens and improve my results. I frequently have other devs ask me how I get such good results when doing agentic reviews of their changes(always my first step now before I start my human review). I also include instructions in the agents files instructing the agent to maintain the agent files if any relevant changes are made. It seems to work quite well for me.
kaydub · · focus · HN ↗
If you got the LLM to generate the docs, they don't need the docs.
mgfist · · focus · HN ↗
All code is written under constraints, and most constraints live outside the code.
kaydub · · focus · HN ↗
When it's not, there are commit messages.
Please for the love of god, quit generating markdown files (especially having the LLM generate the file, because if it could generate it, it doesn't need it), quit generating "decision" docs, and stop having more comments than code.
mike-akdeniz · · focus · HN ↗
[dead]
ChimpWithHat · · focus · HN ↗
kaydub · · focus · HN ↗
It's a LOT of cargo-culting overly complex AI workflows. It's devs/engineers making rube goldberg machines.
Everyone is doing their own little rain dance and when it rains they say "see, I told you it works"
shinokami · · focus · HN ↗
[dead]
JohnBooty · · focus · HN ↗
Trivial example: LLMs struggled with QA on our app. They didn’t know how to find the seeded test accounts. They would create new ones and do it wrong, or find the seeded test accounts but not know their passwords because they were encrypted, so they’d change the passwords but not tell the other agents. Shitloads of tokens burned. It was a no-brainer to just give them the credentials in a skill that gets loaded when they do QA.
You could say that’s an environment issue, not a code issue. But as far as actual code IME at a minimum we have to tell the agents our general repo structure and architecture patterns.
The days of “you are the world’s greatest Python programmer, write good code” or whatever are over (if that style of prompting even worked in the first place) and as I said less is more. But, still….
kaydub · · focus · HN ↗
Yeah, that's exactly what I'd say. Or maybe you're just approaching the problem now.
You have to prompt these QA agents, correct? Why not give the instructions on where to get credentials in the original prompt?
> But as far as actual code IME at a minimum we have to tell the agents our general repo structure and architecture patterns.
I did say I'll keep an AGENTS.md or CLAUDE.md. I keep that SUPER high level. The most depth I'll give is info about example projects (That the llm can reach using an MCP) to follow for architectural patterns.
JohnBooty · · focus · HN ↗
Another concrete example would be codebase conventions. We prefer lean models. Cross-model concerns go into service objects. Without this direction LLMs tend to default to stuffing too much code into the models themselves, as is the de facto standard for most MVC apps and therefore this is how LLMs are trained.
Even if they could discover our large codebase's conventions flawlessly on their own in every session, this absolutely would require nontrivial work repeated in every session: quite a few turns grepping, conversing with LSPs, or whatever.
So yes... we could describe our conventions in every single session by typing it right into the prompts... right after typing the directions to find the test credentials... and the other ten or twelve things we'd be telling it every single time...
kaydub · · focus · HN ↗
Even with your documentation, the llm is gonna do a lot of that grepping and discovery. Unless you have such comprehensive documentation that it's basically code itself... in which case, it should just use the code as documentation.
The good news is that I don't think what you guys are doing is going to be really bad. It's just not near optimal and it's creating this weird dogma around all these "tools"
JohnBooty · · focus · HN ↗
kaydub · · focus · HN ↗
Locally? Our devs can do whatever they want. For anything I'm working on, for local deployments and testing, I generally build it out so it's set in Makefile/Taskfile. If you're running your determinstic tests locally it should really just be a single command that bootstraps everything and runs the tests, or updates a certain part of your app and runs the tests. The LLM shouldn't need credentials in that instance.
Regardless I'm deploying a local stack of some sort all the credentials and everything are probably stored locally or in an env var where the llm will have access. So if I was having the LLM drive a browser during active development, it can look at the files.
Sure, I guess we could put in the top level markdown more details about this... but why? It takes little time or context for it to figure it out. We have shit change so frequently that it's just something else we have to maintain.
soltanov · · focus · HN ↗
kaydub · · focus · HN ↗
You guys are just reinventing the wheel and it's fucking square.
soltanov · · focus · HN ↗
[dead]