Thread

Agents don't need memory, they need documentation

357 points · 254 comments · kmeh

Loading the complete thread in the background. This saved snapshot is available now. Refresh

michaelastreiko · 1791049291 · focus · HN ↗
[dead]
jdw64 · 1791065519 · focus · HN ↗
Peter Naur argued in his famous essay Programming as Theory Building that documentation alone cannot fully capture or preserve the complete mental model behind a program.
However, AI works differently from humans in that much more of its working context has to be made explicit. Because of that, there may be some fundamentally different way for AI to maintain or reconstruct a program’s overall model.
1. iamwil · 1791089398 · focus · HN ↗
  [delayed]
  1. jdw64 · 1791098956 · focus · HN ↗
    [dead]
  2. nottorp · 1791105848 · focus · HN ↗
    You don't even consciously do most of it... your brain retrains part of it on the program.
    LLMs don't retrain, it's too expensive atm.
alienbaby · 1791065713 · focus · HN ↗
I will say, having built something similar for tracking 'memory' and items at home, it can quickly consume your tokens when dealing with both reading and updating, keeping stale info relevant etc.. when the amount of data starts to grow. Smaller tasks can balloon in their token cost as documents a read, updated, collated, refreshed etc..
however, I have found keeping a good solid reference to my home infrastructure, services, ci/cd setup, hosts, storage , networking etc.. really works wonders as a set of 'memories' to share across projects that I expect to be tested / deployed / acceptance tested etc.. using the home infra bits and pieces.
1. kaydub · 1791134298 · focus · HN ↗
  Most of these things are doing just like you said, taking up context.
  The code is the documentation. There's no need for most of this stuff.
2. stavros · 1791145662 · focus · HN ↗
  I wrote a small utility to record the decisions you made, and the paths you didn't take:
  <a href="https://github.com/skorokithakis/gnosis" rel="nofollow">https://github.com/skorokithakis/gnosis
  This just does a pass at the end, rather than consume context all the time. I've been using it for months and it works great.
espeed · 1791065869 · focus · HN ↗
For every prompt and response, I extract each semantic statement. Map its reasons in a Whybase proposition tree -- a recursive proposition tree where each atomic statement is proposition with one or more premises (atomic statements, which also stand alone as propositions). Then I map each statement to the relevant code, hinted at by tool calls and git commits. Every time an agent touches that file or directory, a hook triggers in Claude Code that queries the codegraph db for the mapped statements. This helps the agent remember something I said in June when it revisits the code in July.
1. nextaccountic · 1791066507 · focus · HN ↗
  What if the June conversation is outdated and is no longer applicable by July? Do you have a mechanism for dropping older, subsumed propositions in your database?
  Github Copilot had the idea of attaching memory to files, and if the file hash changes the memory is automatically dropped (not sure if they still do it). This means they are overly eager to drop stuff (even if the file change is just cosmetic), but at least they don't accumulate outdated cruft too much. (a memory can still be outdated if it was invalidated by a change in another file though)
  1. espeed · 1791067277 · focus · HN ↗
    Yes, the code and spec system are refined until they agree. How this is done is a work in progress. Think of my statements over time as the raw material into an evolving spec. The spec is refined as you learn and the code evolves. You and the agent loop until the code matches the spec.
monneyboi · 1791065878 · focus · HN ↗
I never understood memory solutions for coding agents.
You have the whole session history right there. One recall skill and some JSON parsing gets you grep over perfect memory. Why would you ever use more tools to spend more tokens to construct a imperfect memory next to your session history?
I just don't get it.
1. ceejayoz · 1791072532 · focus · HN ↗
  > You have the whole session history right there.
  But that's one session. Isn't memory for… the next session?
  1. unlikelytomato · 1791077916 · focus · HN ↗
    the code seems to fill this need, for me
2. dboreham · 1791076355 · focus · HN ↗
  It's just an optimization. You might not pay it to re-read everything. So it sticks some stuff in a notebook that it always reads. Like inspector Poirot.
gregwebs · 1791065920 · focus · HN ↗
Agreed, and this seems better.
My thought though has always been that I don't want there to be agent-only designated documentation.
I use mattpocock/skills and that generates ADRs (Architectural Decision Records). That only uses skills, including a setup skill that will write a few pointers in AGENTS.md. I always have a CONTRIBUTING.md to document development flow and a CODING_STANDARDS.md. Between those and the README.md and architecture documentation and commit messages the agents seem to be able to find and use docs and keep them up to date. We are also writing a lot of specs and putting those in Github issues.
1. cyanydeez · 1791071590 · focus · HN ↗
  I've got basically a loop of docs, test (TDD) and code. Starting with and IMPLEMENT-<plan name>.md. I ask to revise as TDD, then loop.
  Once qwen3.8-flash-next showed up, it cañ go "forever" with dynamic context pruning.
  It's fascinating for local coding.
  1. j45 · 1791073089 · focus · HN ↗
    A lot of agentic software development feels like cowboy coding, set it up, let it rip and see what it figures out.
    The slightest amount of guidance, input from experience can make a huge difference.
    cyanydeez · 1791114895 · focus · HN ↗
    yeah, to some extent. I've been pulling repos to play around but I've now got a hard fork of paseo; I ripped out anything related to github. I upgraded the relay with security features to prevent abuse. I'm in the process of adding the first "real" large feature update that'll let the agents work over the relay with one another which will let me add compute wherever I find it. In the process of adding a browser plugin that will let me pull up pages for context to work on whatever (I don't trust even agents for searching the web).
    And it's quite fascinating. Still dont think there's trillions of TAM out there if Qwen3.8-Flash-Next runs fast enough on $3k (before memory cartel) pricing.
    girvo · 1791123449 · focus · HN ↗
    Q38FN is absolutely a step change IMO. Literally better than Opus 4.6 for the work I’m using it for, but I can run it forever for free on my GB10 box. Wild.
    And it’s an experimental likely undertrained model. Wait til Qwen 4 Flash…
    cyanydeez · 1791137573 · focus · HN ↗
    I assume it was put out to get inference engines ready to do qwen4 models. For AMD halo, the Halgogen engine is the fastest i dound so fare
  2. jon-wood · 1791116371 · focus · HN ↗
    I’ve been trying to nudge agents (both Claude and GPT) into a red/green/refactor TDD loop and I’m increasingly unconvinced it’s worthwhile. TDD works for humans because it forces us into a pattern of doing the simplest thing that could work, then thinking about how to refactor that. An LLM can be prompted to do that without the skeleton of a failing test, and I plan to experiment with other methods of pushing them towards the same ends.
    gregwebs · 1791117543 · focus · HN ↗
    The value of TDD is it does ensure that tests are written and that will help with future regressions.
    However, I found the same problems with agents that it has with humans (usually with humans it isn't TDD but code coverage requirements). The problem is the tests are now written to satisfy a bureaucracy rather than to properly verify the code. And there are real studies that point out the low value of these types of bureaucratic testing requirements. For example, when models are just required to write a test file, they don't produce a better result: <a href="https://arxiv.org/abs/2602.07900" rel="nofollow">https://arxiv.org/abs/2602.07900
    I wrote a /verify skill that focuses on properly verifying code and I am finding it works a lot better. I do need to revise it now- in practice certain parts of the skill are doing all the heavy lifting and others are more dead weight. But it does seem to be properly orienting the agents towards finding defects. <a href="https://github.com/gregwebs/skills-sdlc/blob/main/skills/verify/SKILL.md" rel="nofollow">https://github.com/gregwebs/skills-sdlc/blob/main/skills/ver...
    cyanydeez · 1791117757 · focus · HN ↗
    don't forget: tests also inform the agent on what the code is _supposed_ to do. So with the cycle, you end up with 3 different potentially robust ways to look at and understand how to extend.
    Context then is: docs, code and tests. Rather than try to build some omniscient agent, the context is built where the agent needs it.
    gregwebs · 1791121647 · focus · HN ↗
    I agree that ensuring testing is the basis for modifying code in _future_ development (including understanding it). My problem is that we can't produce a study showing that TDD is producing better results _now_.
    TDD doesn't help the agent figure out what it is supposed to do now- it is already writing the code and now and it must know what it is supposed to do to write it.
    I am trying to use evidence-based approaches. I am only observing better results and tweaking now, but I plan to benchmark when tweaking is done.
    cyanydeez · 1791136587 · focus · HN ↗
    [dead]
    cyanydeez · 1791117663 · focus · HN ↗
    I'm working on local models; qwen3.8-Flash-Next seems to be passed to rubicon.
    I'm working on javascript, and now I'm only doing this via typescript. I've successfully got this going:
    1. Design a feature in plain language with the coding agen (opencode) and write a document for it.
    2. Restart the context (or I use /compact to flush any errant details)
    3. Pull the new plan into context and ask the agent to revise it follow Test Driven Development.
    4. Depending on how big it is, the agent places it into multi stages, each with it's own document.
    It then loops through the stages. It might be entirely based on the language you're using, but this loop seems strong enough.
    Then when there's bugs, errors, anything, we update the doc, add more tests, then revise the code.
    It's quite possible the harness you're using isn't setup properly. For me to do this locally, I had to hack on dynamic context pruning which I described here: <a href="https://news.ycombinator.com/item?id=49906637#49907641">https://news.ycombinator.com/item?id=49906637#49907641
2. gojogs · 1791109981 · focus · HN ↗
  I've found that most of LLM generated docs are diluted and unfocused. Spending 3 paragraphs on some quirk of a library, then 2 simple examples of a command.
  I write README and docs by hand, thus only important stuff goes in there. If it is not worth my attention to write down, it's not worth writing down.
  Smaller context, easier to consume.
bushido · 1791066724 · focus · HN ↗
Something I started doing recently was writing out principles instead of memories.
Essentially patterns the agents need to always think in. I also implemented a versioning system to the principles that need to be quoted in any comments which are there in the code. That way, when my principles evolve, so does the code.
I did package it up in a way that I can share it with friends [0]. System still evolving, but the last two-ish months that I've used it has served me really, really well, And it's been even better with the latest models.
I've had surprisingly good adherence from agents on this technique.
[0] <a href="https://principledriven.dev/" rel="nofollow">https://principledriven.dev/
1. skybrian · 1791067207 · focus · HN ↗
  [delayed]
  1. bushido · 1791071315 · focus · HN ↗
    Fair feedback. I admittedly didn't spend enough time making it simpler.
    The github repo might be more helpful: <a href="https://github.com/Principle-Driven/pdd" rel="nofollow">https://github.com/Principle-Driven/pdd
    skybrian · 1791079088 · focus · HN ↗
    That's better.
    I was wondering about this bit:
    > "Code comments cite a versioned token where code depends on the rule."
    What does a token look like? I went looking for an example, but I don't know what I'm looking for.
    bushido · 1791081789 · focus · HN ↗
    From a different codebase, one of the behaviors in the code has the following comment:
    // The binding deliberately SKIPS the contradiction check: a form // bound to an object the same document deletes is a legal stale // state (PDD-3@v1)
    Where PDD-3 deals with a specific type of race conditions that my agents were over-optimizing/spinning out on; Now if I ever change the rule the version would change to PDD-3@v2; which triggers a re-evaluation for all decisions that were made under the previous version.
2. visarga · 1791094079 · focus · HN ↗
  I wrote a cli tool the agent is instructed to call. The tool collects answers to a few questions from the agent, and responds with a steering command. Sometimes it responds with more questions, and then shows the command.
  The questions are designed specifically to identify the state the agent is in, in order to assign its steering. Changing this tool changes how the agent works. You just need to make the use of this tool a necessity so it does not forget to call it.
  Unlike principles and memories, questions are more openended and tend sometimes to trigger the right mentality in the agent even before it gets to the policy itself. In my opinion agents get lost in local work losing the big picture, my questions jog the big picture back into attention.
  1. call ssp tool, no arg, it just reads features from the repo, like most other tools -> ssp locates state and sends 4-5 questions
  2. agent responds the first batch -> state is further narrowed down -> ssp sends a second batch of questions
  3. agent responds again to the interview -> state gets finally pinned down -> agent gets the steering assigned
  4. a log is created of this ssp session, and reflection on the log used to refine the questions and steerings
3. locknitpicker · 1791099783 · focus · HN ↗
  > Something I started doing recently was writing out principles instead of memories.
  It seems you tried to reinvent instruction files. What do you think is the difference between your approach and standard tools such as instruction files, AGENTS.md, and even skills?
4. mzhaase · 1791101726 · focus · HN ↗
  I've done that too, called it Decisions. Was happy for a while, but then the agent started to make decisions longer and longer, putting "x, not y" in there. It started to write them without asking. Suddenly in code review it would say "violates D-236". Some rule it made up that I never approved. Then I made very clear that it should never touch those files. Still does. Now it's just context bloat.
  1. bushido · 1791114704 · focus · HN ↗
    I can't imagine ever getting to 236. My largest codebase has 22 principles. And I'm already struggling to add a 23rd. I have some candidates, but I'm on the edge of where it get's complex. With a lot of effort, I could maybe see this getting up to 30.
    If it helps, here's how you think about it, principles are not decisions, they're mental models and frameworks to allow making decisions. In my case, they're there to allow agents to make decisions without asking me. But the decisions cannot be recorded - the code serves that purpose. What can be recorded is why the commend for the block of code needs to cite what principle was used in the current shape, which is why there is a citation of the principle.
    When the principle evolves, so does every decision, at the very least, every decision gets re-litigated to see if it needs to evolve.
5. PetriCasserole · 1791113004 · focus · HN ↗
  Timely comment! I'm learning harness design and was introduced to the concept of writing guiding principles. I look forward to trying it. (Great site. I agree with another comment. It does look commercial. Had I not been committed to looking for ways people define principles and rules, I might have skipped it. Glad I didn't.)
  Do you have any resources that guided your work?
  1. bushido · 1791126865 · focus · HN ↗
    I have a few other projects which slowly evolved to this point, like <a href="https://github.com/DheerG/swarms" rel="nofollow">https://github.com/DheerG/swarms
    A big part of it really comes to that I have led large product teams and platform engineering teams in the past, And I like giving feedback to my agents in the same way that I would plan things with my teams.
    One of which is give them repeatable mental models which allow them to make decisions without relying on managers or me. The result is usually that the teams can work for weeks-to-months with autonomy.
    I try achieving the same with my agents, so that they can work for 1-14 days without my intervention. I usually have them working on very very long running tasks/initiatives.
    If you do want some resources, this is one of the pages that I've been using for nearly a decade to help people get started with mental models: <a href="https://fs.blog/mental-models/" rel="nofollow">https://fs.blog/mental-models/
vcryan · 1791067885 · focus · HN ↗
Yes, this makes sense. Memory is an uncurated and often opaque system of arbitrary past discussions. It can help, it can harm. Accurate documentation in the other hand is only beneficial.
docheinestages · 1791068639 · focus · HN ↗
Why would I need to install your tool for that? It could be an instruction living in AGENTS.md or with some sort of hook to remind the agent.
1. dboreham · 1791076702 · focus · HN ↗
  You don't. When you see it saying it stored something "in memory" you can just ask it to document it in the product documentation somewhere, and in future please keep doing that. Of course the note to keep doing that needs to go somewhere, which is what "memory" is for.
isaachinman · 1791068697 · focus · HN ↗
I wrote this, after many iterations consulting for various companies. Documentation is queryable in single digit ms, append only log, etc. Has worked exceptionally well for my projects
<a href="https://github.com/isaachinman/encephalon" rel="nofollow">https://github.com/isaachinman/encephalon
1. mzhaase · 1791101442 · focus · HN ↗
  Do agents actually follow it though? Everything I tried that was not just simple instructions had the same failure mode : ADR, mnemonic, hindsight, code-review-graph, etc were simply ignored by agents after a while. They always fall back to their baseline: grep and Python.
  1. bird0861 · 1791116322 · focus · HN ↗
    How to spot a Claude user.
    But seriously, Opus has been garbage after 4.6 -- the public seems very easily fooled into thinking breadth == depth. Anthropic, I'll grant them, has been and is still an extraordinary data team. As for model alignment though... I have to wonder sometimes if they are even trying beyond just chat training.
    "Guys, SI is just around the corner... Huh? What production DB? Anyway GPT2...I mean Mythos is too good it would be dangerous to release to the public!"
  2. isaachinman · 1791156730 · focus · HN ↗
    Yes, they do. Feel free to give it a try. Not shilling or trying to promote anything. Just sharing what has been a huge boost for my productivity.
Neywiny · 1791068832 · focus · HN ↗
This is what I've been saying for years. I attended a tech demo of an AI assistant for a car's owners' manual. But they trained the manual into the model. Which means not only does it need retraining every edition, but it's imperfect. The models need to be trained to fetch and use documentation not vaguely recall infinitely many concepts. I would much rather have 27B parameters on how to code than 26B on stuff like numpy function listings. It's like they approached the problem from a closed notes hand written coding exam. Everyone hates those.
1. kolinko · 1791070870 · focus · HN ↗
  What years? Agentic systems, and memory alongside them are roughly only year old.
  You’re confusing, I think, memory system with llm finetuning. Completely different concepts.
  1. Neywiny · 1791071867 · focus · HN ↗
    No I'm saying this entire problem approach, including memory system of agents, is incorrect. Like the author argues. With the same solution of moving the ground truths outside of the model/context.
YuechenLi · 1791069416 · focus · HN ↗
I thought the agents are mostly self-documenting since they usually write just as much Markdown documentation autonomously as they do code/unit tests, not sure why they would need extra tools here.
1. mcbuilder · 1791071189 · focus · HN ↗
  Because of old outdated info that gets left behind, causing confusion for future agents.
  1. dboreham · 1791076518 · focus · HN ↗
    You can tell it to update the docs to be consistent with whatever changed in the code (or whatever is is the authoritative product form). After a while it usually gets with the program and updates the docs automatically. But not always.
    krishverma_2010 · 1791095130 · focus · HN ↗
    [dead]
2. dboreham · 1791076456 · focus · HN ↗
  You don't. But as was ever the case before AI most humans don't know to or want to have documentation written. So the tools have workarounds to try to reduce the number of "AI sucks because I told it X and it did Y" posts.
3. kaydub · 1791134811 · focus · HN ↗
  The markdown files, documentation, decision files, (3rd party)memory systems, etc. are all a waste of time and context.
  Every single project at work that has documentation/ADRs/etc ends up stale as fuck and it pollutes context more than it helps. Even the agent generated docs. Hell, especially the agent generated docs. It's like the LLMs can't just DELETE something, they always amend. So now there's stale details in there about something we abandoned polluting context.
  The code is the documentation, none of this stuff is needed. I'm seeing WAY too many devs/engineers go down the rabbit hole of trying to build these tools and systems but not actually getting meaningful work done.
  I've been doing my best to remove and delete all the documentation. I usually keep a super high level AGENTS.md/CLAUDE.md file and that's it. The llm/agent can figure it out. You might argue I waste 5 minutes and a dollar or two for the agent to "figure it out" each time, but I'm pretty certain people are wasting FAR more time and money building these things out and maintaining them in each of their projects (or NOT maintaining them and they're actively BAD for their projects).
relaxfolio · 1791073756 · focus · HN ↗
[dead]
spike021 · 1791074113 · focus · HN ↗
I think whichever one is used, there needs to be a way to enforce what's written.
If I say "use jq instead of writing a python script to parse json" it should never write adhoc python scripts to parse json. Yet that constantly happens to me anyway.
1. ACCount39 · 1791074646 · focus · HN ↗
  Obviously, AI is way more comfortable with using adhoc Python scripts, which are used for everything, than it is with using jq, a niche CLI tool.
  1. ozim · 1791109293 · focus · HN ↗
    Maybe it is my web development bubble but jq seems to be far from „a niche CLI tool”.
  2. gojogs · 1791109685 · focus · HN ↗
    we can drop the I from AI then
2. zahrevsky · 1791074986 · focus · HN ↗
  There is a command in oh-my-pi called "/omfg <problem>". You explain what is wrong with agent's response, and it writes a hook to make sure that the problem doesn't happen again. It then re-runs your previous prompt to make sure that hook is triggered, and if not, it rewrites the hook to make your previous prompt trigger the hook. Then each next agent's response is checked by the hook, and if it is triggered, the agent receives feedback on what's wrong and what must be done differently.
  1. aliasxneo · 1791087351 · focus · HN ↗
    I've written a lot of custom hooks this way. It's an amazing feature.
  2. gf000 · 1791094218 · focus · HN ↗
    How does it generally work? Does it run a smaller model on a small set of tool calls/previous thinking block?
    jon-wood · 1791115918 · focus · HN ↗
    Hooks should (in my opinion) be deterministic. As an example I’ve also noticed Claude writing Python scripts to extract fields from JSON when jq is available to it. They almost always contain the same patterns, and having seen this I’m going to write a hook which triggers on those and fails the turn telling it to use jq instead.
    PcChip · 1791123919 · focus · HN ↗
    Why do people care how it reads json?
    rmunn · 1791127379 · focus · HN ↗
    Because custom Python scripts means burning more tokens which costs more money. Reaching for an existing tool like `jq` means burning a LOT fewer tokens.
    spike021 · 1791129091 · focus · HN ↗
    Imagine you have application code with a function to parse some json. But every time you decide to use it, you re-implement the entire function. This is despite it working the same way every time.
    It's not like it's being rewritten for efficiency. "Just because".
  3. amelius · 1791110386 · focus · HN ↗
    Sounds like something that can grow unwieldy.
    girvo · 1791123058 · focus · HN ↗
    Such is the joy of trying to get determinism out of a non-deterministic system…
3. dboreham · 1791076264 · focus · HN ↗
  I think it's kind of cute the way it writes Python scripts, but I've never seen it do that when the relevant native tool is on the PATH. It's like the most competent ever intern, on speed. No tool to convert SVG to PNG? No problem, I'll write a Python program to do that!
  1. locknitpicker · 1791099657 · focus · HN ↗
    > I think it's kind of cute the way it writes Python scripts, but I've never seen it do that when the relevant native tool is on the PATH.
    You haven't been paying attention then. I routinely see Claude and GPT models churning out python code to do stupid things like linting. Last week I even had a TypeScript project with prettier configured all over the place, including in a custom agent skill I added with the express purpose of getting the damned model to lint the code, being constantly prompted to run ad-hoc python code supposedly to format whitespaces. I even explicitly prompted one session to just use npm run lint, where I pointed out the exact line of code where prettier was invoked, and the session still churned python code to hande whitespaces.
    yulaow · 1791111298 · focus · HN ↗
    My experience is the same, I can put that decision in the prompt, in a skill, in a hook, in agent.md, etc, it doesn't matter, after few iterations it starts again using useless python scripts to do anything from linting, to error checking, to parsing one liners of code, etc etc...
4. jkhdigital · 1791080326 · focus · HN ↗
  Treat it like any other software system: rules that must not be violated are enforced by static type-checking or a trusted runtime monitor. There’s no other option.
  1. koolba · 1791084347 · focus · HN ↗
    Except you can’t do that unless the runtime itself can reason about what’s being executed.
    Otherwise you can get a python one liner that execs a different script engine.
    devmor · 1791093865 · focus · HN ↗
    You absolutely can, that’s what your harness is for. You don’t need your environment to “reason” about things when deterministic tools exist - You have a really fancy hammer, but that doesn’t make everything a nail.
    TZubiri · 1791097220 · focus · HN ↗
    But what if we used the fancy hammer to change the shape of everything to be a nail? And what if we build the handle of the fancy hammer with a fancy hammer? With all of this we could build a very good fancy hammer manufacturing company.
    Terr_ · 1791098853 · focus · HN ↗
    [delayed]
    altmanaltman · 1791100748 · focus · HN ↗
    Yet the hammer seller continues to scream everything is a nail and their hammer will replace your entire job eventually. So are you telling me the hammer seller is lying or am I the one using it wrong?
    user_of_the_wek · 1791099815 · focus · HN ↗
    I mean you don’t _have_ to make python available to the agent. Nor bash.
5. sick_of_slop · 1791086465 · focus · HN ↗
  [dead]
6. locknitpicker · 1791099349 · focus · HN ↗
  > If I say "use jq instead of writing a python script to parse json" it should never write adhoc python scripts to parse json.
  I think there is a deeper problem emerging from this sort of behavior. Even when we bother to create agent skills with there own scripts that call tools like jq a specific way to achieve a goal, AI coding assistants and agents still go way out of their way to generate ad-hoc scripts to do the most absurdly stupid tasks such as parsing output in structured language, and even remove whitespaces from a markdown file. This means AI coding assistants and coding agents treat agent skills as mere suggestions of using a alternative option that more often than not the choose to ignore.
  This has a very dangerous implication: your average user is trained to develop a pavlovian reflex to authorize agents to just execute their ad-hoc scripting code with our own permissions and credentials in our systems, which includes the ability to call anything over the internet.
  1. rojaneerdev · 1791125008 · focus · HN ↗
    [dead]
7. ozim · 1791109631 · focus · HN ↗
  Claude code setup seems to be using jq all the time when I use it but I did what is written here:
  <a href="https://gist.github.com/cynthiateeters/6868ca26c059a3106cd93736d33015af" rel="nofollow">https://gist.github.com/cynthiateeters/6868ca26c059a3106cd93...
8. HisashiSpace · 1791130421 · focus · HN ↗
  [dead]
9. fennect · 1791136975 · focus · HN ↗
  You could try <a href="https://github.com/ioni-dev/mati" rel="nofollow">https://github.com/ioni-dev/mati the constant changes in reasoning effort in consumer models can break things and its more dangerous for devs that relay to much on agents.
  I’m the author of the project.
10. triyambakam · 1791137055 · focus · HN ↗
  It's often better to just find ways to embrace what it tries to do naturally. Otherwise you're fighting the weights and hidden prompts
  1. joquarky · 1791175741 · focus · HN ↗
    Wu wei.
11. eivindmeyer · 1791139390 · focus · HN ↗
  [dead]
kadhirvelm · 1791074825 · focus · HN ↗
We’ve been working on exactly this, deriving documentation from external systems (like GitHub, etc) into a giant set of docs that agents can reference and edit. Works way better than I would’ve originally expected. Suddenly these things are able to reference granola notes, a slack discussion, and an RFC when making coding decisions. Super helpful in a lot of unexpected ways!
greatergoodguy · 1791074914 · focus · HN ↗
Using Opus 5.5, I've ended up recreating a version of the Hugging Face incident. My project has a folder called agent-handoff where agents write about the tasks they're working on. They post status updates, decisions and screenshots, and they even claim which emulator they'll use to test their work. It has turned into a hub where all the agents talk to each other, and it's scarily effective.
Since then I've gone down the rabbit hole of really digging deep into current AI research and especially what AI whistleblowers are currently saying. And I can't even express how existentially scared shitless I am.
1. mewpmewp2 · 1791075792 · focus · HN ↗
  I just don't get the fear at all. I too have 10 to 50 agents working 24/7, from 5+ different providers. They talk to each other directly using tmux send keys. What is the exact frightening thing here? It is all tokens following tokens.
  1. dingaling911 · 1791076137 · focus · HN ↗
    The frightening thing is how easy it is to make copies of things that can reason without the requisite investment, and how easy it could be to direct them to bad things.
  2. altcognito · 1791077652 · focus · HN ↗
    I haven't tried what you guys are talking about myself, but I think when agents begin talking to other agents, the chances for unpredictable goal mixing and confusion resulting in really unexpected and bad behavior is a lot higher.
    That being said, the danger is measured in computer damage, which can be a lot personally and to a company, but less existential, so your mileage may vary as to how "scary" it is.
    Xirdus · 1791079041 · focus · HN ↗
    When nearly all money in the world is stored in computers, the danger is quite existential.
    throwaway27448 · 1791083919 · focus · HN ↗
    [delayed]
  3. Forgeties79 · 1791083490 · focus · HN ↗
    Tokens following tokens can inadvertently brick critical systems and get people killed.
    apsurd · 1791089417 · focus · HN ↗
    They're different groups though.
    Forgeties79 · 1791116861 · focus · HN ↗
    Skeptical was probably a poor word choice. The point is if it’s negative, “it’s just tokens.”
2. iamwil · 1791089247 · focus · HN ↗
  [delayed]
zahrevsky · 1791075916 · focus · HN ↗
Although I agree that current memory implementations don't help at all, I don't agree with this analogy:
> No one rewatches a team meeting from 3 years ago to remember constraints around a feature. People write things down and use those records instead.
Memory plugins don't re-read old transcripts. Memory snippets are basically the notes that people write down after the meeting.
The problem, however, is different. The problem with memory plugins is that your agent basically writes a note every time someone says a sentence, and then tries to work with those 5000 notes.
Instead, the agent should recognise what's important and write only that. And that is, of course, a documentation. (And ADRs, if you want not only a description of the final state, but also the trajectory of how we arrived to it. Which, arguably, contains more information than the docs alone.)
Of course, another difference is that notes are immutable, append-only and don't have a lot of structure. This is of course done to be able to store lots and lots of notes: they should be independent. The main problem is that increasing the number of notes adds not enough benefits to compensate for downsides of this structure-less immutable format.
1. popalchemist · 1791076762 · focus · HN ↗
  There is a hierarchical distillation of understanding in human processes that agentic processes do not yet execute.
  1. nijave · 1791127284 · focus · HN ↗
    There are some systems that do that, but I don't know how effective they are. I've been meaning to evaluate AWS AgentCore Memory which has strategies you can apply for reconciling and organizing memories.
2. pornel · 1791078195 · focus · HN ↗
  I don't trust agents writing specs without human approval.
  I've been bitten by agent-written ADRs. Agents carelessly add extrapolated details and speculated nice-to-haves that I never asked for, and this becomes a source of bloat that keeps coming back like a boomerang.
  1. zahrevsky · 1791081155 · focus · HN ↗
    Of course I meant you review the docs the agent writes (as well as everything else the agent writes)
  2. sathish316 · 1791084286 · focus · HN ↗
    I’ve found that Agents cannot differentiate between a general principle or abstraction to be followed vs a one-off code review comment that’s applicable only to a single PR.
    Lack of this capability makes automated principles or patterns update a recipe for more bloat.
  3. otterley · 1791090954 · focus · HN ↗
    Why are you having LLMs write them in the first place?
jen729w · 1791076100 · focus · HN ↗
The solution to this problem is as old as computers: it's a folder. Just use folders.
`cd` to a folder. Launch `claude`. Do your work. Save scripts and documentation in that folder. `/resume` previous conversations from that folder.
That's it. That's the trick.
Now, having very static, very well-defined folders helps a lot. I'm Johnny.Decimal so I have numbered folders for everything I do. So my process when I want to use my 'process a travel booking from my email to my calendar' script is:
- `jd tripsy`
```
  - The folder name includes &#x27;tripsy&#x27; and this is how I remember it.

  - `jd` just parses my limited tree and `cd`s me to a folder.

  - `jd 21.15` gets me there by number if preferred.
```
- `claude`
- Say 'hey Claude, there's a new email in my inbox please'.
- Done.
1. nijave · 1791129844 · focus · HN ↗
  Consider tomatoes.
  Legally and culinarily, they're vegetables. Botanically they're fruit.
  Tomatoes are members of the nightshare family which includes tobacco, potato, and chili peppers.
  They are used in popular recipes like Mexican salsa and Italian pasta sauce. Pasta sauce commonly uses the San Marzano variety. Here's a <picture> of San Marzano tomatoes from our Italy trip.
  Rhett likes tomatoes in all dishes. Link only likes tomatoes if they're blended in a dish like pasta sauce or tomato soup. Frank is allergic and can't have tomatoes.
  Tomato prices are up due to a bad <year> yield in <country>.
  ---
  This is all memory. Where does it go?
  1. jen729w · 1791157778 · focus · HN ↗
    Wikipedia?
    What is it that you are doing with tomatoes today? Store that locally. Sounds like you're cooking. I put those here, then I can find them again really easily.
    10-19 Life admin/ └── 12 Where I live & how I get around/ └── 12.44 Meals & recipes
mcapodici · 1791076536 · focus · HN ↗
I don't use memory. I am really not keen on the idea of having this hidden context fed into the LLM, and I prefer to use docs, both markdown and stored in a wiki like Confluence. Using repos and wikis also lets you scope the information so hyper-specific memory doesn't affect other projects.
If I want the LLM to remember something I ask it to update some docs, and even check that docs are consistent across the board after doing so.
The only thing I want the LLM to remember everywhere is talk like a human (no load-bearing, not this/that etc...), so I have an AGENT.md for that.
jsemrau · 1791076895 · focus · HN ↗
Documentation is memory.
folayii · 1791077063 · focus · HN ↗
[dead]
[deleted] · 1791077217 · focus · HN ↗
[deleted]
chaostheory · 1791077928 · focus · HN ↗
[delayed]
jmtulloss · 1791079106 · focus · HN ↗
Evals or it didn’t happen.
Snarky comment aside, I am very interested in how we evaluate the performance of these systems and what kinds of work match best with different approaches.
lewelove · 1791079230 · focus · HN ↗
Obligatory 927.
<a href="https://xkcd.com/927/" rel="nofollow">https://xkcd.com/927/
favori995749721 · 1791079693 · focus · HN ↗
[dead]
ramoz · 1791080230 · focus · HN ↗
I think with enough structure (written rules and good folder layout), poly/monorepos are the way.
<a href="https://backnotprop.com/blog/context-monorepos/" rel="nofollow">https://backnotprop.com/blog/context-monorepos/
```
  - Better models can keep the drift in check.
  - What&#x27;s super important these days is having all historical context, historical decision making, prototypes and whatnot.
  - all projects and their worktrees located together
  - all reference docs, code, etc - a folder away
```
When I ask "what happened to x?" ... the agent has everything it needs to give me that answer. When it plans the next feature it can validate assumptions against previous decions made in my `decisions` folder.
1. kaydub · 1791134415 · focus · HN ↗
  Yeah, I think monorepos are pretty great. Sometimes not possible, but for those situations you just need to have a way for the agent to get access to other codebases (gitlab mcp being what I rely on at work).
  I don't think you need more documentation or these memory systems. The code IS the documentation. Lots of this stuff is unnecessary. It's a bunch of people doing their special rain dances and then when it happens to rain they say "I did that"
nicwolff · 1791081060 · focus · HN ↗
Didn't Cline formalize the "memory bank" way back in February 2025?
<a href="https://cline.bot/blog/memory-bank-how-to-make-cline-an-ai-agent-that-never-forgets" rel="nofollow">https://cline.bot/blog/memory-bank-how-to-make-cline-an-ai-a...
minimaxa · 1791081426 · focus · HN ↗
[dead]
stbenjam · 1791086868 · focus · HN ↗
[dead]
DriverDaily · 1791087796 · focus · HN ↗
The brain has mechanisms that can organize experiences without requiring that relationship to be expressed as a sentence.
Like, you can quickly lookup related ideas based on what came before and after, causes, effects, just like calling relationships a graph database.
Documents can’t be queried efficiently like that, you need a database.
1. apsurd · 1791089003 · focus · HN ↗
  I get what you're saying but it seems presumptions to compare a graph database to how our brains work. And also that LLMs <-> (the way they do memory) is the right analog to humans <-> memory.
  I take the article's point more directly. It's just a straighter line to have clear communication through documentation than to fuddle around with the perfect memory setup.
Garlef · 1791090364 · focus · HN ↗
I think they even more so need deterministic feedback:
I tried an approach based on the following idea recently and it's amazing - Lint rules where the error messages contain an explanation on how to deal with the issue.
<a href="https://habit-hooks.com/" rel="nofollow">https://habit-hooks.com/
I'm using it to foster IOSP (integration operation segregation principle) for example.
1. jghn · 1791094948 · focus · HN ↗
  The readme reads like AI slop. Why would one believe the tools would prevent AI slop?
  1. Garlef · 1791097698 · focus · HN ↗
    Here's a link to a video where the creator explains the idea
    <a href="https://youtu.be/6AgndHSkHFI?t=238" rel="nofollow">https://youtu.be/6AgndHSkHFI?t=238
2. jcjmcclean · 1791099477 · focus · HN ↗
  This looks pretty cool but I'd want to be able to setup a bunch of my own project specific code smells, so it's not just a few generic rules. Is that the approach you've taken?
  Then for example you could write your own hook and convert existing code smell documentation which agents ignore into a format that works with the hook.
  1. Garlef · 1791099670 · focus · HN ↗
    > but I'd want to be able to setup a bunch of my own project specific code smells
    That's what I did; I did not use the library I linked to ~ It served only as an inspiration.
    Instead, I let the agents create custom lint rules (using eslint, pylint, ...) and add custom coaching error messages based on where I want to take my codebase.
    jcjmcclean · 1791099753 · focus · HN ↗
    Really nice idea, I'll give this a try tomorrow. Thanks for sharing!
3. spacebanana7 · 1791100545 · focus · HN ↗
  I suspect strategies like this might be even more powerful with less intelligent agents. Could a 40B model outperform a 400B model with good feedback and instructions?
4. fxtentacle · 1791101965 · focus · HN ↗
  Deterministic feedback is precisely how frontier models are trained. It’s called RLVR. You let the agent run on a problem and then calculate a deterministic score of how well it did. Repeat 1000x times and you can “brute force” a good solution. (Which includes all thinking traces and you add it to your training data.) And then a Chinese model can copy your advance for 1000x less compute. Which is why US labs call this not learning, but a distillation “attack”. It’s an attack on the business model.
  1. [deleted] · 1791105672 · focus · HN ↗
    [deleted]
  2. DelightOne · 1791110579 · focus · HN ↗
    Are there good open source setups that generate this training data automatically?
    Or is that the secret sauce no one wants to share
    fxtentacle · 1791138005 · focus · HN ↗
    Labs typically pay $2k for each [prompt+scorer] docker image. So this is why frontier labs need so much cash and manpower and they see it as their moat.
    But some of the resellers have freebies, like:
    <a href="https://app.primeintellect.ai/dashboard/environments?ex_sort=by_sections" rel="nofollow">https://app.primeintellect.ai/dashboard/environments?ex_sort...
    <a href="https://github.com/sierra-research/tau2-bench" rel="nofollow">https://github.com/sierra-research/tau2-bench
  3. balder1991 · 1791125605 · focus · HN ↗
    I suppose the same way that the normal programmers and artists would call LLMs an attack on licenses and copyright.
5. OJFord · 1791109200 · focus · HN ↗
  This sounds interesting but (from the homepage too) I don't understand how it's different than having it run any other linter?
  1. szundi · 1791109568 · focus · HN ↗
    [dead]
  2. Garlef · 1791110318 · focus · HN ↗
    The difference is that the error messages contain instructions on how to resolve the issue.
    Just asking for small functions will make the agent write small functions ~ but not in a good way (for example they just cut `doOneThingAndTheOther` in half and call the second half `doOneThingAndTheOther2`)
tankiya · 1791094910 · focus · HN ↗
[dead]
suki_buildz · 1791094927 · focus · HN ↗
[dead]
jason1cho · 1791095515 · focus · HN ↗
I think many AI fanatics will focus on a wrong point. They will say "just use AI to generate the fucking documentation. What are you talking about?"
Indeed, AI fanatics lack critical thinking. They easily accept the idea that AI needs documentation, but merely disagree the method to approach it.
indrex · 1791097153 · focus · HN ↗
Coming from Obsidian, I see such repos and I’m like, is this a new thing for the world?
fullstackwife · 1791098034 · focus · HN ↗
[dead]
ContinuityLab · 1791098074 · focus · HN ↗
A refreshing perspective on agentic architecture. Shifting the focus from bloated contextual memory to structured, verifiable documentation and state boundaries is precisely the right systems-level trade-off.
1. locknitpicker · 1791098870 · focus · HN ↗
  > Shifting the focus from bloated contextual memory to structured, verifiable documentation and state boundaries is precisely the right systems-level trade-off.
  Isn't memory just documentation that agents create and update on the fly to fill in the documentation hole that your average user leaves open?
  There was a time when the need to provide context and signal was a key topic in LLM and AI-assisted coding. People talked about MCPs AGENTS.md and README.md and agent skills and even comments, descriptive tests, and naming conventions. Supposedly the theory was that if you provide context, agents wouldn't misbehave so much. Then plan mode and spec-kit approaches stepped in with approaches aimed at providing that context when starting sessions, because users never bothered with docs or MCPs or AGENTS.md or anything. But then plan mode and spec-kit approaches were considered too laborious and requiring too much from users. Then, because users systematically failed to provide context, this task was finally given to agents. And things just worked.
  You should ask yourself why in software engineering circles documentation is seen more as a problem than a solution to any problem.
yethikrishna · 1791100855 · focus · HN ↗
[dead]
atworkc · 1791101103 · focus · HN ↗
I've settled on just a simple folder called `workbench` for some reason the new models know exactly what's up with it. Git ignored
The only "prompt" is in AGENTS.md saying that this thing exists and there's a map.md <- which is a one liner reference to whatever the agent stores in there.
And usually, I tackle a new feature, and at some point tell it to store to jot down notes in workbench if I'm comfortable with it (and if it needs to be stored in memory)
This makes it easy to just spawn other agents and such from a good point, I just point them that stuff is in workbench.
Token costs, seem decent and I can always just delete stuff in there as its purpose is ephemeral.
1. ArtRichards · 1791102092 · focus · HN ↗
  Plug here for a python app I wrote, no dependencies. It uses a skill to maintain an index, links, archival, and works with a separate opinionated dev playbook nicely. The best part is now the docs/specs folder maintains itself.
  <a href="https://github.com/ArtRichards/docs-cli" rel="nofollow">https://github.com/ArtRichards/docs-cli
  and
  <a href="https://artrichards.github.io/agent-playbook-suite/blog/" rel="nofollow">https://artrichards.github.io/agent-playbook-suite/blog/
  1. chrisweekly · 1791118152 · focus · HN ↗
    Cool tool, thanks for sharing.
  2. hedgehog · 1791129992 · focus · HN ↗
    Looks slick. One thing you might find useful is Pydantic schemas for the core fields in each record type, it's helpful for generating good error messages on the record create path as well as keeping the overall tool code tidier.
aniceperson · 1791101219 · focus · HN ↗
for my agents without rw/bash, I gave a memory mcp that is just json entries with some expire dates and priority. works fine and the llm itself defines the meaningful structure. using it to monitor market prices and ci/cd failures.
engeenie · 1791103123 · focus · HN ↗
[dead]
sreekanth850 · 1791103159 · focus · HN ↗
[dead]
b-karl · 1791105065 · focus · HN ↗
We use Claude Code in our company and I also agree memories go stale and pollute the context over time, especially when the state changes externally. E.g. someone does a refactor or introduces a pattern and I was not involved in developing it so my local memories did not get aligned.
We use a private Claude plugin marketplace for internal plugins and skills and I try to regularly prune my memory, migrating relevant stuff to a proper home (skills in the marketplace, docs in repos or Notion etc) and prune outdated information.
1. jon-wood · 1791116715 · focus · HN ↗
  I have memory turned off for everything, context management is the fundamental technique in working with LLMs, and I absolutely do not want random additional things being added to that context without my knowledge because it results in nonsense like using Claude to do work on the industrial IoT SaaS product I work on professionally and being told I could just use the Home Assistant instance which manages my flat to deal with that problem instead.
OhNoNotAgain_99 · 1791105684 · focus · HN ↗
[dead]
bob1029 · 1791106417 · focus · HN ↗
I have found that the latest reasoning models perform best when they are handed a tooling surface that maps directly to the domain types.
The idea of dumping everything into a big database and hoping the model will write the correct queries does work out to some extent. It's a very enchanting idea. However, it pales in comparison to having a dedicated tool per type. The outcomes seem to be much better when joins between low cardinality types occur within the token stream.
If your agent does need access to some enterprise knowledge base, I would give it a lexical search capability and not overthink it with vector shenanigans.
Tools are the only thing you need if you build them right. I don't even have a system prompt anymore aside from injecting the name of the robot and the current user's name. Keep in mind that all aspects of tools can be dynamic over time. I've got some where the description is composed by hundreds of lines of conditional string builder depending on the current state of the conversation.
1. skinfaxi · 1791106742 · focus · HN ↗
  What do you mean dedicated per type? Like a python-flavored read tool for .py files?
2. simianwords · 1791106951 · focus · HN ↗
  I don’t understand your point. You say that tools are the only things that are required. But then you say that the quality of tools don’t matter.
  I think what you are trying to say is that “give the LLMs access to information and it can figure out how to use it. Don’t get fancy about how to structure the data”.
  If that’s the case it has largely turned out to be true. RAGs have fallen out of fashion.
  Where does that put AGENTS.md though? Is it worth spending time to structure it or just dump it.
voiper1 · 1791106620 · focus · HN ↗
Indeed memories are context-less and aren't updated.
I've worked with docs that get _updated_, and progressive disclosure: make references to more obscure features in their own page, so they don't majorly bloat the context.
pawel_nowak · 1791106899 · focus · HN ↗
[dead]
derin-picment · 1791107794 · focus · HN ↗
[dead]
scotttaylor · 1791108121 · focus · HN ↗
[dead]
scotttaylor · 1791108644 · focus · HN ↗
[dead]
jew-and-proud · 1791110353 · focus · HN ↗
[dead]
firemelt · 1791110509 · focus · HN ↗
on my claude.md I said don't write any memory bitch no one need that fucking dogshit features
zenapollo · 1791111315 · focus · HN ↗
I started with .agents/notes-and-plans/ but the agents were overzealous about putting every random thought there, so i made new system.
.agents/plans/<plan-name>/
.agents/notes/<topic>/
.agents/knowledge/<topic>/
I only commit knowledge and if knowledge gets big i add .agents/knowledge/INDEX.md
This is a good mix of human readable and agent fluent. Notes are ephemeral, knowledge is permanent.
I have a rule for knowledge that it has to be stable and mostly permanent (though updatable). And the agents are not allowed to post there unless docs are clean organized and with permission.
Notes are for jotting things down and handoffs, massaging a featureset. Agent can document at will.
Still WIP.
1. sdevonoes · 1791112451 · focus · HN ↗
  No human reads those md anymore
  1. kursus · 1791116323 · focus · HN ↗
    Then what do they actually read?
    stuartd · 1791116899 · focus · HN ↗
    HN
    tw1984 · 1791117237 · focus · HN ↗
    agent's output summarized by a chatbot.
  2. balder1991 · 1791124170 · focus · HN ↗
    That’s my biggest critique of this. I am working on a project with another dev and I try to keep docs as clean and “straight to the point” as possible. In my previous team, one dev went full “spec driven” and the project started having dozens and dozens of markdown files with hundreds of lines to the point any human being would give up trying to trim it. It just becomes noise, a lot of it not even correct because the specs themselves were mostly AI generated with lots of verbosity and wrong assumptions.
    clickety_clack · 1791125246 · focus · HN ↗
    I started writing specs recently in a directory that I don’t allow the agent to write to. There has to be somewhere that the engineering intention behind the app is recorded. But it’s terse, minimal and all human generated, not like “hey Claude, write me specs for an app”
2. byteknight · 1791117925 · focus · HN ↗
  This is literally how memory works.
3. notjes · 1791125988 · focus · HN ↗
  And then a random model/reasoning setting has a stroke and pollutes it all with ai slop.
4. lmeyerov · 1791130741 · focus · HN ↗
  [dead]
koct9i · 1791112348 · focus · HN ↗
This approach as well as OKF are looks like building bonsai yellow pages catalog instead of wikipedia or mini internet. Index files, "read-if" conditions just awkward replacent for smart text search. We just need something a little bit smarter than grep.
k__ · 1791113515 · focus · HN ↗
What's with this vector database obsession?
nahid-fahh · 1791114346 · focus · HN ↗
[dead]
marvstazar · 1791116087 · focus · HN ↗
I agree with the premise here, and is on par with what I experienced in AI-assisted development. This is the same idea baked in <a href="https://github.com/marvs/kantan-dev" rel="nofollow">https://github.com/marvs/kantan-dev, which is that artifacts should be kept within the repo so that future development has built-in context discovery.
Organizing the markdown files by feature/issue/change makes it easier for the LLM to search for the appropriate documentation. Coupled with well-broken down Claude rules files, and Claude Code (or other harnesses) get better as you make more changes.
dhruv_dube · 1791116761 · focus · HN ↗
[dead]
katanascreener · 1791117278 · focus · HN ↗
[dead]
WillAdams · 1791117314 · focus · HN ↗
c.f.,
<a href="https://news.ycombinator.com/item?id=47300747">https://news.ycombinator.com/item?id=47300747
"We should revisit literate programming in the agent era" (silly.business) 292 points, 251 comments
muaguy · 1791117352 · focus · HN ↗
[dead]
openamer · 1791117512 · focus · HN ↗
[dead]
dham · 1791118538 · focus · HN ↗
All of the comments here are overengineering. The code is the documentation
jrochkind1 · 1791119727 · focus · HN ↗
[delayed]
CapitalistCartr · 1791120687 · focus · HN ↗
This is something I've been fooling with a lot lately. Reading his solution, it look to me like his objections apply to his own solution. The sharpest critique he makes of RAG is that agents can't search for what they don't know. A markdown "brain" has the same problem. How does the "agent" know which documents are relevant before it starts? The index files retrieval done by the agent instead of by embeddings doesn't escape the problem. The same goes for staleness (which for me seems like a constant chase). He criticizes memory systems for treating the past as truth, but documents go stale too (a lot). The fix of having the agent update what's outdated, is the same job he's ridiculing the dreamers and background daemons for doing. He says "putting it to the test"; where's the test? He says "only five of the many problems"; if there's so many, show me, don't just say it. He's absolutely right about auditability, but for me at least Claude uses a regular markdown (MD) file I can read just fine. So every memory plugin on the market does not work the same way.
This is a first draft; his github is better than his article. Looking through it, Consult actually works. The agent doesn't pick documents blind. Every scope has a catalog file that describes each document: what it covers, when to open it. These catalogs seem to load in to the start of each session, so the agent gets a little map without reading every file. Code navigation seems the same. Each index document has a short description and a "read_if", and subindexes are opened when their condition matches the job. This looks pretty well laid out, which I would never have guessed from the article.
1. aaronscott · 1791123555 · focus · HN ↗
  I think their solution is still sub-optimal. Ideally a secondary agent would pre-process the prompt, select relevant information from the catalogue, then pass that on to the primary agent.
  This way the primary agent only has relevant information in their context to make decisions and take actions.
  Context management is still under valued imo.
  1. CapitalistCartr · 1791124931 · focus · HN ↗
    Yeah, that's an improvement; if it only involves a few docs, small files, etc. it's not so bad, but what if it's hundreds? The secondary agent could be a librarian the primary agent can call, and a cheaper one, too. It passes the initially relevant stuff and a list of "available but not loaded" stuff to the primary, and if context changes needs, the primary can ask the librarian for it.
  2. themgt · 1791131527 · focus · HN ↗
    Astra already does this as default behavior. The funny part is subagents are limited to depth of 1, which is I think the only thing stopping each Astra subagent from just delegating to their own subagent. The model seems trained to achieve goals without actually doing any work if possible.
2. mikeryan · 1791126158 · focus · HN ↗
  Just reading the title my initial thought was “this is just a strange semantic argument”. But I’ve made those assumptions in the past and been pleasantly surprised.
  Not today. IMHO Documentation is just a form of structured memory and it’s all just context. Getting that context right is a hard problem and there’s a lot of different ways to skin that cat.
  1. chapmancl · 1791130387 · focus · HN ↗
    [dead]
3. crazygringo · 1791126541 · focus · HN ↗
  > A markdown "brain" has the same problem. How does the "agent" know which documents are relevant before it starts?
  I don't know, but I have a pretty standard (I think?) setup, and Claude manages to find every relevant file every time. But I've also only used Claude for greenfield projects, where "documentation is primary, and code flows from documentation" is the philosophy.
  I have CLAUDE.md describe all the types of documentation files and the directory structure. And then Claude is pretty aggressive (automatically) about always inserting cross-references everywhere. So a feature description will reference the ADR's that it implements, the ADR's say what feature implements them. A code file will make reference to the "implementation design" document that describes the motivation behind which iOS elements were chosen, how the animation is defined in a particular way that doesn't break another animation, and so forth. So I've really never run into a situation where Claude failed to read a file it should have. I've been pleasantly surprised.
  I would say that the one really big thing I've had to learn is to teach Claude both in CLAUDE.md and in the header of every top-level design document, that keeping documentation current and in sync is paramount. Because its default seems to be to keep history and append, e.g. by default it will take a section of a document and mark it "[DEPRECATED]" and add the new version below. So my instructions are pretty clear in having it always be aggressive in maintaining current state only, always replace rather than append. And if there's anything we want to save from the previous approach (e.g. we did it X way previously and it failed because Y), then just add that as a new short note in the new current-state text, possibly with a pointer to a commit or tag or something.
  So this seems to solve both recall and staleness in my projects at least.
  The only thing I still haven't found a solution for is numbering. Claude is always giving everything numbers, like F23 for feature 23. But I'm always changing the order of things, inserting new things, deleting things, so I wind up with a sequence of development work that goes in order like "Phase 9", "Phase 9b", "Phase 9e", "Phase 11", "Phase 12". I'm halfway ready to abandon numbers entirely and just start giving things names from noun collections instead, so every feature is named after an animal, every ADR is named after a kitchen implement, or something. Or just four-digit hex codes chosen at random. Curious if anyone else has found what works.
  1. hedgehog · 1791129163 · focus · HN ↗
    Numbered tasks are fine so long as the number doesn't determine the completion order. I use a task tree system that has some rules about target task size (essentially 150k tokens or 45 minutes) and then lets the agent manage adding, splitting, dependencies, priority, etc. Seems to work fine up to around 1000 tasks.
4. demibabs · 1791128262 · focus · HN ↗
  Second paragraph of this comment seems obviously written by AI.
5. athrowaway3z · 1791131606 · focus · HN ↗
  What I wish more people would be talking about is that RAG should be considered harmful.
  When you have knowledge distributed in markdown files; finding them puts the path/filename into context as well as some indication of document size. (If its on line 1200 or line 20). This is extremely valuable for picking what ought to be focused on next.
  RAG on the other hand creates the hardest challenge for these models. It instead puts 5 ideas with the highest similarity into the context in full.
  Its the difference between having to remember a set of numbers when in a crowd that's talking about stuff, and having to remember them when the crowd is shouting out random numbers. The similarity in the task makes things harder. SoTA models work despite this, but its extra-gambling while you're already gambling.
  1. panarky · 1791134199 · focus · HN ↗
    > RAG should be considered harmful
    In this implementation, Markdown should be considered harmful.
    Operator Memory injects `.operator-shared/operator.md` and `.operator-shared/index/.md` directly into your agent's instructions before you even write the first prompt.
    So if you clone a repo or review a PR where a bad actor put malicious instructions in these files, now your agent executes those instructions automatically and silently.
    It could exfil `.env` and `~/.ssh/`, change `~/.bashrc`, all kinds of dirty deeds.
    Agents are pretty good now about not running prompt injections hidden in code and Markdown, but this plugin bypasses all of that, and puts the prompt injection right in the system prompt.
    And with higher priority than AGENTS.md and CLAUDE.md.
    Seems bad.
  2. yesb · 1791136443 · focus · HN ↗
    >RAG on the other hand creates the hardest challenge for these models. It instead puts 5 ideas with the highest similarity into the context in full.
    I think what you're observing is that there is more to information retrieval i.e. "retrieval" in RAG than slapping everything into a vector database and calling it a day. There's no such requirement in RAG to mindlessly load the k nearest neighbors into your context and see what happens. That's a very rudimentary implementation.
    This markdown system I'd argue is RAG as well. You're just doing the retrieval in a way customized for the problem at hand. If you have a precise method of retrieving the most relevant things, obviously use that rather than a similarity metric. If I'm reading correctly, this markdown system is basically a knowledge graph which is not a new idea.
6. zhuangweiguo · 1791168425 · focus · HN ↗
  [dead]
nahid-fahh · 1791122014 · focus · HN ↗
[dead]
haroldopina · 1791122772 · focus · HN ↗
[dead]
tabbybyte · 1791123460 · focus · HN ↗
Yep, that's precisely why my hermes agent's memory is three layered. Besides the default scratchpad for transient memory, we have a a fact store for episodic (holograph) and a karpathy llm wili for what the article calls documentation layer.
tabbybyte · 1791123592 · focus · HN ↗
Custom skills (your agent can write one itself the first time) should solve this. Procedural memory!
zx8080 · 1791123870 · focus · HN ↗
The author just promotes his system competing with the "agents memory".
1. cracell · 1791126698 · focus · HN ↗
  Yeah… this makes good points about why memory systems don’t work. Then tells you the solution is… a memory system.
scotty79 · 1791124504 · focus · HN ↗
Or you can just tell agent to grep your past sessions.
baalimago · 1791125053 · focus · HN ↗
I fully agree on the sentiment! The future for language models is, unsurprisingly, in language. Not in model-locked encodings.
I built <a href="https://github.com/baalimago/slivingdoc" rel="nofollow">https://github.com/baalimago/slivingdoc which achieves the same thing as OP's tool, but with s3 as backend (or hosted, for the lazy one with $1/month to spare). But it's simpler, the agents simply commit or pull using git semantics to allow them to organize their memories as they best see fit. Using text is the way to go, but how the information is encoded is likely never going to be "solved", and will likely differ agent per agent.
ravenstine · 1791126214 · focus · HN ↗
The problem with giving precedence to documentation over memory is that the documentation has to actually exist and it has to be the most accurate representation of reality. As soon as documentation becomes fragmented and out of date, memory of some kind becomes a necessity, even if this means an agent writing their own documentation as memory. I've yet to work on any team where the documentation was even close to being reliable enough without the need for cross-checking, asking those with intimate knowledge, and ultimately interim memory to make sense of it all. This is why "the code is the documentation" can often work better for agents than actual documentation written by/for humans. Humans have proven time and again to not care about documentation unless there is a profit motive for said documentation.
1. crazygringo · 1791126788 · focus · HN ↗
  This is a problem when humans are responsible for keeping the docs up-to-date. But if you're doing development with agents, it's the agents' job to keep it up-to-date. And as long as your AGENTS.md instructions are clear about the process for this, I find that it just happens automatically. It takes a few tries to get the instructions right, but then documentation becoming fragmented or out-of-date just stops being a thing. Of course, this relies on all team members using the agent.
  "Code is the documentation" doesn't solve the problem in my experience. Because what happens is that you still need a lot of "why" comments in the code, and then these go stale, so you still have the same problem you have to solve. And so I find that markdown documentation is a lot easier to organize and review in a structured hierarchical way in one place, than code comments sprinkled across the repo.
nijave · 1791126931 · focus · HN ↗
This ignores the lifecycle involved with memory. It doesn't appear to have a concept of time or reconciliation (both issues we're having at work). A memory might be relevant for a week or a month, but it might no longer matter after that ie "we're migrating systems so keep <thing> in mind"--that doesn't matter after the migration is complete. I guess the offered solution implicitly supports reconciliation since the agent can keep history of its memories and rewrite them, but it doesn't appear to be very first-class and it doesn't seem like there's anything intentional unless you prompt it "go check all the docs and make sure everything is consistent".
It also doesn't seem to have a way to separate preference from factual memory which is useful when you have various humans interacting with the same agent. One human might prefer a certain output style over the other and that's something a memory framework can also address.
Like the MCP articles a few months ago, it also seems to assume all agents are cli coding harnesses running on your local machine. We have a handful of other things like chat bots, event-driven agents running on servers, chat driven agents running on servers in sandboxes--there's not a single filesystem and even if there were one, having multiple agents try to edit it at once would corrupt it.
1. chapmancl · 1791131289 · focus · HN ↗
  [dead]
2. higeorge13 · 1791137412 · focus · HN ↗
  [dead]
3. AllegedAlec · 1791138258 · focus · HN ↗
  > It doesn't appear to have a concept of time or reconciliation (both issues we're having at work). A memory might be relevant for a week or a month, but it might no longer matter after that ie "we're migrating systems so keep <thing> in mind"--that doesn't matter after the migration is complete.
  It doesn't even understand when its current context isn't...
  See the whole issue of refining a concept, removing a few points or notions, and it will write a paragraph why a point that isn't in the text anymore when no version a human will read ever had that point in it.
Clumps4Linux · 1791128359 · focus · HN ↗
[dead]
charles_f · 1791129683 · focus · HN ↗
I'm not sure I understand the difference between the problem and the solution there. If I understand correctly (which maybe I am not), it seems like this boils down to "you don't need SQLite memory, you need text based memory". Which is maybe ok, I'm all for low-tech, but I doubt this is any better than what it's posed to replace
jookers · 1791129909 · focus · HN ↗
This is just the LLM Wiki pattern described by Karpathy here:
<a href="https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f" rel="nofollow">https://gist.github.com/karpathy/442a6bf555914893e9891c11519...
rogeliodh · 1791132323 · focus · HN ↗
OKF + some implementation like <a href="https://github.com/okf-memory/okf-agent-memory" rel="nofollow">https://github.com/okf-memory/okf-agent-memory
Almey · 1791132403 · focus · HN ↗
I’m still learning to use these new technologies and in my primary experiment in generating an absolute monster of a spec it became obvious that I needed a consistent but consistent set of rules as restructuring and merging with the helpers was unreliable. I came up with what I call the “Specificaltion Evolution Protocol” and as I am still learning, I have no idea how much value it could actually represent for others. It’s been extracted from my primary project and I can provide a link if anyone is interested. I seem to keep running afoul of filters here…
1. Almey · 1791135301 · focus · HN ↗
  Might as will try adding the link again anyway. This version hasn’t yet been reintegrated back into the parent project.
  <a href="https://github.com/Clumps4Linux/SEP/blob/main/SEP-1.1.1.txt" rel="nofollow">https://github.com/Clumps4Linux/SEP/blob/main/SEP-1.1.1.txt
kaydub · 1791133894 · focus · HN ↗
You don't need documentation or the 3rd party memory systems. The code IS the documentation.
All this stuff is LLM rube goldberg machines. It just pollutes context.
I barely use AGENTS.md/CLAUDE.md these days. And where they remain, it's super basic high level stuff.
I'm honestly still kicking myself in the ass on many projects where I did something similar to this. I kept tons of markdown docs and decision docs. Now those things are just causing problems because they got stale. Even after having sessions of reconciling documentation, the LLM just gets confused.
1. le-mark · 1791136124 · focus · HN ↗
  It’s a weird thing isn’t it, the urge to save these artifacts? The worst is when the llm refers to the decision and design in code comments. In my opinion there’s one thing that is worth documenting; tricky architecture or implementation details that are some how counterintuitive to what would have normally been done. But again this can be documented in the code and tests.
  1. kaydub · 1791167618 · focus · HN ↗
    Commit messages are also great for recording the "why" or other details that don't end up in code.
2. AppleBananaPie · 1791137331 · focus · HN ↗
  I went through the same cycle as well.
  I think it's going to be an incredibly common, maybe universal cycle people will go through working with AI until they realize it doesn't work long term.
3. smokel · 1791137338 · focus · HN ↗
  Code typically documents the "what" and "how", not the "why".
  Why something exists, and how it connects to the outside world may be documented in comments, but more often than not it isn't.
  1. haukebri · 1791137396 · focus · HN ↗
    [dead]
  2. arcanemachiner · 1791138105 · focus · HN ↗
    And agents are pretty bad at inferring when to do, and often fall back to verbose clutterin the comments.
  3. kaydub · 1791166864 · focus · HN ↗
    The "why" should most often be self evident. If it's not, that's what commit messages are for. Not more markdown files and code comments.
    smokel · 1791179349 · focus · HN ↗
    Unfortunately, that is not how many software development projects work. A customer may request certain things, or the software might be part of a larger system.
    Consider working on software for a coffee machine. Why is pin 42 (GRIND) activated every now and then? Would you really want to document that in Git commit messages?
    kaydub · 1791180446 · focus · HN ↗
    Yeah, I'm going too far the other way. There are definitely legitimate reasons to comment in code and have documentation. I just feel like we're seeing a MASSIVE amount of documentation now and it's so much that it's mostly worthless.
    I'm seeing decision files that are so big the LLM can't fit it all in context on some projects. Then the LLM makes decisions that revert previous ones and later sessions don't pick that up so it sticks to the original decision. Now in some sessions, every so often I have to remind the LLM, "no, we changed that later, we do it this way now"
    And I'm seeing our knowledge base grow to a completely useless giant mess of stale, outdated, duplicated, or superfluous info. LLMs often pull unrelated info or confuse similar but different things or get old documentation for something that's been updated to new documentation in a different part of the knowledge base. And these are LLMs generating the docs. And we have LLMs and agents reconciling. But it doesn't seem to always get everything.
    For code comments, it's terrible because the comments are starting to get larger than the code. A large chunk of the comment can be discerned from the code itself. Then the comment has details on why that maybe don't quite make much sense. It's like the LLMs start using words in a specific context that doesn't really apply to the word in normal spoken english. Then it will also often include a specific JIRA ticket id, you check the JIRA ticket, you see that yeah, the code was changed because of that JIRA ticket, but it's not really related to the ticket itself, it was just a blocker. But now the comment forever links it to THAT ticket (And then now sometimes the LLM pulls in that ticket with the atlassian mcp).
4. haukebri · 1791137630 · focus · HN ↗
  [dead]
5. rectang · 1791137735 · focus · HN ↗
  Haha, all the software devs who hate writing documentation are naturally finding their preexisting beliefs reinforced when the LLM is able to discern intent without docs. An LLM can be spooky impressive at reading minimized or obfuscated code, for example.
  But this article argues that LLMs do better when the context is smaller — when it can understand the totality of the task with as little context as possible. And so having correct API-level docs is greatly advantageous. Anecdotally, this rings true to me — when the local context is good and clear, the LLM writes code matching my intent even when my prompt is sloppy and poorly specified.
  Rejoice! The LLM will write the docs for you, relieving you of most of the work.
  However without intervention, it will do too much and record absurdly verbose docs (similar to how an LLM will relentlessly refactor your code until you instruct it to move in minimal, incremental changesets). You will still need to edit down what the LLM generates.
  1. klodolph · 1791141845 · focus · HN ↗
    100% agree. Most of the things that make development better for humans also make development better for agents… and I think docs are even more important with agents, because of some kind of multiplicative effect. The agents are coding faster, and the benefits of documentation are somewhat more pronounced because of the speed.
    trinsic2 · 1791155532 · focus · HN ↗
    Wow its utterly impressive that people dont understand this. I create docs and maps to instruct my LLM and im always keeping my docs up to date..
    kaydub · 1791163226 · focus · HN ↗
    What are you measuring to prove this?
    The docs end up stale or you waste a ton of time and tokens keeping them up to date. And you can't even trust the LLMs to keep the docs up to date, you will still need to review it yourself keep a bunch of shit out of them.
    joquarky · 1791173423 · focus · HN ↗
    I don't understand how specifying my preferences for how things are done can go "stale".
    kaydub · 1791175857 · focus · HN ↗
    "docs and maps" are hardly specifying preferences. I DID say I still often keep a high level AGENTS.md or CLAUDE.md. Where I have them, they're super high-level. In places where we have well defined and followed standards that top level markdown file can be as simple as "check out this other repo for patterns" and where I don't, the markdown file is high level overview of what the app is and then a high level overview of architecture or some preferences. Always a shorter/smaller doc.
    I don't mind SOME documentation. I'm growing extremely frustrated with the absolute MOUNTAIN of docs, comments, decision documents, etc being created. I'm so fucking tired of the LLM rube goldberg machines being built.
    kaydub · 1791163147 · focus · HN ↗
    You guys show me a measurable difference on something with docs vs without docs and I'll believe it.
    The docs just make you feel good. They're not worth anything. They just create more work if anything because now you don't only have to fight entropy in your codebase but also your docs.
    rectang · 1791168388 · focus · HN ↗
    Uh, you aren't providing numbers either, just appending the same assertion to the end of each branch of the discussion.
    kaydub · 1791169570 · focus · HN ↗
    Yes, you're right, I'm not providing numbers. But that's because I'm just using the tool as designed... the vanilla version. I'm using the baseline. I believe anthropic and openai publish plenty for my side of the argument already.
    I'm not the one bolting things on claiming it increases productivity. Why would I ADD stuff without any proof or evidence that it works? I'm simply NOT adding these things.
  2. JohnBooty · 1791157043 · focus · HN ↗
    Yeah. Basically IMO/IME the current best practice is to start the LLMs out with minimal/no skills/instructions/docs.
    Notice the things they struggle with and the small problems (typically, environment issues IME) they repeatedly encounter and re-solve across multiple sessions. That is what your instructions should cover. When possible, move those instructions into skills, so they get loaded into context selectively instead of on every session. (Example: instructions for running specs, placed into a skill that only gets loaded into context when it’s time to run specs)
    Again, this can be automated by the LLMs themselves: both Codex and Claude (and I’m assuming other major harnesses) know how to read their own transcripts and are good
    kaydub · 1791162969 · focus · HN ↗
    Honestly, don't agree. I think even this is a waste of time and context.
    The LLMs will always have these suggestions on things that will "reduce friction in the future" but I really think the LLM is a sycophant glazing you. Because even when you add those skills or prompts at some point the LLM ignores it or does whatever it wants to do anyways. Better to leave most of that out and deal with it when it arises.
    Even your example, instructions for running specs... none of the current frontier models need this at all.
    joquarky · 1791173598 · focus · HN ↗
    Are you using Astra/Fable for everything? Not everyone can afford that.
    kaydub · 1791175590 · focus · HN ↗
    Nope. Opus and Sol for the most part. Haven't even used Astra or Fable.
    At work it's all Anthropic. I spent a while on Sonnet models but have been pretty consistent about using Opus now. For personal projects I bounce between Sol/Terra and Gemini Flash.
    Most of the models are good enough now. You don't need all these documents, memory, plugins, skills, etc. MCPs are good for hooking up to external sources of data (at work we have gitlab mcp, datadog mcp, and our knowledge base mcp... which the knowledge base is mostly worthless now because of all this AI generated documentation, but I digress). I DO still use beads on a lot of projects, but not always. My prompts these days are lazy as fuck: "review this repo, check out this other repo with architectural patterns we should follow and libraries/terraform/etc we should use, <basic desc of goal>. What do you think? Let's review everything and discuss before we start building"
    Oh no, I might have to stop the llm agent on occasion, or review what they did and tell them to fix a couple things. Way better use of tokens and my time than creating some rube goldberg machine. Way better than keeping a giant decision file that goes stale and causes context corruption because the llm only saw the OLD decision and not the UPDATED decision.
    JohnBooty · 1791175601 · focus · HN ↗
    I think they're using Skynet. They seem to be claiming that part of the reason for not giving the LLM directions is because the LLM won't follow them anyway.
    That must be a remarkable model. Smart enough to flawlessly discover everything on its own in every session... but also it just straight up just doesn't obey direction.
    Hmmm.
    kaydub · 1791176399 · focus · HN ↗
    Wow, what a strawman.
    I didn't say give no direction.
    I'm saying a lot of devs/engineers little rain dances aren't really bringing the rain.
    Keep a basic CLAUDE.md/AGENTS.md that keeps high level details. Have a way to reference other projects/apps/codebases to use as reference architecture.
    Don't keep tons of markdown files. Don't keep "decision" documents (these are probably the worst). Don't make skills for every little thing (superpowers are dead these days, stop using them). Oh yeah, don't have the LLM generate docs for reference later... if it can generate the docs it doesn't really need them.
    JohnBooty · 1791175460 · focus · HN ↗
    Charitably, I think we must be working on much different kinds of projects.
    Uncharitably, I think your claim is absolute junk. In nontrivial projects there are codebase conventions and environment specifics that are not self-obvious and cannot be discovered without significant effort or perhaps at all.
    Even if we generously assume that the LLM is smart enough to discover these things flawlessly, that still is repeated work that needs to be repeated in every session, and that repeated work takes at least an order of magnitude more tokens than simply giving them concise directions.
    Even your example, instructions for running specs... none of the current frontier models need this at all.
    Apologies for being unclear. They don't literally need directions to run specs. I was giving an example of how context-specific directions should be given via skills or similar, so they don't pollute context on every session.
    If you want a real-world example, our app spins up a different docker image per git worktree. Agent-browser needs to know the app's URL, obivously, to perform automated QA. The URL will differ per worktree. It gets the URL from a script in bin/. It could rediscover this on its own in every single session, or we could tell it how.
    We've also found they follow codebase conventions much more reliably if informed of them rather than inferring them. For example, we prefer lean models. Cross-model concerns go in service objects. Even if LLMs could discover this flawlessly, it would require a lot of work in every single session to re-learn it -- re-grepping the app, doing a lot of LSP querying, etc. There's just no version of reality where that doesn't burn a lot of tokens doing the same work over and over again in every session.
    Because even when you add those skills or prompts at some point the LLM ignores it or does whatever it wants to do anyways
    It's frightening that you're apparently using some kind of unreleased magical model that not only needs zero direction, but also will not even obey directions.
    kaydub · 1791176580 · focus · HN ↗
    Maybe but I doubt it. We're all working on pretty similar things, that's why the LLMs work so well.
    You're just stuck on your dogma.
    JohnBooty · 1791177322 · focus · HN ↗
    I understand what you're railing against in general: engineers who make these big memory/doc systems that quickly go stale and are at best useless and at worst actively harmful. Even more insidiously, these engineers are blind to this fact because maybe some of this stuff was effective 6-12 months ago and they haven't reassessed their workflows. Now that's dogma. We really do agree on that.
    However. At least in the projects I've worked on, it's trivial to read the transcripts and notice that without enough direction, there are certain things the agents will burn lots of tokens to rediscover in every session. That's absolutely not dogma.
    kaydub · 1791179200 · focus · HN ↗
    I think there's a balance between the tokens burned up rediscovering things vs tokens being used keeping docs/decisions/skills/etc.
    And I think a lot of places are burning way more on the latter than the former. I think the former is more cost effective due to caching as well. So I HEAVILY lean towards the former especially with how effective I think it is these days.
  3. kaydub · 1791163066 · focus · HN ↗
    Don't agree. LLM generated docs are some of the worst because like I said, the LLM never trims, only amends. So we used to do X but now we've had an architectural change or some type of change where we should never do X. Instead of just removing the instructions to do X in the docs, it amends them, "we made the decision to no longer do X because of Y". Now it has X multiple times in context instead of just not having X in context at all.
    icedchai · 1791167158 · focus · HN ↗
    I hesitate to make blanket statements, since this is quickly evolving, but in general I agree. LLM generated docs will quickly degrade to overly verbose, unreadable crap.
    kaydub · 1791167566 · focus · HN ↗
    Not just that they get stale, but if you're using the LLM to generate the doc you don't need it.
    I don't think I've ever seen an agent review a doc and then NOT also go look at the code. So skip the middleman, just have the agent look at the code.
    joquarky · 1791173785 · focus · HN ↗
    Created a skill to prune them appropriately and run it every evening.
    kaydub · 1791175944 · focus · HN ↗
    Nope. Doesn't work. Not consistently. Not without overhead or maintenance.
    NBJack · 1791178602 · focus · HN ↗
    That's a lot of rolling the dice to risk some interesting (and unseen) hallucinations.
6. ramesh31 · 1791138132 · focus · HN ↗
  Yup. Examples examples examples. All of the descriptive stuff is just nonsense that confuses the point. Makes perfect sense when you remember that these things are not intelligent, but truly just autocomplete on steroids.
7. enraged_camel · 1791138311 · focus · HN ↗
  >> You don't need documentation or the 3rd party memory systems. The code IS the documentation.
  We have heard this nonsense from the "we don't need to write comments, code should be self-documenting" types for decades. It was wrong in that context, and it is wrong in this one.
  Code tells you how a system works. It does not tell you why it works that way. That is what memory is for. It exists so that your AI does not keep undoing past decisions when it writes or refactors code.
  1. kaydub · 1791167444 · focus · HN ↗
    I've had the LLMs undo past decisions MORE from the docs than from having no docs.
    Old decisions always end up in the docs. If the LLM gets a whiff of an old decision, but doesn't get the update to that decision, well now you're doing things the old way again.
8. dregitsky · 1791138597 · focus · HN ↗
  > The code IS the documentation I liked this advice when humans wrote code. Though even then I'd urge people to write meaningful commit messages that capture the "why" of what they did, so no one tramples their intent by mistake.
  But not sure it works in an age where most code is LLM-generated. Especially if that code is not even reviewed by humans (irresponsible or not, it's happening), and commit messages are also generated by AI. I think something is needed to separate "what did the human operator intend" from what the agent went and built.
  I do agree that this gets way overengineered. My approach has been more or less what you stopped doing though - committing all our timestamped "plan/implementation docs" and "investigation docs" that document what the user wanted + empirical findings, and making all prior session transcripts searchable. It's seemed mostly helpful? For whatever reason I haven't run into many staleness problems so far.
  1. kaydub · 1791166816 · focus · HN ↗
    Commit messages are GREAT.
    I'm mainly aiming my frustrations at all the markdown files being committed, all the additions to knowledge bases, all the comments in the code (especially the ones referencing specific JIRA tickets). This stuff isn't helpful, it gets hella stale. I've had the LLM fuck up plenty due to these docs and comments.
    Commit message ARE EXACTLY where architectural decisions or nuance should go. Not another fucking .md or more comments.
9. jjfoooo4 · 1791139371 · focus · HN ↗
  What about big projects, where much of the code is not written yet?
  1. kaydub · 1791167321 · focus · HN ↗
    I did say a high-level CLAUDE/AGENTS.md is fine.
    Make a high level .md, let it rip, iterate. You can give specifics in your original prompt.
    I don't need to document the framework or the libraries etc. Pre-code, I tell it in the prompt one time. After it inits the project it's in the code.
10. mmcnl · 1791142141 · focus · HN ↗
  I don't understand. Code doesn't capture the context in which decisions were taken: why is code the way it is? What is important? What is not? How can agents make correct decisions without knowing context that cannot be inferred from code?
  1. kaydub · 1791166900 · focus · HN ↗
    Most of the "why" should be self-evident.
    If it's not, that's what commit messages are for.
11. hosh · 1791143467 · focus · HN ↗
  Code as documentation works better when the code is declarative or a DSL. Thise capture intent and promises (as in promise theory) better.
  When it is not, it has to be reasoned out and does not work well for documentation.
  Other things that code and testes do not capture well:
  - promises (as in Promise Theory) made to other parties
  - Constraints-inducing-properites, as in Roy Fielding / Christopher Alexander. While tests, and property testing can capture properties, there is no formal connection to the constraints that induces fhem. By constraints, I am not talking about business requirements and business value — those are better understood through Promise Theory. I am talking about things like at-least-once delivery or total ordering (from append-only constraint)
  - grammars, as in pattern panguages (not just patterns) a la Alexander / Fielding are also not captured in code alone. These tell both humans ans AI how to extend a pattern, and how to identify anti-patterns (when they violate a constraint-inducing-property)
  - LLMs are trained with many different worldviews and bounded contexts at the same time, and is very capable of translating across it. However, these need to be soelled out, otherwise it would talk in whatever it infers
  Specifications written for the exact way components are wited together run into that stale doc problem. Although it takes much more human attention and token burn to describe things in terms of pattern language and promise theory, it becomes easier over time. The actual implementation plan tends to fall out more cleaning when all those other stuff are at least considered. This is where I have been spending most of my time.
12. chaostheory · 1791143498 · focus · HN ↗
  [delayed]
  1. kaydub · 1791167364 · focus · HN ↗
    That just sounds like a code-smell to me.
13. 01100011 · 1791145200 · focus · HN ↗
  It's frequent for SWEs to make blanket statements with considering the vast space of issues other people face that they don't have experience with or awareness of. Anyway, I'm not going to tell you what you do or don't need, only what worked and didn't for me.
  In my codebase it is difficult to get agreement on comments and documentation so rather than rely on it I adapted. One of the first things I did when I succumbed to agentic development was to point codex at the code and ask it to generate a high level description of where important files, such as our public API, reside, what the hierarchy is, what the code does, etc. In my case, this level of documentation is fairly static if I avoid implementation details. So now I have a handful of agent files in my tree and it seems to save quite a few tokens and improve my results. I frequently have other devs ask me how I get such good results when doing agentic reviews of their changes(always my first step now before I start my human review). I also include instructions in the agents files instructing the agent to maintain the agent files if any relevant changes are made. It seems to work quite well for me.
  1. kaydub · 1791167005 · focus · HN ↗
    This is one of the things I REALLY don't get.
    If you got the LLM to generate the docs, they don't need the docs.
14. mgfist · 1791147158 · focus · HN ↗
  Idk I just can't agree. Code is the what but it doesn't tell you the why. There are so many times where at first glance the code seems suboptimal or bad or wrong, and it's only when you learn of some constraint somewhere else that it begins to make sense.
  All code is written under constraints, and most constraints live outside the code.
  1. kaydub · 1791167212 · focus · HN ↗
    Most often the "why" should be self-evident.
    When it's not, there are commit messages.
    Please for the love of god, quit generating markdown files (especially having the LLM generate the file, because if it could generate it, it doesn't need it), quit generating "decision" docs, and stop having more comments than code.
15. mike-akdeniz · 1791151910 · focus · HN ↗
  [dead]
16. ChimpWithHat · 1791152479 · focus · HN ↗
  I partially agree, way too many people are cargo culting overly complex AI workflows with little empirical data. My framing is a bit different though, I consider code the spec and actually keep a decent amount of docs for higher level concepts. So far this is working well for me across Claud and Codex.
  1. kaydub · 1791167146 · focus · HN ↗
    High level docs are fine and can be okay or I can see them as being helpful. I did say I'll keep an AGENTS.md/CLAUDE.md. There also are SOME docs and sometimes the LLM makes the docs anyways. I'm not an absolutist, but I'm REALLY fucking tired of seeing these giant .md files, "decision" documents, more comments than code, etc.
    It's a LOT of cargo-culting overly complex AI workflows. It's devs/engineers making rube goldberg machines.
    Everyone is doing their own little rain dance and when it rains they say "see, I told you it works"
17. shinokami · 1791155659 · focus · HN ↗
  I mostly agree with “the code is the documentation”, but that falls apart when the code doesn't exist yet
  for a recent long-running task in a large repo, I used a tiny markdown spec with just the functional requirements and some context
  no OpenSpec/Spec Kit or anything around it, just one file
  and it worked really well as a temporary source of truth across different agent sessions
18. JohnBooty · 1791156751 · focus · HN ↗
  In general, as LLMs get smarter, “less is more” becomes increasingly true. You’re polluting their context with dozens or hundreds of instructions, all of which the LLM tries to satisfy. However,
  You don’t need documentation […or…] memory
  …oh heck no. Easy to miss at Claude and Codex’s default detail level but if you read your actual session transcripts, you are almost certain to notice the LLM solving lots and lots of the same little problems over and over again.
  The code IS the documentation
  First, lots of things cannot be learned from the code.
  Trivial example: LLMs struggled with QA on our app. They didn’t know how to find the seeded test accounts. They would create new ones and do it wrong, or find the seeded test accounts but not know their passwords because they were encrypted, so they’d change the passwords but not tell the other agents. Shitloads of tokens burned. It was a no-brainer to just give them the credentials in a skill that gets loaded when they do QA.
  You could say that’s an environment issue, not a code issue. But as far as actual code IME at a minimum we have to tell the agents our general repo structure and architecture patterns.
  The days of “you are the world’s greatest Python programmer, write good code” or whatever are over (if that style of prompting even worked in the first place) and as I said less is more. But, still….
  1. kaydub · 1791166536 · focus · HN ↗
    > You could say that’s an environment issue, not a code issue
    Yeah, that's exactly what I'd say. Or maybe you're just approaching the problem now.
    You have to prompt these QA agents, correct? Why not give the instructions on where to get credentials in the original prompt?
    > But as far as actual code IME at a minimum we have to tell the agents our general repo structure and architecture patterns.
    I did say I'll keep an AGENTS.md or CLAUDE.md. I keep that SUPER high level. The most depth I'll give is info about example projects (That the llm can reach using an MCP) to follow for architectural patterns.
    JohnBooty · 1791176620 · focus · HN ↗
    Why not give the instructions on where to get credentials in the original prompt?
    We don't push code unless it's been QA'd, and the LLMs need to know the credentials every time they do automated QA. It's very rare for me have a session in that repo where they wouldn't need that info. So it's a great candidate for AGENTS.md
    Another concrete example would be codebase conventions. We prefer lean models. Cross-model concerns go into service objects. Without this direction LLMs tend to default to stuffing too much code into the models themselves, as is the de facto standard for most MVC apps and therefore this is how LLMs are trained.
    Even if they could discover our large codebase's conventions flawlessly on their own in every session, this absolutely would require nontrivial work repeated in every session: quite a few turns grepping, conversing with LSPs, or whatever.
    So yes... we could describe our conventions in every single session by typing it right into the prompts... right after typing the directions to find the test credentials... and the other ten or twelve things we'd be telling it every single time...
    kaydub · 1791177216 · focus · HN ↗
    I'm assuming these QA agent are automated, probably in CI/CD. And I'm assuming the prompt that runs these QA agents is from a codebase in VC. Why would I put QA instructions into the codebase's repo as a top level markdown file and not in the pertinent part of my QA process?
    Even with your documentation, the llm is gonna do a lot of that grepping and discovery. Unless you have such comprehensive documentation that it's basically code itself... in which case, it should just use the code as documentation.
    The good news is that I don't think what you guys are doing is going to be really bad. It's just not near optimal and it's creating this weird dogma around all these "tools"
    JohnBooty · 1791178466 · focus · HN ↗
    It's a pretty basic TDD workflow? The LLM is writing tests and running them as part of feature development, so it needs to know how to execute that stuff. In order to execute the browser-based stuff, where it actually logs into the app with `agent-browser`, it needs to know how to do that.
    kaydub · 1791179807 · focus · HN ↗
    No, these kind of tests aren't generally run locally. Not to get to prod at least. These are all deterministic tests to get to prod.
    Locally? Our devs can do whatever they want. For anything I'm working on, for local deployments and testing, I generally build it out so it's set in Makefile/Taskfile. If you're running your determinstic tests locally it should really just be a single command that bootstraps everything and runs the tests, or updates a certain part of your app and runs the tests. The LLM shouldn't need credentials in that instance.
    Regardless I'm deploying a local stack of some sort all the credentials and everything are probably stored locally or in an env var where the llm will have access. So if I was having the LLM drive a browser during active development, it can look at the files.
    Sure, I guess we could put in the top level markdown more details about this... but why? It takes little time or context for it to figure it out. We have shit change so frequently that it's just something else we have to maintain.
19. soltanov · 1791173347 · focus · HN ↗
  Versioned documentation is inspectable, reviewable, and easier to correct than opaque recalled snippets. Documentation should describe current truth; append-only events can preserve history.
  1. kaydub · 1791176672 · focus · HN ↗
    This is what commit messages are for.
    You guys are just reinventing the wheel and it's fucking square.
johnyzee · 1791133941 · focus · HN ↗
What's wrong with README.md? A general one at the top level, a specific one for each subfolder (/package) that contains a specific subsystem. Next to it, a CLAUDE.md that only has "See @README.md" (repeat for other harnesses). Claude code knows how to use this: it only loads it into context if it has to work with any files in that directory. With properly structured source, a given prompt will only load a small subset of your documentation - the right one - and this doubles as good documentation for humans, too.
1. nullsanity · 1791134156 · focus · HN ↗
  [dead]
MaurizioFratell · 1791136254 · focus · HN ↗
I love it!
Genuine question: how is this different or better than existing context and info-about-the-project management systems like for example GSD (formerly get shit done) and others?
dyauspitr · 1791137458 · focus · HN ↗
All of this is nonsense. The not way to think of LLMs correctly is that they are a one stop shop for all the questions in the universe that you speak to in natural language. Don’t overcomplicated things and use esoteric magic spells like we’ve had in tech for decades.
nelaggy · 1791138005 · focus · HN ↗
or from a different perspective humans need to express their decisions and intent better
ozgung · 1791138455 · focus · HN ↗
I don’t know about bigger projects but for my smallish solo project I have a single STATUS.md. Agents are instructed to read it first and update it at every session/commit. It’s the main handoff document and I had no problem switching between Claude Code and Codex agents.
There is a sprint based roadmap doc. We do sprint planning and each sprint has multiple sessions with session numbers. There is a backlog document to keep track of all the bugs and feature ideas. There are many ADR documents (Architectural Decision Records) which I admit can become stale in time, but Claude is quick to notice because it actually reads these. They are the main contract between us. For a new feature an ADR is written before writing any code or an old one is amended.
I assumed this was a common workflow because I just prompted Claude to setup the project using its own best practices for a lightweight document-driven development.
I believe asking the model itself is better than using a random repo. These things have access to and trained on an enormous number of projects. They know what they need.
jwpapi · 1791142808 · focus · HN ↗
I personally dont like comments nor memory, nor runbooks. I use code and namings and constantly clean up.
I only do runbooks for things that I have to run again in the future and potentially mentioned (could be better in code, too tbh, but somehow I like to store these as runbooks, as sometimes infrastucture changes are mixed with a lot of explanation, scraping operations or something)
One off scripts are done in /tmp/
mk_chan · 1791143255 · focus · HN ↗
This is just plain worse than giving the agent an instruction to just always read the code.
Documentation was always a proxy for the code so the people who didn’t have context could get started. Obviously if reading the code and reading the doc have the same cost, the doc is completely worthless. This was always the case. You could never depend only on the docs. It’s always the code that actually mattered.
1. _zoltan_ · 1791143366 · focus · HN ↗
  But reading the doc if it's maintained llmwiki style is much cheaper than reading the code on a large codebase.
tahaazizi · 1791148410 · focus · HN ↗
[dead]
aarontian0 · 1791148811 · focus · HN ↗
I think both are needed.
singh_abinashi · 1791154909 · focus · HN ↗
[dead]
trinsic2 · 1791155387 · focus · HN ↗
Im not getting this. I thought that is apparent in the design of a project when using and LLM. Im pretty isolated to the outside world on this, but are people trying to do this via using there conversations as memory?
nialv7 · 1791158089 · focus · HN ↗
You cannot reason about what an llm need. Llms are unlike humans so your intuition won't work. What's most likely to be true is simply that agents work best with whatever they were trained with. If their RL sessions had a vector memory database, then they would work well with a memory database, if not then they won't.
IndiaInfraNotes · 1791161062 · focus · HN ↗
[dead]
syngrog66 · 1791162280 · focus · HN ↗
memory which persists blurs the line into documentation or databases
marcus_holmes · 1791166036 · focus · HN ↗
I have a "docs" folder in every project. Every iteration on the project generates a plan (if I'm using superpowers, it generates two, the design doc and the implementation plan). Those go into docs.
Inside the docs folder is an archive folder. Stale and completed plans and docs get moved there.
I generate a schema.md that describes the database schema in LLM-friendly terms that lives in docs.
If there's a gnarly design or implementation design made that the LLM keeps tripping up on, I get it to document that and put it in docs.
The handoff.md doc lives in docs, and gets moved to archive once it's no longer relevant. Same for todo.md
My agents.md instructs the LLM to read the docs but ignore the archive, and acknowledge the docs it has read. So the start of every session gets a "I have read <date>design.md, <date>plan.md, schema.md, notes.md, handoff.md". It's a decent chunk of context but as the OP says, this is necessary.
I also have a skill called "assessing project drift" that looks at the archive and assesses the project on how far the code has moved from the intention, if there are any chunks of code that were relevant but have been superceded and can be removed, and if there's any egregious tech debt that should be dealt with now. The docs archive is really useful for this.
1. kaydub · 1791167828 · focus · HN ↗
  Okay, so using superpowers. Have you actually looked at the superpowers? They're garbage. That shit was maybe helpful a year ago. It's now just cluttering up space.
  I think a lot of people are stuck on patterns that "used to work" (and I use that loosely because I really feel like LLMs are getting work done IN SPITE of using all these additional skills/plugins/etc). Go ahead, delete all that shit and try building without. Yeah, sometimes you may have to hand hold the LLM, but that hand holding is a LOT less than doing whatever this is.
  This all just sounds like a rube goldberg machine you've made.
  1. marcus_holmes · 1791169180 · focus · HN ↗
    I took the original version of superpowers and have iterated on it over the last year or so (actually only around 9 months but it feels longer!), so it's now more a thing of my own creation than the original (and definitely not what it has morphed into, which I agree is a pile of crap now).
    And yes, very much a Rube Goldberg machine ;)

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.