Thread

ImpactGate: A merge gate that scores the structural decay AI adds

37 points · 48 comments · sagenschneider

MichaelNolan · 1789565768 · focus · HN ↗
Maybe I missed it, but it look like this has just a single metric. Maybe instead of making a new project, you could try to get this metric added to a existing tool like <a href="https://dekobon.github.io/big-code-analysis/index.html" rel="nofollow">https://dekobon.github.io/big-code-analysis/index.html which already has dozens of metrics.
1. sagenschneider · 1789566059 · focus · HN ↗
  Yes, I'm doing my own research on AI augmented pipelines <a href="https://blog.officefloor.net" rel="nofollow">https://blog.officefloor.net . I actually found most code quality tools look for bugs and complexity, but nothing much about cohesive erosion. The nice thing about this metric, is that it determine the files where the erosion is occurring. I turned it into a GitHub action to make it easier to access to get wider feedback on the metric. The GitHub action triggers on your merge request and tells you the files where erosion is occurring to refactor. This stops erosion before it gets too expensive to change (big refactors or rewrite). Yes, happy to work with others to get the metric into other tools.
  1. visarga · 1789570826 · focus · HN ↗
    I just dump all user messages from all sessions in a project into a flat .md file and have agents synthesize the user's intent. Then, using that extracted intent, the agents review code and tests. I call this a retro/reflection pass. It checks whether the code matches the intent and whether the tests match the code.
    Compactly formatted user messages are something an agent can ingest in a few minutes, even if they are thousands of lines long. And the quality of those messages is great: they don't track what the agent does well, only what changes and what breaks.
    Having this top-down view helps a lot. Usually, within a session and deep into a task, the agent loses the global perspective and optimizes for local success. I find it weird there is no harness that treats user messages as high value signal (except my own, of course, I have it, <a href="https://github.com/horiacristescu/playbook-harness" rel="nofollow">https://github.com/horiacristescu/playbook-harness).
    sagenschneider · 1789576460 · focus · HN ↗
    Keeping all the specifications and user discussion does create more context, which is useful for AI.
    However, I'd bring in Brooks discussion on essential and accidental complexity. In other words, there being No Silver Bullet <a href="https://www.cs.unc.edu/techreports/86-020.pdf" rel="nofollow">https://www.cs.unc.edu/techreports/86-020.pdf
    The problem with specification and user discussion is they still have errors that code has. But unlike code, there are no tests to confirm correctness.
    So now we have a definition of the system in a non-exact language with no ability to test to confirm it's correctness. The code holds the essential complexity and now we are adding accidental complexity on top to manage.
    Again agree the specifications and user discussion provides context for the AI. However, a well written test suite provides similar context that can actually confirm correctness of the system.
    However, saying all the above. Focus of ImpactGate ( <a href="https://impactgate.officefloor.net" rel="nofollow">https://impactgate.officefloor.net ) is about erosion of the code, not correctness.
    visarga · 1789582216 · focus · HN ↗
    You usually don't know what you want upfront, in real life it is a stream of specification and steering.
    sagenschneider · 1789603906 · focus · HN ↗
    Yes, agree. It's a learning process. I tend to find when I build systems that at some point you need to stop analysis and just start building things to explore the problem. As you do, you prototype, refactor and possibly throw out ideas in favour of understanding the problem and discovery the real solution.
    The code becomes a reflection of that.
    I'm interested in your experiences of capturing specifications and user discussion on whether this captures the end intentions? Or whether it keeps you focused on earlier dead end directions?
    visarga · 1789620679 · focus · HN ↗
    I think agents are pretty capable of reading a log when given the explicit task of extracting the latest version of what the user wants. In general, they work well for direct tasks like this. They don't forget and do something else the way they do when they are deep into development work or debugging.
    Besides intent, I also mine signs of "user friction," which I use as input for the agent to come up with new tests. What I complain about is one of the signals driving testing.
    sagenschneider · 1789637629 · focus · HN ↗
    Yes, agree very capable of consuming large amounts of information
    I'd be interested to see what happens:
    - to token counts after a year of so of changes, as the specification list grows?
    - how it goes with concurrent changes in teams?
    Plus whether asking AI to add good commenting to the code could achieve the same thing?

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.