I think whichever one is used, there needs to be a way to enforce what's written.
If I say "use jq instead of writing a python script to parse json" it should never write adhoc python scripts to parse json. Yet that constantly happens to me anyway.
There is a command in oh-my-pi called "/omfg <problem>". You explain what is wrong with agent's response, and it writes a hook to make sure that the problem doesn't happen again. It then re-runs your previous prompt to make sure that hook is triggered, and if not, it rewrites the hook to make your previous prompt trigger the hook. Then each next agent's response is checked by the hook, and if it is triggered, the agent receives feedback on what's wrong and what must be done differently.
Hooks should (in my opinion) be deterministic. As an example I’ve also noticed Claude writing Python scripts to extract fields from JSON when jq is available to it. They almost always contain the same patterns, and having seen this I’m going to write a hook which triggers on those and fails the turn telling it to use jq instead.
Because custom Python scripts means burning more tokens which costs more money. Reaching for an existing tool like `jq` means burning a LOT fewer tokens.
Imagine you have application code with a function to parse some json. But every time you decide to use it, you re-implement the entire function. This is despite it working the same way every time.
It's not like it's being rewritten for efficiency. "Just because".
And on top it's far more likely to be faulty than a well-designed tool specifically crafted for the job.
Like it may issue an incorrect jq invocation but that's surely cheaper to fix than rewriting the python script. And then it's a simple problem, but let's say you clearly see that "manually editing git files instead of using git" is worse, and most tasks are somewhere in-between.
spike021 · · focus · HN ↗
If I say "use jq instead of writing a python script to parse json" it should never write adhoc python scripts to parse json. Yet that constantly happens to me anyway.
zahrevsky · · focus · HN ↗
gf000 · · focus · HN ↗
jon-wood · · focus · HN ↗
PcChip · · focus · HN ↗
rmunn · · focus · HN ↗
spike021 · · focus · HN ↗
It's not like it's being rewritten for efficiency. "Just because".
gf000 · · focus · HN ↗
Like it may issue an incorrect jq invocation but that's surely cheaper to fix than rewriting the python script. And then it's a simple problem, but let's say you clearly see that "manually editing git files instead of using git" is worse, and most tasks are somewhere in-between.