> It’s actually fine if agents make a lot of boneheaded mistakes. What’s not ok is if they keep making the same mistakes.
I worked in life sciences for a bit. There is a process in clinical trials called corrective and preventative actions (CAPA). You'll also find this in other areas where failure tolerance is low (e.g. aircraft).
It's simple: when a mistake happens, you run your CAPA process (Google CAPA form and see examples to extrapolate what that process might look like) and determine the root cause and the correction to the process that allowed the mistake to happen in the first place.
(At least as a SaaS vendor in life sciences, when we had a CAPA (e.g. after a SEV0 failure), it would be folded into our SOPs and then we would be required to retrain on the SOP. Auditors would want to see our evidence of CAPAs, the versions of our SOPs, the records of training. All to extreme for most shops, but I add this for context/color)
This is something most eng shops do not have the discipline for since it requires some diligence.
Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
Why couldn't you express all those as test cases rather than instructions?
In test cases i can do anything, a test framework is just a way of discovering and then scheduling functions to run. I can emit useful instructions to the agent from the failed test case: "After walking the AST of all use of state machine X, a branch was found at Y which reused stale state. Ensure stale references are dropped..."
I can force the agent to pass the test suite before it considers itself done. I can reject edits of such test cases to partially mitigate reward hacking. etc etc
You can express them as tests, but you also need a feedback mechanism that creates the rule that when the LLM generates some net new code or performs some refactor, that there are these CAPAs that it needs to cover with test cases.
The CAPA is a learning that sits outside of the mechanism of verification; it is a record of problem:root_cause:preventative_action. I see it as the instruction that would be required to generate the test case to prevent the next occurrence of a class of failures.
In a real-world process, for example, there is usually a QA lead that is verifying that the process is followed by looking at the paperwork and evidence.
CharlieDigital · · focus · HN ↗
It's simple: when a mistake happens, you run your CAPA process (Google CAPA form and see examples to extrapolate what that process might look like) and determine the root cause and the correction to the process that allowed the mistake to happen in the first place.
(At least as a SaaS vendor in life sciences, when we had a CAPA (e.g. after a SEV0 failure), it would be folded into our SOPs and then we would be required to retrain on the SOP. Auditors would want to see our evidence of CAPAs, the versions of our SOPs, the records of training. All to extreme for most shops, but I add this for context/color)
This is something most eng shops do not have the discipline for since it requires some diligence.
Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
grey-area · · focus · HN ↗
It’s all very well having a list of actions to avoid but that doesn’t help if your agents won’t reliably follow it.
CraigJPerry · · focus · HN ↗
In test cases i can do anything, a test framework is just a way of discovering and then scheduling functions to run. I can emit useful instructions to the agent from the failed test case: "After walking the AST of all use of state machine X, a branch was found at Y which reused stale state. Ensure stale references are dropped..."
I can force the agent to pass the test suite before it considers itself done. I can reject edits of such test cases to partially mitigate reward hacking. etc etc
CharlieDigital · · focus · HN ↗
The CAPA is a learning that sits outside of the mechanism of verification; it is a record of problem:root_cause:preventative_action. I see it as the instruction that would be required to generate the test case to prevent the next occurrence of a class of failures.
In a real-world process, for example, there is usually a QA lead that is verifying that the process is followed by looking at the paperwork and evidence.