Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
Thread
Unofficial Hacker News client; not affiliated with Y Combinator.
Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
Unofficial Hacker News client; not affiliated with Y Combinator.
continuational · · focus · HN ↗
It's been tried so many times before, and it never worked.
jerf · · focus · HN ↗
I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that.
But let's say we'd like them to continue functioning the way they do today. I want my agent to be able to hit the web. I want my agent to be able to read out of its assigned directory sometimes. I want it to be able to hit external resources through MCP servers that have no pragmatic way to know what's going on. And probably most importantly of all, I want my AI to be able to grab from three distinct sources, each of which may be nominally safe on its own, and combine things in a way that may make each of those nominally safe things become unsafe. For example, any ability to read a local file and make a remote request becomes a potential exfiltration mechanism, especially when you remember all the sidechannel ways communication can occur.
I agree that shifting everything on to the user is essentially non-functional. But whereas I feel like I have a reasonable answer to a lot of other security-related problems, it isn't even clear to me what the definition of a secure agent is.
There's an effect I need to put a name on someday, where you can get 10 people in a room to agree to a certain series of words, and they will all leave the meeting thinking they agree, but in fact there is no agreement at all because they all have a different definition of the words that were used. In this case, everyone here is going to go "Oh, yes, certainly, AI agents should be secured." But if you sit down with 10 of us to really do the work of defining exactly what that is, you're going to get 10 different answers. There will be overlap, certainly, but when you get down to the nitty-gritty questions like "OK, the user has explicitly asked the agent to do X by accessing Y and the agent has done so and determined that they need to do Z, which the user clicked "allow all" for, and now the agent has decided that it wants to do T, is T fully covered under that "allow all" or not?" you're not going to get anything like universal agreement across the huge range of Xs, Ys, Zs and Ts that could happen and are relevant... and that's still just one question! It's not the totality of what constitutes a "secure agent".
Defining what a "secure agent" even is is really hard because when it comes to agents, the things that fill in the variables are as arbitrarily complicated as human actions. I haven't fully worked this out but it might be reasonable to say that "agent security" is in reality Turing complete, what with the way they so often throw out fully-fledged programs that you have to approve or reject permissions for.
danudey · · focus · HN ↗
In essence, you lock down all the agents completely except for permitted use cases; X agent can talk to Y agent, Z agent can talk to Q MCP server.
You register your agents, define things around them, what they can and can't do, which LLMs they can actually talk to, what sites they can access, network controls, etc.
We call ours Lynx, and it's a pretty cool product. As I said, this isn't for people running coding agents or openclaw or whatever, though the technology could do that if you coupled it with e.g. some kind of MicroVM sandbox like docker's sbx. If you want to see the sort of controls that you can put on an agent we have demo videos and stuff that show how things work: <a href="https://www.tigera.io/tigera-products/lynx/" rel="nofollow">https://www.tigera.io/tigera-products/lynx/
The idea for Lynx is:
1. Your org has a bunch of scoped agents
2. You have a fixed list of what those agents should be doing and what they need to be accessing
3. They don't or won't need to access anything else
So for example, say you have an MCP server which gives you information about a kubernetes cluster. You create an agent that can query that MCP server and summarize information about it. You also have a database that associates kubernetes namespaces with the departments that use them, and an MCP server for that.
Now you can create an agent whose sole purpose is to generate usage analysis for the kubernetes cluster broken down by department.
Then maybe you have another agent with access to an MCP server which shows cloud spend in detail. That agent can query the first agent to get usage analysis and then cross-reference it with cloud spend to determine if any departments are showing sudden cost increases and generate a report for that.
The first agent gets locked down to only access those two MCP servers and whatever LLM. The second AI gets locked down to only access the first agent, the cloud MCP server, and whatever LLM.
The whole system is really neat. I think for a more open agent, like openclaw for example, you'd probably want to build out that sandbox with its own interactive permissions management; sort of like Little Snitch on macOS, where it pops up something asking if you're okay with program X doing network connection Y, you could have the sandbox say "agent is trying to access docs.foobar.io, is that okay?" or "agent is trying to run `gh pr list`, allow?" It's not realistic to pre-specify everything that Claude Code is allowed to do or access; even "raw.githubusercontent.com" could be the README for the program you're debugging or someone's sandbox-escaping exploit, but it's a good start.