Thread

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

208 points · 169 comments · Wirbelwind

continuational · 1786022679 · focus · HN ↗
It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake".
It's been tried so many times before, and it never worked.
1. jerf · 1786024911 · focus · HN ↗
  What would a serious security model for an agent even look like?
  I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that.
  But let's say we'd like them to continue functioning the way they do today. I want my agent to be able to hit the web. I want my agent to be able to read out of its assigned directory sometimes. I want it to be able to hit external resources through MCP servers that have no pragmatic way to know what's going on. And probably most importantly of all, I want my AI to be able to grab from three distinct sources, each of which may be nominally safe on its own, and combine things in a way that may make each of those nominally safe things become unsafe. For example, any ability to read a local file and make a remote request becomes a potential exfiltration mechanism, especially when you remember all the sidechannel ways communication can occur.
  I agree that shifting everything on to the user is essentially non-functional. But whereas I feel like I have a reasonable answer to a lot of other security-related problems, it isn't even clear to me what the definition of a secure agent is.
  There's an effect I need to put a name on someday, where you can get 10 people in a room to agree to a certain series of words, and they will all leave the meeting thinking they agree, but in fact there is no agreement at all because they all have a different definition of the words that were used. In this case, everyone here is going to go "Oh, yes, certainly, AI agents should be secured." But if you sit down with 10 of us to really do the work of defining exactly what that is, you're going to get 10 different answers. There will be overlap, certainly, but when you get down to the nitty-gritty questions like "OK, the user has explicitly asked the agent to do X by accessing Y and the agent has done so and determined that they need to do Z, which the user clicked "allow all" for, and now the agent has decided that it wants to do T, is T fully covered under that "allow all" or not?" you're not going to get anything like universal agreement across the huge range of Xs, Ys, Zs and Ts that could happen and are relevant... and that's still just one question! It's not the totality of what constitutes a "secure agent".
  Defining what a "secure agent" even is is really hard because when it comes to agents, the things that fill in the variables are as arbitrarily complicated as human actions. I haven't fully worked this out but it might be reasonable to say that "agent security" is in reality Turing complete, what with the way they so often throw out fully-fledged programs that you have to approve or reject permissions for.
  1. bryanrasmussen · 1786026308 · focus · HN ↗
    I suppose there would have to be a capability based model in conjunction with a user oversight model and a time model.
    <a href="https://en.wikipedia.org/wiki/Capability-based_security" rel="nofollow">https://en.wikipedia.org/wiki/Capability-based_security
    Thus some agents with higher capabilities can only be run with user oversight at the same time.
    Some agents can not be run during some part of the day - for example these agents can not run within two hours of office closing time, and cannot run on weekends.
    Maybe also the idea of agents writing code - throwing "out fully-fledged programs that you have to approve or reject permissions for."
    Would work better with a capabilities based model where you choose capabilities for the program before hand, meaning the capabilities are not written by agent itself, you read through the code, some of it looks hairy but everything is fine, but oh no dumb human missed the part where agent writes to system32! But luckily enough the program you were expecting actually needed no write capabilities and thus when it tries to go past its assigned capabilities that part of the program fails and the exception is registered.
    Googling it seems like lots of people have thought this (at least where Capability based security is concerned), which seems reasonable to me as it also seems pretty self-evident it must be this way. Have not really seen anything about time based controls but then that is probably because I'm not devoting a lot of effort as I am just doing a bit of procrastination to build up the energy to finish something off.
    jerf · 1786028865 · focus · HN ↗
    A language that revives capabilities, brings them up-to-date, and works in the modern environment is my #1 request from the programming language community right now. I don't need another language with sum types and higher-order functions and a functional focus. I need a language with capabilities. That language may have the other goodies as well, sure, no problem, but we all need capabilities.
    I've done some stabby stabs at a design for it, using an AI as the rubber duck. My initial research indicates that the field of "static language that natively supports capabilities" is surprisingly uncovered and there may be a rich field there. E, the closest match, was tied at the hip to Java, which has some advantages but also comes with disadvantages for languages that are trying to do something as exotic as this. Other existing work was on dynamic languages, and hardly rose to the level of "practical for any use" let alone something that could solve our supply chain issues.
    My issue is primarily that the reward for successfully designing a language and creating a community around it is that you're in charge of a language community... and, uh, my personality is not suited for that, that sounds more like something I'd pay to avoid then something I'd spend months and years of hard work to attain.
    (My advice to anyone doing this is to spend some time with the AI researchers to find the existing work on the topic, not to just sit down and sketch out your initial ideas and run with them. Learn from the past. Expect this to be weeks and probably months of just thinking and noodling before you get to a design. Also, don't try to hook deeply to an existing language, as tempting as it is. This is way too large an impedance mismatch with existing languages. Any external code has to be treated like a nuclear bomb anyhow.)
    zdragnar · 1786030504 · focus · HN ↗
    What is a capability in terms of programming language design? It sounds more like the sort of thing that would belong at the standard library level, where builtin APIs are guarded by flags.
    Deno has something vaguely built in with permissions flags, and old school Blackberry (at least in the J2ME days) had permissions settings for almost everything that an app could do, but again, those are all external to the language design itself.
    jerf · 1786033028 · focus · HN ↗
    In this context, a capability is something that allows the code, or the transitive closure of the code that it may call, to access some particular function, to put it very briefly. So you could have a single function that, if accessed in one manner, is permitted to read from the directory /tmp/blahblah, but accessed in another manner, is permitted to read from the directory /home/zdragnar/.config/myprogram, and it is guaranteed by the language and runtime that the function will never do anything else on the file system. Or, even more importantly, it can be guaranteed that "from this code, nothing, no matter how the code is arranged, can access the file system at all".
    This has massive overlap with a lot of things, like capabilities as implemented by Linux, effects systems, monadic data types as a not-really-very-good capabilities system (Haskellers have been playing with this for years and nobody really loves this approach, many practical problems beyond the scope of this message that would affect any language that tries that approach), dependently-typed programming, and so forth.
    It is not a flag, though; flags can't handle that "transitive environment" aspect. It is also granular on the level of the programming language. This would allow you to do things like have your program be given access to a given part of the mobile file system using the mobile OS' permissions, but you could know beyond a shadow of a doubt that the image library you are using can not at any point access the file system, no matter what changes the author makes to it, because you can just look at the capabilities given to the image library and see that file system access is not among them. This is where real opportunity is over the next few years, in my opinion, because supply chain attacks are going to continue to get worse. A neat aspect of this approach is that it makes huge swathes of the ecosystem unattractive targets by statically ensuring that they can't sneak anything in to something that doesn't need file system or network access, so hackers won't even attack those libraries. Thus the ecosystem can concentrate on monitoring just the high-touch libraries that need to access high-risk resources.
    (I should make it clear that the image parsing libraries can be passed a file; what I am saying is that they can't spontaneously originate arbitrary file system access in a system like this. Really what they would get is probably a "stream" and they would be forbidden from poking into the stream to see what it is made of, at which point, if some other code handed it a file presumably it meant to do that, but it does not give the image library any ability to do anything else with the filesystem.)
    Moreover, if such a benefit was available, that would tend to have people squeeze down those dependencies as much as possible too, e.g., the aforementioned image library. You don't need file system access to parse images, that's just some convenience functions easily worked around that are provided because why not? The number of things that truly need direct high-risk access can actually be surprisingly small, and often, the application can also easily scope the permissions down quite tightly so the HTTP request library is limited in what it can hit, etc.
    We actually have some semi-decent stabs at capabilities at the OS level; we can quibble with them but they are there. But inside an OS process, broadly speaking, anything can do anything in the vast majority of programming languages. The only way to be sure that the string concatenation function doesn't start crawling your file system looking for crypto keys is to examine the code, most languages have no ability to tell it that it can't. There are exceptions, like the aforementioned Haskell, that have at least some ability to do this, but this is an HN post, not a complete guide to a major topic. Really this is more about loading the reader up with keywords they can hit Google or an AI with.
    The term is overloaded, too; Pony has something it calls "capabilities" but it really resembles more a sort of response to Rust's borrow checker, and if there is a way to lift it into this style of capabilities coherently it isn't clear to me. And even if you did, the entire rest of the ecosystem wouldn't support it, which is one of the reasons why this has to be a new language. You can't bodge this on to the side of an existing language.
    (Plus, IMHO, there are some other ideas this may shake loose. Programming languages seem to be in a rut right now. My crack in my previous message about sum types and such isn't really about those things but the way almost every language going by is just a respelling of previous languages, churning over some other iteration of "The Perfect 2015 Language" that is already covered by any number of existing projects. I don't know that there's a lot of room there anymore. We need something big. Once you try something big the design will inevitably lead to other interesting things nobody else is trying either. Capabilities is one distinct possibility... like I said, if you dig in to the history you will discover there are entire huge segments of the capabilities space that haven't even been tried. If nothing else, if you are a PL nerd, I guarantee it'll be fun to explore those spaces that almost nobody has covered. No criticism intended to those who have, who have done a good job. It just hasn't been enough people and enough exploration to truly map the space.)
    bryanrasmussen · 1786039058 · focus · HN ↗
    If you want a fun read that might educate via entertaining I would read Satan Comes to Dinner by Douglas Crockford <a href="https://www.crockford.com/ec/dining.html" rel="nofollow">https://www.crockford.com/ec/dining.html
    I wrote a bunch more, but I have a habit to verbosity when capabilities come in which I should attempt to combat, and have deleted it. Crockford says these things better than I can anyhow.
    continuational · 1786031108 · focus · HN ↗
    Been working on something like that for years: <a href="https://www.firefly-lang.org/" rel="nofollow">https://www.firefly-lang.org/
    Natfan · 1786033385 · focus · HN ↗
    if firefly has no nulls, how do you indicate that a value is unset?
    danudey · 1786037667 · focus · HN ↗
    In Firefly (as in Rust) you can define fields as Optional, so you can do Option[String]; that lets you say "this variable is a String but it might not be here". That lets you then check to see if something is set, rather than checking to see if it's null.
    In Rust an Option is a separate thing that you need to disambiguate to use. For example:
    match result { // The division was valid Some(x) => println!("Result: {x}"), // The division was invalid None => println!("Cannot divide by 0"), }
    Likewise in Rust, you can't have a null pointer, but you can have an Optional pointer, which is either a pointer to something or is not anything.
    Firefly seems to have a similar case structure, though the first example I could find is in the Exceptions section: <a href="https://www.firefly-lang.org/reference/exceptions" rel="nofollow">https://www.firefly-lang.org/reference/exceptions
    grabOption[T](option: Option[T]): T { | Some(v) => v | None => throw(GrabException()) }
    Natfan · 1786038888 · focus · HN ↗
    oh duh! sorry, morning brain didn't think things through -- i do love Option types!
    ux266478 · 1786037415 · focus · HN ↗
    > My initial research indicates that the field of "static language that natively supports capabilities" is surprisingly uncovered and there may be a rich field there.
    You want to look for white papers that talk about object-capability systems. It's a fairly old and well-trod area of research. The E programming language[1] was all about that, and it was pretty late in the game on this stuff.
    You emphasize natively, but the problem is that's not really well defined. For static capabilities, you're just essentially asking for a suffciently strong module system with parameterized abstract data types. It's literally a subset of the grammar and what it's designed to express. Mark Miller (one of the creators of E) demonstrated that[2].
    The knock on effect of that quality is that anything which fulfills that requirement natively supports capabilities. It's part of the grammar. Doesn't even have to be object-oriented. A hackjob demonstration of an SML filesystem library with a brand/mint object capability pattern:
    brand.sig:
    signature BRAND = sig type token end
    mint.sig:
    signature MINT = sig include BRAND val mint : unit -> token end
    makebrand.fun:
    functor MakeBrand () = struct type token = unit ref fun mint () = ref () end
    filesystem.sig:
    signature FILESYSTEM = sig type token val readFile : token -> string -> string val writeFile : token -> string -> string -> unit end
    filesystem.fun:
    functor FileSystem (B : BRAND) :> FILESYSTEM where type token = B.token = struct type token = B.token fun readFile (_ : token) (path : string) : string = "contents of " ^ path fun writeFile (_ : token) (path : string) (_ : string) : unit = () end
    trusted_fs_setup.sml:
    local structure FileAuthority :> MINT = MakeBrand () in structure FS :> FILESYSTEM = FileSystem (FileAuthority) val rootFileToken : FS.token = FileAuthority.mint () end
    trusted_fs.cm:
    Library signature FILESYSTEM structure FS val rootFileToken is brand.sig mint.sig makebrand.fun filesystem.sig filesystem.fun trusted_fs_setup.sml
    Now for any given library using the trusted_filsystem library:
    val doc = FS.readFile rootFileToken "/etc/motd" (* Works fine *)
    Delegation is function application:
    fun helper (t : FS.token) = FS.readFile t "log.txt" val log = helper rootFileToken
    And these all fail:
    val fake : FS.token = ref () (* Trying to forge a token *) val t = FileAuthority.mint () (* Trying to bypass the trusted kernel in trusted_fs_setup.sml by calling the mint *) (* Trying to self-issue authority by making our own brand and mint *) structure MyCap = MakeBrand () val t : FS.token = MyCap.mint () (* type mismatch *)
    What's nice about this is... it's just normal modular programming. It's a very natural grain. It's also completely compile-time, no runtime overhead.
    You can also do a lot of this with phantom types, and it'd be much more terse and easier to handle dynamic capabilities and stuff like a capability algebra, but it ends up way less auditable and is easy to have subtle errors which defeats the point. Also compiler errors will be much more opaque. IMO needing to manually make wrappers for composite capabilities, or to handle dynamic capabilities, is the lesser of two evils. With higher order modules, those problems go away entirely.
    [1] - <a href="https://en.wikipedia.org/wiki/E_(programming_language)" rel="nofollow">https://en.wikipedia.org/wiki/E_(programming_language)
    [2] - <a href="https://homepages.ecs.vuw.ac.nz/~kjx/papers/ARND2018.pdf" rel="nofollow">https://homepages.ecs.vuw.ac.nz/~kjx/papers/ARND2018.pdf

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.