Thread

1. GuB-42 · 1790374877 · focus · HN ↗

   So ugly...

   It looks like a primitive chess engine, trying every move, no matter how stupid, until it works. Relying on its ability to do millions of operations rather than having a plan.

   People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.

   Also, it looked so "loud", querying millions of URL with weird requests. The sandbox as weak as it can get, and there is absolutely zero smart extrusion detection or it would have found it. They used their best AI for attacking, but nothing for protection.

   1. ctolsen · 1790377693 · focus · HN ↗

      My biggest takeaway from this is just how godawful the sandboxing is. The stuff written up in OpenAIs report says more about lack of extremely basic sysadmin skills than anything else.

      I’m not that surprised about models with endless compute being capable of this, I’m more surprised that a company with the resources they have apparently can only create a sandbox that a half skilled human operator could have broken out of easily.

      1. no-name-here · 1790396620 · focus · HN ↗

         > can only create a sandbox that a half skilled human operator could have broken out of easily

         The exploit:

         > The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]

         Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?

         [1] <a href="https:&#x2F;&#x2F;openai.com&#x2F;index&#x2F;hugging-face-model-evaluation-security-incident&#x2F;" rel="nofollow">https:&#x2F;&#x2F;openai.com&#x2F;index&#x2F;hugging-face-model-evaluation-secur...

         1. ctolsen · 1790400860 · focus · HN ↗

            &gt; Are &quot;half skilled human operators&quot; &quot;easily&quot; able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?

            Yes. It’s a fairly simple SSRF attack as far as I can tell. One of the first things I’d try. Especially considering that I would already be armed with the information that I have no internet access except through a thing that downloads things off the internet for me.

            Calling it zero day makes it sound elusive. It’s a bug in closed software that has like 40 CVEs this year alone. Tools like that, especially in internal networks, don’t get much scrutiny and are often riddled with issues.

            1. pishpash · 1790476544 · focus · HN ↗

               In fact, the narrower the constraint the simpler it becomes to focus effort towards the remaining opening. To any problem solver which these agents are it is like being given a hint. Real defense often employs misdirection, message black holes, honeypots and other boobytraps, not a jail with one obvious door.

Unofficial Hacker News client; not affiliated with Y Combinator.