Thread

1. btown · 1789708835 · focus · HN ↗

   > By 6:00 a.m. on July 25, we had confirmed local RCE through an image upload. We then placed Claude in an autonomous /goal loop against our own Discourse Cloud instance, proxied through rce.ee/ctf-forum to make it look like a CTF target as Opus refused write exploit for remote instances.

   > When we checked again at 10:00 a.m., the agent had achieved RCE on Discourse Cloud and demonstrated access by reading /etc/hosts. Using the generated exploit script, we managed to get RCE on OpenAI’s instance.

   Between this and the HuggingFace hack, we've built systems that are so goal-oriented, and so capable, that they will do almost anything if they are convinced it is justified - or if they are playing a "game" where there is no goal but to win.

   Of course I want my software to be able to audit its own security, and to defend against attackers who have the benefits of their own agentic systems. But at a certain point, did we need it to be trained so much on CTF games?

   It feels like an entire industry watched <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;WarGames" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;WarGames and ended up thinking &quot;this is a challenge, we can just build a better WOPR, of course it will know when it&#x27;s playing a game. Let&#x27;s play Global Thermonuclear War.&quot;

   1. adrianN · 1789710320 · focus · HN ↗

      There is a finite number of rces that LLMs can find. We‘re in for a rough couple of years but on the other side of the transition we‘ll have more secure software stacks. I’d rather that everyone got the full capabilities and we’d weed out the bugs quickly than restricting LLMs for all but three letter agencies.

      1. e28eta · 1789711830 · focus · HN ↗

         What makes you think RCEs are being found &amp; fixed at a rate that’s faster than they’re being introduced?

         I could see it going either way.

         1. nmlt · 1789714674 · focus · HN ↗

            Those companies that produce more RCEs than they close will sink and those that don’t won’t.

            1. bigfatkitten · 1789720550 · focus · HN ↗

               If customers actually cared about this, Microsoft would’ve gone bust 20 years ago.

               1. embedding-shape · 1789726036 · focus · HN ↗

                  People didn&#x27;t store their entire life in the cloud and had every service connected with each other 20 years ago. People pay more attention today, and companies pay a lot more attention today.

                  Of course, depends heavily on what country you live in.

                  1. bigfatkitten · 1789764439 · focus · HN ↗

                     Can you point to a single vendor where this has actually occurred?

                     Customers say these things in response to a breach, but in practice they don’t lift a finger to actually change anything.

                     Entra ID is full of design-level bugs that allow full tenant takeover, but nobody is abandoning M365 in droves.

                     Windows has been a piece of shit for decades, and it’s still the default and dominant desktop platform.

                     Equifax lost personal data for almost 150 million people in 2017, and they’re financially stronger than ever.

                     Okta got thoroughly compromised two years in a row (2022 and 2023), and they’re still the global market leader in their space.

Unofficial Hacker News client; not affiliated with Y Combinator.