Thread

1. GuB-42 · 1790374877 · focus · HN ↗

   So ugly...

   It looks like a primitive chess engine, trying every move, no matter how stupid, until it works. Relying on its ability to do millions of operations rather than having a plan.

   People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.

   Also, it looked so "loud", querying millions of URL with weird requests. The sandbox as weak as it can get, and there is absolutely zero smart extrusion detection or it would have found it. They used their best AI for attacking, but nothing for protection.

   1. doginasuit · 1790376063 · focus · HN ↗

      This is why I have a very low p(doom). LLMs have an incredible working memory, but they have a hard limit on translating that into good decisions. They get by entirely on their persistence. That works fine in the digital world, but once you cross the boundary into physical space the advantage disappears.

      1. pyronite · 1790376390 · focus · HN ↗

         I don’t know how you quantify a very low p(doom), but this is why mine is high enough to worry me.

         A million AI monkeys at a million AI typewriters, banging away at random, could do amazing damage.

         1. Leynos · 1790453276 · focus · HN ↗

            I think of them as being like the Watchmakers in The Mote in Gods Eye who don't design, don't plan beyond the next 15 minutes, don't have any overarching goal other than an innate need, and customize everything to fit the current situation.

            In the nearterm, I am personally more worried about a never ending background noise of colonies of feral agents running 27bn parameter models on compromised or leased hardware. It turns out that being agentic with a time horizon long enough to do damage without intent doesn't actually take that many parameters if RL'd and any open weight model gets an abliterated version fairly quickly.

            Not foom, just patches of digital grey goo effectively becoming normal.

Unofficial Hacker News client; not affiliated with Y Combinator.