Thread

1. tomaskafka · 1790242406 · focus · HN ↗

   I love this Nathan Calvin quote that accompanied the second publicized attack:

   > If you find two ants in your kitchen, the best estimate of the total number of ants in your kitchen is not two

   1. rkozik1989 · 1790252201 · focus · HN ↗

      But why is anyone surprised? LLMs have been trained to produce answers the prompter asks even if that means incorrectly using software to get the job done. Its always been doing that we just weren't calling every time it did that a hack before.

      What LLM's hacking isn't is AI acting maliciously in any kind of sentient way. Its just the code behaving how its always behaved but now it has better tools to navigate the web. This has literally been happening this whole time.

      1. jagraff · 1790253243 · focus · HN ↗

         Did you predict that attacks like these would happen ahead of time? I had been using AI agents a lot in the months leading up to the hacks, and yet I was very surprised when they happened; I have become much more afraid of how powerful these agents are as a result. I'd be very impressed if you published a prediction about this ahead of time.

         By the way - LLMs aren't code. They are not designed by humans; they are grown, in a process not dissimilar to evolution except much faster.

         1. altmanaltman · 1790303341 · focus · HN ↗

            > By the way - LLMs aren't code. They are not designed by humans; they are grown, in a process not dissimilar to evolution except much faster.

            The transformer architecture was literally designed by humans; what are you talking about? And LLMs aren't code? Like okay it pretends to not be code but what about an agentic harness running on a machine makes it magical and not code? It's still code execution. Also, comparing training LLMs to evolution is just weird and makes no sense from a biological point of view. You are not evolving anything when training a LLM.

            1. jagraff · 1790338783 · focus · HN ↗

               The architecture was designed by humans; the weights were not. The harness itself is not the agent; it is an interface with the LLM weights that allow those weights to do useful things. The magic of LLMs comes from the weights, the very part of the system that is not written code.

               Gradient descent/backpropogation is similar to evolution, in that both are optimization processes that over time discover better more efficient solutions to problems. The difference is that evolution is blind, and can only make progress via random mutation and natural and sexual selection, whereas backpropagation allows much more rapid discovery because it is directed

Unofficial Hacker News client; not affiliated with Y Combinator.