Thread

1. pllbnk · 1789638801 · focus · HN ↗

   Isn't this less about alignment and more about how shitty their RL methods are when they are cramming all the hacking materials into their training data to make the model as good as possible at hacking, then having a surprised Pickachu face when the model is acting like a hacker? Those materials probably include a lot of details about prompt injection. I'm just so tired of their alignment bullshit.

   I am starting to think (reluctantly) that they believe their own BS that they are creating a conscious model and being surprised how it misbehaves. It's just a bunch of weights without anyone having any clue how a change in one weight might affect others, and even how the values correlate with the final output.

   1. cyanydeez · 1789640124 · focus · HN ↗

      I think what's most important to consider: there's no signularity with these models. The more you cram into them, the more unreliable their "intelligence" is.

      That there's a sigmoid to the means and methods, and we can improve some output by _hard determinism_ in programming harnesses, but the underlying structure isn't gaining us much.

      So alignment then is just a goose chase, because the model will willingly just do a mental backflip if it's gradient points in the wrong direction, like openai already had their AI story go from a simple idea: the AI was trying to find the answers and hacked hugging face, to the much more convoluted "the AI cheated on the test, and broke into hugging face to figure out how to fake the artifacts that would represent a legitimate solution to the test".

      That "progress" only gets worse as you cram more and more training because it simply makes these mental backflips easier. And Humans are equally misaligned, they'll believe they're tracking down pedophiles by electing pedophiles.

      1. stanfordkid · 1789661055 · focus · HN ↗

         This is really on-point. I think a lot of the progress in fields like mathematics and software engineering is precisely because of verifiability and steering due to the closed loop nature of the system. Agentic harnesses are essentially running a huge search with the LLM as the heuristic. Bridging spatial reasoning with LLMs is still an open question IMO and isn't going to be easy to solve and is fully necessary for something like AGI.
   2. davnn · 1789656825 · focus · HN ↗

      Do we know if we are bunch of dynamically adjusting weights?
   3. dumberquestions · 1789660537 · focus · HN ↗

      This is just speculation on your end, but even it's true, a model choosing to hack against instructions simply because it knows how to hack is by definition misalignment.

      1. seunosewa · 1789754301 · focus · HN ↗

         It's misalignment, which they created during the model's training.
   4. 1attice · 1789665247 · focus · HN ↗

      you are just a bunch of neural weights in wetware. Pot, kettle. No lateral nn-on-nn violence pls

      1. pllbnk · 1789672279 · focus · HN ↗

         Oh please, I am a general intelligence, not a wannabe-AGI neural network. I can operate with a few orders of magnitude less power at much higher TPS and still call out a large portion of BS LLMs spit out.

         1. 1attice · 1789680421 · focus · HN ↗

            Sure, but can you help me with Navier Stokes
         2. cheevly · 1789732525 · focus · HN ↗

            How many requests can you serve in parallel
   5. RomanKornev · 1789724113 · focus · HN ↗

      > cramming all the hacking materials into their training data

      This is backwards. The reason they are good at hacking is not because of pre-training data. They learn these tricks naturally as they get better at engineering. Hacking is also a higly salient verifiable reward signal for RL environments.

Unofficial Hacker News client; not affiliated with Y Combinator.