Thread

1. goldemerald · 1790264329 · focus · HN ↗

   It's nice to see people actively working on this type of research, but OP's baseline is implemented incorrectly. You are not supposed to simply steer away from refusal, but compute the projected vector and subtract only that. The projected/orthogonalization approach is what's done by the original "Refusal is mediated by a single direction" paper.

   1. phatak-dev · 1790264594 · focus · HN ↗

      Afaiu the projection works with the weight update. But here the vector is getting added to residual stream from engram lookup not as direct updat. So the approach is little different

      [Edited] Yes correct.

      In the original paper, they measure how much refusal is actively present in the current token and subtract only that specific amount.

      In my early baseline step, I used a simpler approach where I just subtracted a fixed vector across the board. This is just to see if the approach is even feasible.

      That's actually the main reason I moved to the Engram module, I wanted a smartness that reads the context and turns steering on only when refusal triggers pop up, leaving normal tokens untouched.

      1. [deleted] · 1790265316 · focus · HN ↗

         [deleted]

Unofficial Hacker News client; not affiliated with Y Combinator.