Thread

1. lostmsu · 1789843911 · focus · HN ↗

   > Monarch Hadamard MLP: replaces the dense FFN with three learnable Walsh-Hadamard-initialized Kronecker (Monarch) factor pairs interleaved with per-channel diagonal scales, fixed permutations, a SiLU nonlinearity, and a rank-8 input-conditioned gate, so each token gets a fully mixed nonlinear transform of its d_model channels at O(d√d) parameters and compute instead of the O(d²) a dense 4x-expansion MLP would cost.

   Wow, I was just researching W-H in transformers. Did yours seem to work? In my experiments swapping various components for W-H-like transforms caused extreme quality degradation.

   UPD. according to the comments here, this model simply does not work at all, so I guess the answer is NO

Unofficial Hacker News client; not affiliated with Y Combinator.