Nicely done. For me the most fascinating thing about attention heads is the place where Attention matrix is already computed and is getting multiplied by Value vector. It behaves exactly like pushing Value vector through Dense layer of ordinary network where Attention matrix forms weights of that layer. So attention head is trained to construct this small single layer network dynamically during inference from Key and Query. And that's the point. That's rarely underlined in explanations of LLMs architecture and for me it's quite amazing that it works so well. This mechanism easy to observe in this particular visualization if you click through it.
It's definitely part of many intro explanations, but it sits in a weird position where you have to know MLPs well but not Transformers for this to be enlightening, which is a shrinking set of people nowadays. Either someone is a novice to all this, or know both, so beginner Transformer explanations can rarely assume deep familiarity with MLPs.
But yes, a Transformer block can be thought of as basically input-dependently deciding the weight vector of a dense layer. In classical MLP-like networks there was never any multiplication between input-dependent values (there are exceptions, like Squeeze-and-Excitation layers). Multiplication is always between parameters (that don't depend on the input) and activations that depend on the input. With Transformers attention provides a lot of multiplicative interactions between input dependent activations.
Also obligatory: Schmidhuber talked about this a long time ago.
andblac · · focus · HN ↗
bonoboTP · · focus · HN ↗
But yes, a Transformer block can be thought of as basically input-dependently deciding the weight vector of a dense layer. In classical MLP-like networks there was never any multiplication between input-dependent values (there are exceptions, like Squeeze-and-Excitation layers). Multiplication is always between parameters (that don't depend on the input) and activations that depend on the input. With Transformers attention provides a lot of multiplicative interactions between input dependent activations.
Also obligatory: Schmidhuber talked about this a long time ago.
logicchains · · focus · HN ↗
And again recently: <a href="https://arxiv.org/abs/2102.11174" rel="nofollow">https://arxiv.org/abs/2102.11174
groby_b · · focus · HN ↗
That is basically axiomatic ;) (Seriously, Schmidhuber's body of research is mind-blowing)