It's a hashmap; both the keys and the values are token positions.
It's a "fuzzy" hashmap; insead of hashmap.get("ball")=="threw" it assigns a probability to every pair of words.
Each hashmap captures some kind of relationship between words.
For example, every LLM has lots of heads whose relationship measures "is token1 the noun on which the verb token2 is acting"? So "I threw the ball" would have a high probability for ("ball", "threw").
But most of the hashmaps don't capture such easy-to-explain relationships. Some of them do. The rest probably capture relationships that we haven't figured out yet. This is the truly mysterious stuff.
But it's just hashmaps. Hashmaps all the way down.
bilsbie · · focus · HN ↗
octoberfranklin · · focus · HN ↗
It's a "fuzzy" hashmap; insead of hashmap.get("ball")=="threw" it assigns a probability to every pair of words.
Each hashmap captures some kind of relationship between words.
For example, every LLM has lots of heads whose relationship measures "is token1 the noun on which the verb token2 is acting"? So "I threw the ball" would have a high probability for ("ball", "threw").
But most of the hashmaps don't capture such easy-to-explain relationships. Some of them do. The rest probably capture relationships that we haven't figured out yet. This is the truly mysterious stuff.
But it's just hashmaps. Hashmaps all the way down.