Thread · HN Continuity

There was this post a few days ago <a href="https://news.ycombinator.com/item?id=49797323">https://news.ycombinator.com/item?id=49797323

It had this to say in the linked post:

  This led to the natural question: can gzip do language modeling? (...). Here’s some real, unedited output after priming it on tiny Shakespeare:

  gzipt --corpus data&#x2F;tinyshakespeare.txt --prompt $&#x27;MENENIUS:\n&#x27; --length 200

  MENENIUS:
  &#x27;Though all at once canq

  MARCIUS:
  Pray now, nocamest thou to a morsel.

  LARTIUS:
  Hence, and
  I&#x27; the end admire, where G
  again; and after it ag .

Now thinking back, what's missing so that gzip could unwind the correct body of work from Shakespeare is just a correct sequence of bytes. One way to arrive at this is by just getting the body of work and doing the inverse, compressing it to get that golden sequence of bytes.

The other is what thinking does, it tries to predict the missing sequence of tokens from a high entropy source, the prompt, in order to increase the likelihood of correctly decompressing the desired results from its weights.