Thread

Can gzip be a language model?

414 points · 165 comments · networked

jll29 · 1790067648 · focus · HN ↗
Yes: you can classify a test file by topic with gzip as follows:
```
  gzip -9 sports.txt   testfile.txt

  gzip -9 politics.txt testfile.txt

  gzip -9 business.txt testfile.txt
```
(ass. sports.txt politics.txt and business.txt are text docs pertaining from the sports, politics and business domains, respectively, and have equal size)
The test file belongs to the topic with the smallest size *.gz file.
Witten's group at Waikato uni were perhaps the first to work on this.
Also check out the Hutter prize if you are interested in this.
1. LPisGood · 1790073275 · focus · HN ↗
  There are some deep connections between machine learning, compression, and cryptography with information theory as a common thread.
  Also, I’ve never seen “ass.” Used to shorten “aside” — I typically use N.B. but perhaps only for important ones.
  1. shoo · 1790074503 · focus · HN ↗
    For anyone wanting an introductory text for information theory & that explores some of these connections & applications, it's worth checking out the late David MacKay's 2003 textbook Information Theory, Inference & Learning Algorithms <a href="https://www.inference.org.uk/itila/" rel="nofollow">https://www.inference.org.uk/itila/
  2. arrowsmith · 1790074634 · focus · HN ↗
    3Blue1Brown has a good video about this: <a href="https://www.youtube.com/watch?v=l6DKRf-fAAM" rel="nofollow">https://www.youtube.com/watch?v=l6DKRf-fAAM
  3. matzf · 1790076082 · focus · HN ↗
    The "ass." more likely stands for "assuming".
    chrisweekly · 1790080295 · focus · HN ↗
    Yeah, I read it as "assuming" too.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.