Thread

1. jll29 · 1790067648 · focus · HN ↗

   Yes: you can classify a test file by topic with gzip as follows:

     gzip -9 sports.txt   testfile.txt
   
     gzip -9 politics.txt testfile.txt
   
     gzip -9 business.txt testfile.txt
   

   (ass. sports.txt politics.txt and business.txt are text docs pertaining from the sports, politics and business domains, respectively, and have equal size)

   The test file belongs to the topic with the smallest size *.gz file.

   Witten's group at Waikato uni were perhaps the first to work on this.

   Also check out the Hutter prize if you are interested in this.

   1. stingraycharles · 1790073336 · focus · HN ↗

      Back in the day - maybe two decades ago - I implemented language detection like this.

      I seeded gzip compressors’ dictionaries with Wikipedia articles in different languages.

      I would then try to use said dictionaries on any random text, and the one that was best able to compress it, was the correct language.

      Absolutely totally not the best approach, but very fast and super simple to implement.

      1. ape4 · 1790085725 · focus · HN ↗

         Or maybe make a list of the most used 1000 words in each language. And see which list has the most occurrences.

         1. wongarsu · 1790087162 · focus · HN ↗

            That requires you to decide what a "word" is, which is not trivial (if you think that ignoring punctuation gets you to a clean "letters surrounded by spaces" you will get lots of issues with various Asian languages)

            Also some languages have a lot of prefixes and suffixes on their verbs or even nouns, which dilutes your list of 1000 words by just adding the same common words over and over again with different suffixes designating grammatical tense, grammatical gender, etc.

            The gzip version sounds more general and more obviously correct

            1. basilgohar · 1790088048 · focus · HN ↗

               Instead of deciding on words, maybe you can break them up into smaller, subword parts – let's call them "quantums". And these quantums can be the units the quantumizer works on to operate on inputs and outputs. We can then use them to build Expansive Dictionary Models, or EDMs. I suppose we'd need a software library to mak working on this easier, think something speedy, fast, hot, like fire: we can call it PHPFlame...

               1. ashkankiani · 1790096993 · focus · HN ↗

                  I'm guessing this is an allusion to reinventing something that already exists, but do you mind explaining what that is to me, since I don't know?

                  1. BobaFloutist · 1790097578 · focus · HN ↗

                     I suspect they're talking about LLM tokens.
                  2. dspillett · 1790097624 · focus · HN ↗

                     It is describing essentially how tokenisation is done for LLMs (Large Language Model → Expansive Dictionary Model).

                     1. giancarlostoro · 1790100793 · focus · HN ↗

                        You mean Quantumnisation ;)
            2. kragen · 1790088619 · focus · HN ↗

               You can just use 2-grams of Unicode code points.

               1. npb · 1790214362 · focus · HN ↗

                  Very much agree. Character bigrams will solve this perfectly with tiny amounts of training data (like a couple KB).
            3. thesz · 1790091939 · focus · HN ↗

               Byte Pair Encoding [1] will be different for different languages. Application of the per-language BPEs to the input text will produce encodings with different lengths.

               [1] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Byte-pair_encoding" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Byte-pair_encoding

               It naturally takes care of common prefixes and suffixes.

               It is easy and fast to apply using radix tree or with finite automata. Even without radix tree, it is possible to have processing speed in the range of hundredths of thousands of bytes per second.

            4. colejohnson66 · 1790104113 · focus · HN ↗

               Even German has issues with &quot;letters surrounded by spaces&quot;. They love to smash words together to get one giant word.

               <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=m8niIHChc1Y" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=m8niIHChc1Y

Unofficial Hacker News client; not affiliated with Y Combinator.