‹ BackHN Continuity

Thread

Can gzip be a language model?

414 points · 165 comments · networked

  1. Culonavirus · · focus · HN ↗
    This tracks perfectly with Winrar being more profitable than OpenAI... coincidence? I think not!
    1. wolfi1 · · focus · HN ↗
      winrar is profitable? sure? well, on the other hand, they sure don't make losses
      1. shezi · · focus · HN ↗
        They are a German GmbH and must publicly state their financials: <a href="https:&#x2F;&#x2F;www.northdata.de&#x2F;win%C2%B7rar%20GmbH,%20Berlin&#x2F;Amtsgericht%20Charlottenburg%20(Berlin)%20HRB%20109885%20B" rel="nofollow">https:&#x2F;&#x2F;www.northdata.de&#x2F;win%C2%B7rar%20GmbH,%20Berlin&#x2F;Amtsg...

        Looks pretty profitable to me.

        1. amiga386 · · focus · HN ↗
          They&#x27;re one of the few companies that actually manage to sell &quot;boxed software&quot; (i.e. has not changed much in years but new customers keep buying it)

          That said, Windows users should use 7-Zip. Better compression format, unpacks more kinds of archives

          1. xxs · · focus · HN ↗
            &gt; Windows users should use 7-Zip

            Please no - no native zstd support. NanaZip is the better option (it&#x27;s a different build of 7-zip) and it&#x27;s available at windows store.

            &gt; Better compression format, unpacks more kinds of archives

            winrar has supported zstd for 5 years[0]

            In short - Everyone should be using zstd, and 7-zip does not support it.

            [0]: <a href="https:&#x2F;&#x2F;www.win-rar.com&#x2F;singlenewsview.html?&amp;L=0&amp;tx_ttnews%5Btt_news%5D=175&amp;cHash=aca38142108c76995eba6c674bfd0e14" rel="nofollow">https:&#x2F;&#x2F;www.win-rar.com&#x2F;singlenewsview.html?&amp;L=0&amp;tx_ttnews%5...

            1. tnelsond4 · · focus · HN ↗
              Yeah, zstd is awesome. I built a webapp that uses it via wasm and the decompression speed is incredible, so much so that I store everything in zstd and decompress it on the app load. My wasm binary also does advanced search and tag insertion and stuff in addition to zstd but it&#x27;s only 38kb. I wish zstd was supported natively by web browsers. The 8kb implementation of zstd is only half as slow as wasm, so even that is still viable.
              1. cgio · · focus · HN ↗
                Lzma. I did a simple test, same payload repeated with a gap. Lzma ruled it.

                |gap |gzip |bz2 |lzma | |---------|----------|-----|--------| |0 |2.7% |18.9%|*0.9%*| |8 KB |2.4% |17.8%|0.9% | |*40 KB*|*94.4%* |17.7%|0.4% | |1 MB |*104.3%*|19.7%|*0.9%*|

                1. notpushkin · · focus · HN ↗
                  HN doesn’t support Markdown tables. You can prepend two spaces to show it in monospace though.

                  I don’t see zstd in your comparison?

                  1. Sweepi · · focus · HN ↗
                    dont know the point of the table, but here it is:

                      |gap      |gzip      |bz2  |lzma    | 
                      |---------|----------|-----|--------|
                      |0        |2.7%      |18.9%|  *0.9%*| 
                      |8 KB     |2.4%      |17.8%|   0.9% |
                      |*40 KB*  |*94.4%*   |17.7%|   0.4% |
                      |1 MB     |*104.3%*  |19.7%|  *0.9%*|
                    1. gcr · · focus · HN ↗
                      seconding others&#x27; request to try with zstd! I&#x27;d be very curious what something like zstd -14 would give you
                      1. cgio · · focus · HN ↗
                        Good point. Looks like it’s related to window size (incl for lzma btw). Lzma stays 0.9 and goes to 88.5 post 8.8M gap. Zstd -14 is close to lzma up to 4.4M where it jumps to 91.5. 14L seems to be the best overall for this scenario staying at 1.9.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.