Thread

Can gzip be a language model?

414 points · 165 comments · networked

Culonavirus · 1790062666 · focus · HN ↗
This tracks perfectly with Winrar being more profitable than OpenAI... coincidence? I think not!
1. wolfi1 · 1790063517 · focus · HN ↗
  winrar is profitable? sure? well, on the other hand, they sure don't make losses
  1. shezi · 1790065233 · focus · HN ↗
    They are a German GmbH and must publicly state their financials: <a href="https://www.northdata.de/win%C2%B7rar%20GmbH,%20Berlin/Amtsgericht%20Charlottenburg%20(Berlin)%20HRB%20109885%20B" rel="nofollow">https://www.northdata.de/win%C2%B7rar%20GmbH,%20Berlin/Amtsg...
    Looks pretty profitable to me.
    amiga386 · 1790065827 · focus · HN ↗
    They're one of the few companies that actually manage to sell "boxed software" (i.e. has not changed much in years but new customers keep buying it)
    That said, Windows users should use 7-Zip. Better compression format, unpacks more kinds of archives
    xxs · 1790067958 · focus · HN ↗
    > Windows users should use 7-Zip
    Please no - no native zstd support. NanaZip is the better option (it's a different build of 7-zip) and it's available at windows store.
    > Better compression format, unpacks more kinds of archives
    winrar has supported zstd for 5 years[0]
    In short - Everyone should be using zstd, and 7-zip does not support it.
    [0]: <a href="https://www.win-rar.com/singlenewsview.html?&L=0&tx_ttnews%5Btt_news%5D=175&cHash=aca38142108c76995eba6c674bfd0e14" rel="nofollow">https://www.win-rar.com/singlenewsview.html?&L=0&tx_ttnews%5...
    aleph_minus_one · 1790069723 · focus · HN ↗
    > Everyone should be using zstd
    Why?
    shawabawa3 · 1790070934 · focus · HN ↗
    It's just the best general purpose compression algorithm, in terms of compression ratio to CPU used, for the vast majority of use cases
    [deleted] · 1790080073 · focus · HN ↗
    [deleted]
    Dylan16807 · 1790123197 · focus · HN ↗
    I'm not that fussed about CPU use.
    <a href="https://github.com/mcmilk/7-Zip-zstd" rel="nofollow">https://github.com/mcmilk/7-Zip-zstd
    By these charts, if I only need 5-10 megabytes per second of compression on a single core, LZMA2 wins significantly on ratio, and still decompresses at well over 100. If I'm doing a backup, or sending/receiving over my internet connection (which only has 2MB/s of upload), LZMA2 easily wins. If I need speed then zstd wins.
    xxs · 1790071830 · focus · HN ↗
    It's just this good.
    On a more realistic note: few years back, I've added zstd compression to our log subsystem (hand written direct buffers, native code, in-process, java). For the same CPU utilization if provides twice dense compression compared to regular [-6] gzip (the topic in the title). Zstd is =much= faster on decompression as well, and it this case - unparalleledly better as it uses twice less disk.
    zstd is 'silicon valley' (the tv show) - life imitates fiction, except entirely open source
    tnelsond4 · 1790072415 · focus · HN ↗
    Yeah, zstd is awesome. I built a webapp that uses it via wasm and the decompression speed is incredible, so much so that I store everything in zstd and decompress it on the app load. My wasm binary also does advanced search and tag insertion and stuff in addition to zstd but it's only 38kb. I wish zstd was supported natively by web browsers. The 8kb implementation of zstd is only half as slow as wasm, so even that is still viable.
    cgio · 1790073990 · focus · HN ↗
    Lzma. I did a simple test, same payload repeated with a gap. Lzma ruled it.
    |gap |gzip |bz2 |lzma | |---------|----------|-----|--------| |0 |2.7% |18.9%|*0.9%*| |8 KB |2.4% |17.8%|0.9% | |*40 KB*|*94.4%* |17.7%|0.4% | |1 MB |*104.3%*|19.7%|*0.9%*|
    notpushkin · 1790080694 · focus · HN ↗
    HN doesn’t support Markdown tables. You can prepend two spaces to show it in monospace though.
    I don’t see zstd in your comparison?
    Sweepi · 1790084807 · focus · HN ↗
    dont know the point of the table, but here it is:
    |gap |gzip |bz2 |lzma | |---------|----------|-----|--------| |0 |2.7% |18.9%| *0.9%*| |8 KB |2.4% |17.8%| 0.9% | |*40 KB* |*94.4%* |17.7%| 0.4% | |1 MB |*104.3%* |19.7%| *0.9%*|
    gcr · 1790130508 · focus · HN ↗
    seconding others' request to try with zstd! I'd be very curious what something like zstd -14 would give you
    cgio · 1790153572 · focus · HN ↗
    Good point. Looks like it’s related to window size (incl for lzma btw). Lzma stays 0.9 and goes to 88.5 post 8.8M gap. Zstd -14 is close to lzma up to 4.4M where it jumps to 91.5. 14L seems to be the best overall for this scenario staying at 1.9.
    ndriscoll · 1790085164 · focus · HN ↗
    zstd is a good default for e.g. filesystem compression, but if you're making an archive file, presumably you're looking for higher compression and LZMA would be a better fit.
    pixl97 · 1790089628 · focus · HN ↗
    >> Windows users should use 7-Zip
    7z is now built into W11 right click so that or zip is what will be used by default anyway.
    hypercube33 · 1790116526 · focus · HN ↗
    Last I checked windows handles compressed files in 4 byte or 4kb or something chunks and is incredibly slow to compress or decompress a lot of files.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.