Thread

Can gzip be a language model?

410 points · 164 comments · networked

Loading the complete thread in the background. This saved snapshot is available now. Refresh

Culonavirus · 1790062666 · focus · HN ↗
This tracks perfectly with Winrar being more profitable than OpenAI... coincidence? I think not!
1. wolfi1 · 1790063517 · focus · HN ↗
  winrar is profitable? sure? well, on the other hand, they sure don't make losses
  1. shezi · 1790065233 · focus · HN ↗
    They are a German GmbH and must publicly state their financials: <a href="https://www.northdata.de/win%C2%B7rar%20GmbH,%20Berlin/Amtsgericht%20Charlottenburg%20(Berlin)%20HRB%20109885%20B" rel="nofollow">https://www.northdata.de/win%C2%B7rar%20GmbH,%20Berlin/Amtsg...
    Looks pretty profitable to me.
    amiga386 · 1790065827 · focus · HN ↗
    They're one of the few companies that actually manage to sell "boxed software" (i.e. has not changed much in years but new customers keep buying it)
    That said, Windows users should use 7-Zip. Better compression format, unpacks more kinds of archives
    xxs · 1790067958 · focus · HN ↗
    > Windows users should use 7-Zip
    Please no - no native zstd support. NanaZip is the better option (it's a different build of 7-zip) and it's available at windows store.
    > Better compression format, unpacks more kinds of archives
    winrar has supported zstd for 5 years[0]
    In short - Everyone should be using zstd, and 7-zip does not support it.
    [0]: <a href="https://www.win-rar.com/singlenewsview.html?&L=0&tx_ttnews%5Btt_news%5D=175&cHash=aca38142108c76995eba6c674bfd0e14" rel="nofollow">https://www.win-rar.com/singlenewsview.html?&L=0&tx_ttnews%5...
    aleph_minus_one · 1790069723 · focus · HN ↗
    > Everyone should be using zstd
    Why?
    shawabawa3 · 1790070934 · focus · HN ↗
    [delayed]
    azatom · 1790080073 · focus · HN ↗
    is it future proof for gpu acceleration?
    Dylan16807 · 1790123197 · focus · HN ↗
    I'm not that fussed about CPU use.
    <a href="https://github.com/mcmilk/7-Zip-zstd" rel="nofollow">https://github.com/mcmilk/7-Zip-zstd
    By these charts, if I only need 5-10 megabytes per second of compression on a single core, LZMA2 wins significantly on ratio, and still decompresses at well over 100. If I'm doing a backup, or sending/receiving over my internet connection (which only has 2MB/s of upload), LZMA2 easily wins. If I need speed then zstd wins.
    xxs · 1790071830 · focus · HN ↗
    It's just this good.
    On a more realistic note: few years back, I've added zstd compression to our log subsystem (hand written direct buffers, native code, in-process, java). For the same CPU utilization if provides twice dense compression compared to regular [-6] gzip (the topic in the title). Zstd is =much= faster on decompression as well, and it this case - unparalleledly better as it uses twice less disk.
    zstd is 'silicon valley' (the tv show) - life imitates fiction, except entirely open source
    tnelsond4 · 1790072415 · focus · HN ↗
    Yeah, zstd is awesome. I built a webapp that uses it via wasm and the decompression speed is incredible, so much so that I store everything in zstd and decompress it on the app load. My wasm binary also does advanced search and tag insertion and stuff in addition to zstd but it's only 38kb. I wish zstd was supported natively by web browsers. The 8kb implementation of zstd is only half as slow as wasm, so even that is still viable.
    cgio · 1790073990 · focus · HN ↗
    Lzma. I did a simple test, same payload repeated with a gap. Lzma ruled it.
    |gap |gzip |bz2 |lzma | |---------|----------|-----|--------| |0 |2.7% |18.9%|*0.9%*| |8 KB |2.4% |17.8%|0.9% | |*40 KB*|*94.4%* |17.7%|0.4% | |1 MB |*104.3%*|19.7%|*0.9%*|
    notpushkin · 1790080694 · focus · HN ↗
    HN doesn’t support Markdown tables. You can prepend two spaces to show it in monospace though.
    I don’t see zstd in your comparison?
    Sweepi · 1790084807 · focus · HN ↗
    dont know the point of the table, but here it is:
    |gap |gzip |bz2 |lzma | |---------|----------|-----|--------| |0 |2.7% |18.9%| *0.9%*| |8 KB |2.4% |17.8%| 0.9% | |*40 KB* |*94.4%* |17.7%| 0.4% | |1 MB |*104.3%* |19.7%| *0.9%*|
    gcr · 1790130508 · focus · HN ↗
    seconding others' request to try with zstd! I'd be very curious what something like zstd -14 would give you
    cgio · 1790153572 · focus · HN ↗
    Good point. Looks like it’s related to window size (incl for lzma btw). Lzma stays 0.9 and goes to 88.5 post 8.8M gap. Zstd -14 is close to lzma up to 4.4M where it jumps to 91.5. 14L seems to be the best overall for this scenario staying at 1.9.
    ndriscoll · 1790085164 · focus · HN ↗
    zstd is a good default for e.g. filesystem compression, but if you're making an archive file, presumably you're looking for higher compression and LZMA would be a better fit.
    pixl97 · 1790089628 · focus · HN ↗
    >> Windows users should use 7-Zip
    7z is now built into W11 right click so that or zip is what will be used by default anyway.
    hypercube33 · 1790116526 · focus · HN ↗
    Last I checked windows handles compressed files in 4 byte or 4kb or something chunks and is incredibly slow to compress or decompress a lot of files.
    TonyTrapp · 1790084221 · focus · HN ↗
    While I also like 7z a lot and use it all the time, it's really not a better format that RAR. Higher compression ratios? Sure, marginally. But for archival RAR is still the better choice as it supports recovery records, which 7z still does not. I also found it more difficult to restore data from a partial 7z file (i.e. trailer data is missing) compared to an incomplete RAR file.
    gsich · 1790097123 · focus · HN ↗
    but worse filebrowser
    jurgenburgen · 1790065847 · focus · HN ↗
    That’s surprising. Seems there is a niche for everything.
    Betelbuddy · 1790069845 · focus · HN ↗
    You need to visit Japan...
    dd8601fn · 1790067609 · focus · HN ↗
    Huh. I guess the warez kids grew up and have money, now?
    cavoirom · 1790082662 · focus · HN ↗
    There is 2 softwares I will buy for myself if I still use Windows: WinRAR and Internet Download Manager. In fact, I will buy them in the future regardless the Windows usage.
    firtoz · 1790068801 · focus · HN ↗
    TIL they're Turkish
    tecleandor · 1790070529 · focus · HN ↗
    The original author and also creator of the RAR format, Eugene Roshal, is Russian. I think he moved to the US way back in the day, but not sure about that. He has been very private with his life.
    The company doing the software distribution, is located in Berlin. The Managing Directors for that company seem to have Turkish names, but I don't know if they're Turkish.
    BTW, Looking for some info I just found a website [0], clearly AI generated (but not necessarily meaning the content is false) claiming Eugene Roshal had severe kidney failure this past month, and he's waiting for surgery. They're asking for donations. There are some names on who's theoretically behind it [1] but they don't link to any LinkedIn profile or personal site. I can't find any other references. The BTC wallet they're using for donations hasn't seen any traffic ever. BE WARY, SMELLS FISHY.
    --
    0: https://eugeneroshal.org/ 1: https://eugeneroshal.org/about/
    grezql · 1790083844 · focus · HN ↗
    [dead]
    pshan · 1790087106 · focus · HN ↗
    WHOIS information for the domain shows it's registered in the Philippines. The name of the owner does not show up in any of the about section people. So I think those are fake.
    vova_hn2 · 1790089337 · focus · HN ↗
    In addition to what the other comments said ("WHOIS information for the domain shows it's registered in the Philippines"[0] etc), the "hospital bed" picture is blatantly AI-generated. If you zoom on his face, you can see that it looks extremely unnatural.
    I suspect that someone's running a script that looks for public (-ish, Eugene has a Wikipedia page at least) figures without active social media presence and creates fake donation websites with AI-generated texts and pictures.
    If my suspicion is correct, this is one of the most evil scams I can imagine.
    [0] <a href="https://news.ycombinator.com/item?id=49801838">https://news.ycombinator.com/item?id=49801838
    zamadatix · 1790098479 · focus · HN ↗
    I read GP more as "sure they're profitable, but they didn't exactly have many expense to cover so why try to compare"
    make3 · 1790098913 · focus · HN ↗
    ggp is obviously hiking
    zamadatix · 1790099015 · focus · HN ↗
    Culonavirus? Yeah - you could make a good guess of that even just by looking at the username they chose :D. Doesn't mean wolfi1 and shezi aren't having a serious conversation from it now.
mg · 1790062807 · focus · HN ↗
```
    give it a normal text prompt, and it
    continues that prompt by searching
    for the byte sequences that compress
    best.
```
One moment, how are we supposed to know how well that search was done? There is no way to search a meaningful part of the search space.
So the result only gives us some lower bound of how well gzip works as a "plausibility tester" of a continuation of a text. The space of possible sequences is many orders of magnitude larger than what was searched. So there might be sequences in there that compress much better.
The text mentions beamsearch, but I don't see a discussion about how well beamsearch performs in finding the global optima when it comes to gzip compressibility of a text?
1. shoo · 1790066095 · focus · HN ↗
  That's a fair question. Suppose we have a way to find a byte sequence x that globally minimises len(gzip(context + prompt + x)) over all sequences x of length n. Here + denotes string concatenation.
  It's unclear if this is very useful.
  The reason it may not be very useful is that one of Deflate's ingredients is a pass that replaces repeated substrings with backreferences to the earlier occurrence in the plaintext input stream.
  E.g. suppose we want to find an n=200 byte sequence x that minimises len(gzip(context+prompt+x)).
  If there exists any 200 byte sequence y such that prompt+y is a substring of context, then Deflate can encode prompt+y as a backreference to that earlier sequence - it needs to store a match-length & a distance-length, encoded using its Huffman trees. This candidate solution y may not be a global minima to our stated objective function, but if not, it's probably going to be a very good near-optimal approximate solution.
  Taking a step back, repeating huge chunks of the input context produces something that's great for minimising compressed output size but doesn't seem particularly helpful as a generative model.
  edit:
  Yep, I tried it out by running an experiment. Searching for the prompt in the context & then copying the following text as the solution produces solutions that are much better, in the sense of minimising the compressed output length, than beam search, while also being unhelpful as a generative tool.
  With the same example as the blog post:
  context: first 30,000 bytes of tinyshakespeare.txt prompt: 'MENENIUS:\n'
  Let x denote a solution, x is a string of length 200.
  Let L(x) denote len(gzip(context+prompt+x)), our objective function
  Let's call the proposed search method of searching for the prompt in the input rfind (after python's str.rfind).
  Then we have
  search method soln soln length feasible? objective value search time (wall clock, s) ------------- ---- ----------- --------- --------------- --------------------------- emptystring "" 0 no 13,023 0.04s gzipt beam search see blog post 200 yes 13,051 11.93s rfind see below 200 yes 13,026 0.04s
  So 'rfind' is finding a solution that does a better job of minimising the objective function -- it only takes 3 bytes more to encode than the infeasible emptystring solution, and costs 25 fewer bytes than the solution found by the beam search implemented by gzipt per the blog post.
  Here's the solution 'generated' by rfind copying and pasting from the input context, starting from the rightmost occurrence of "MENENIUS:"
  MENENIUS: O, true-bred! First Senator: Your company to the Capitol; where, I know, Our greatest friends attend us. TITUS: COMINIUS: Noble Marcius! First Senator: MARCIUS: Nay, let them follow: The Volsces
  Here's the code for 'rfind' - our complete 'generative algorithm':
  def find_candidate_solution_from_context(context, prompt, length): n = len(context) i = context.rfind(prompt, 0, n-length) if i < 0: return b'' i += len(prompt) return context[i:i+length]
  Can hook it into gzipt.py by adding this line after out is defined, but before the beam search begins
  out += find_candidate_solution_from_context(corpus_window, prompt, length)
2. StilesCrisis · 1790078468 · focus · HN ↗
  Read to the end: they aren't actually looking for the best-compressing output, because this quickly devolves into aaaaaaaaaaa. They keep a sliding window over a small portion of recent text and use that.
  Basically I think the entire premise falls apart due to that choice--they forced an interesting-looking outcome by adjusting the algorithm until gzip started picking random slabs of letters instead of ever-larger repeating runs.
  1. im_down_w_otp · 1790083797 · focus · HN ↗
    Meh. That’s nothing compared to the amount of curation and tuning the LLMs are coerced with.
    StilesCrisis · 1790087120 · focus · HN ↗
    No--even a really tiny, underpowered model like GPT-2 with no system prompt produces coherent (though not necessarily desirable or correct) responses.
    Dylan16807 · 1790198585 · focus · HN ↗
    That's still an algorithm that underwent immense amounts of tuning. The tuning here is very simple, very few parameters, and generic. There is no reason to reject it.
    Also I don't know about calling GPT-2 "really tiny". You can get coherent responses out of 5-10M parameters.
    StilesCrisis · 1790434669 · focus · HN ↗
    Heck, a few weeks ago someone posted a 25K-ish generator running on an 8-bit machine (ZX Spectrum? BBC Micro?). It emitted semi-plausible random English phrases. So tiny is subjective but I still think GPT-2 qualifies as a tiny model by any modern standard. I could run it on a laptop.
    Dylan16807 · 1790455445 · focus · HN ↗
    Modern laptops and desktops are so powerful though. That's not a great way to measure tiny.
    With some patience you can run huge models directly out of flash. Edge0–35B-A3B, based on Qwen, has 35 billion parameters and will do 15 tokens per second on pretty boring hardware. If you treat Kimi K3 with 2.8 trillion parameters similarly (importantly, cutting the number of simultaneous experts in half) you can get the active weights under 12GB and get just over 1 token per second on a good laptop. Models in between have speeds in between.
    The terminology does seem to be all over the place, I see some people calling <100M tiny and others calling <3B tiny.
  2. Ohentis · 1790089071 · focus · HN ↗
    I had actually thought of doing this, but didn't for this exact reason. I knew I would have to fudge things to make it anything interesting.
bob1029 · 1790063625 · focus · HN ↗
[delayed]
1. amelius · 1790063801 · focus · HN ↗
  Perhaps a better question is if LLMs are used as compressors, how well is that expected to work.
  1. Retr0id · 1790064324 · focus · HN ↗
    Extremely well, aside from speed.
    bob1029 · 1790064597 · focus · HN ↗
    [delayed]
    K0balt · 1790070503 · focus · HN ↗
    I wonder if, on an information theoretical level, compression radiates heat, and decompression absorbs it?
  2. magicalhippo · 1790064398 · focus · HN ↗
    [delayed]
    amelius · 1790068892 · focus · HN ↗
    Makes me wonder if compression ratio can be used as a measure for intelligence. Any benchmarks using it?
    magicalhippo · 1790070431 · focus · HN ↗
    [delayed]
  3. [deleted] · 1790064529 · focus · HN ↗
    [deleted]
2. Retr0id · 1790064262 · focus · HN ↗
  > Gzip scales linearly. I can gzip a one terabyte file.
  In part because gzip only has a 32KiB window size, and I think it'd be at least quadratic within that window if you were going for optimal compression.
  1. Sesse__ · 1790064720 · focus · HN ↗
    Match-finding does not need to be quadratic. However, truly optimal gzip block splitting is very slow, indeed.
  2. bob1029 · 1790065556 · focus · HN ↗
    I'll concede the window part, but Gzip runs within the physical confines of a single cpu core and is typically entirely resident in local caches. The point is not just the quadratic scaling but also what it scales with.
    Show me an LLM that can run at 300 megabytes per second. Even dedicated ASICs with weights burned in will never move this fast.
    pishpash · 1790069600 · focus · HN ↗
    It won't move as fast as gzip provided the same resources, but to say that LLM's can never run at 300mbps on any hardware is ... a bold bet.
3. fedeb95 · 1790065737 · focus · HN ↗
  I agree, but also equating LLMs with intelligence is wrong.
Tornhoof · 1790064251 · focus · HN ↗
Previous discussions of that specific page <a href="https://news.ycombinator.com/item?id=48557691">https://news.ycombinator.com/item?id=48557691
tromp · 1790064453 · focus · HN ↗
I'm more interested in the converse question: how well does an LLM perform as a compressor? In particular, how well could it do on the Hutter Prize [1] ?
[1] <a href="https://en.wikipedia.org/wiki/Hutter_Prize" rel="nofollow">https://en.wikipedia.org/wiki/Hutter_Prize
1. gkbrk · 1790064714 · focus · HN ↗
  Top contestant in the Hutter Prize uses a neural network for compression. So fair to say, LLMs would perform pretty well compared to gzip.
  1. computably · 1790066098 · focus · HN ↗
    Even ignoring speed per GP, the Hutter Prize's metric includes the size of the decompressor. LLMs would be disqualified for being larger than 1GB.
    Ohentis · 1790109937 · focus · HN ↗
    Hutters prize does have speed restrictions. If it did not, LLMs would win even with counting the size of the model (which is the most reasonable choice imo) as per the main benchmark: <a href="https://www.mattmahoney.net/dc/text.html" rel="nofollow">https://www.mattmahoney.net/dc/text.html.
    computably · 1790984130 · focus · HN ↗
    Huh? The main benchmark is compressed size + program size. The uncompressed size of enwik9 is defined as 1GB, so no LLM qualifies.
  2. londons_explore · 1790067449 · focus · HN ↗
    [delayed]
2. asdfsa32 · 1790064731 · focus · HN ↗
  lossless vs lossy is the question.
  1. anon48293 · 1790066692 · focus · HN ↗
    How lossy? Because I can lossy compress anything into 0 bits.
    sire-vc · 1790070074 · focus · HN ↗
    0 bits is just a very small bloom filter.
3. anax32 · 1790066648 · focus · HN ↗
  hallucinations are lossy compression artefacts
4. Legend2440 · 1790084189 · focus · HN ↗
  Extremely well, over twice the compression ratio of gzip: <a href="https://arxiv.org/html/2409.17141v1" rel="nofollow">https://arxiv.org/html/2409.17141v1
  But it takes 4 hours to compress 10MB.
5. alienbaby · 1790100374 · focus · HN ↗
  can they be considered to have compressed the entirety of their training data into their weights?
  1. nomel · 1790104035 · focus · HN ↗
    If you consider them lossy compression, then yes.
    The goal would be to find the minimum model that, with a fixed seed, would exactly reproduce your text.
    anothereng · 1790109344 · focus · HN ↗
    So I could hide information in a model basically
    nomel · 1790120269 · focus · HN ↗
    I think this would be the exact opposite. This model would be optimized to only output your information.
    anothereng · 1790266434 · focus · HN ↗
    thats what I meant. For a given input give me the output I want without anyone knowing why that is there
6. Ohentis · 1790108978 · focus · HN ↗
  Much better
mentalgear · 1790065436 · focus · HN ↗
Interesting approach, I wonder how this could be used as a classifier. :)
1. Sesse__ · 1790065869 · focus · HN ↗
  I've used LZO as a spam classifier on chat. Spam tends to be very content-less and repetitive...
2. networked · 1790066361 · focus · HN ↗
  [delayed]
0x20cowboy · 1790065466 · focus · HN ↗
Paper from 2022: <a href="https://arxiv.org/abs/2212.09410" rel="nofollow">https://arxiv.org/abs/2212.09410
Crazy coder playing with it: <a href="https://www.youtube.com/watch?v=9n39SbRPXKQ" rel="nofollow">https://www.youtube.com/watch?v=9n39SbRPXKQ
montebicyclelo · 1790065557 · focus · HN ↗
This is fun, but historically people have gone a bit overboard with saying that models like this, or n-gram language models, are anywhere close to large neural network models. There is certainly a connection though.
1. Matumio · 1790068709 · focus · HN ↗
  [delayed]
  1. montebicyclelo · 1790074025 · focus · HN ↗
    Yep agree, good details, and this doesn't contradict my point above, about people going "overboard" with the comparison.
    I do think when making these comparisons, it is worth emphasising that neural nets are really different. E.g. I used to see people equating LLMs to n-gram models, etc. which is overly simplistic, (especially in the early days when the models weren't as good).
GodelNumbering · 1790065967 · focus · HN ↗
3blue1brown did a series on this topic: <a href="https://www.youtube.com/watch?v=l6DKRf-fAAM" rel="nofollow">https://www.youtube.com/watch?v=l6DKRf-fAAM <a href="https://www.youtube.com/watch?v=GlYgs6v2YfU" rel="nofollow">https://www.youtube.com/watch?v=GlYgs6v2YfU (i think one more is yet to release)
1. kevinrineer · 1790102123 · focus · HN ↗
  Wonderful link. Just commenting to add: in the first video, Grant mentioned it would be a trilogy series.
  1. jrflo · 1790132571 · focus · HN ↗
    I keep thinking that surely I missed the 3rd video in the series but no, 2 months later we are still waiting for the conclusion. I'm sure it'll be worth the wait though.
    bigmadshoe · 1790133617 · focus · HN ↗
    He made a 'trilogy' about probability distributions that is still missing its third video five years later. Let's hope it doesn't turn out the same!
networked · 1790066259 · focus · HN ↗
[delayed]
berkes · 1790067526 · focus · HN ↗
I've been pondering on something related: can an LLM be a chat?
Some models are reproducible, in that the same prompt will generate the same output. Say that we could wire up such a model to generate some code.
In that case, we could create a prompt that generates, say, an entire codebase, or a large piece of text. The prompt (or really, the tokens) would then be the compressed version of the codebase or the text.
I am not talking about an "AI agent", but really a model that we call in a reproducible manner. Preferably one call, with one prompt. An agent could just run `git clone` to "decompress" a codebase, which conflates the idea of compression. If that were compression, then the "compressed version of the git kernel" would be a single line of text: `git clone <a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git" rel="nofollow">https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...`. I am really talking about having an LLM re-generate text based on a prompt.
Does that make sense? I can imagine that this is highly impractical and inefficient. But would this count as "compression" at all?
1. evgpbfhnr · 1790067744 · focus · HN ↗
  You're describing <a href="https://bellard.org/ts_zip/" rel="nofollow">https://bellard.org/ts_zip/ ("Text Compression using Large Language Models") ?
  1. stackbutterflow · 1790069516 · focus · HN ↗
    Man,did that page load fast. It made me realize how slow the rest the (my) web is.
2. LoganDark · 1790067755 · focus · HN ↗
  <a href="https://bellard.org/ts_zip" rel="nofollow">https://bellard.org/ts_zip
3. eru · 1790069523 · focus · HN ↗
  You sound a bit confused.
  A large language model itself (the network) give you the probabilities for the next token given some prefix of tokens so far. You can use arithmetic coding to go from these probabilities to a deterministic compression / decompression algorithm.
  When you use an LLM to generate text, you sample from that probability distribution. You can use a true random sample. Or you can make it trivially deterministic by using a seeded pseudo-random-number-generator or you just pick the highest probability each time. But that's all a red herring; really, what you want is arithmetic coding.
  <a href="https://en.wikipedia.org/wiki/Arithmetic_coding" rel="nofollow">https://en.wikipedia.org/wiki/Arithmetic_coding
4. flyinglizard · 1790071729 · focus · HN ↗
  It makes a lot of sense. I thought about it in the context of pull requests or change sets: if the text-to-code process is reliable, why don't you give me prompts instead of code? Code becomes just an intermediate representation.
5. dist-epoch · 1790071970 · focus · HN ↗
  This was tried many times in the past for images, even before LLMs.
  <a href="https://imalogic.com/blog/2024/06/03/image-compression-decompression-solution-based-on-text-prompt-generation-and-regeneration/" rel="nofollow">https://imalogic.com/blog/2024/06/03/image-compression-decom...
6. vasco · 1790072198 · focus · HN ↗
  Sounds like an interesting way for future OS included apps to be distributed.
  Like when you click the Calculator button on your android, it wouldn't actually exist yet, your click actually prompts it into existence. But naively that has problems because you don't want a different UI every time. There's something to your idea.
  1. StilesCrisis · 1790078582 · focus · HN ↗
    Someone did it!
    <a href="https://youtu.be/7NfyZhV1dKM?is=YOUXCHuFUiPdlD0p" rel="nofollow">https://youtu.be/7NfyZhV1dKM?is=YOUXCHuFUiPdlD0p
7. meindnoch · 1790077519 · focus · HN ↗
  >I've been pondering on something related: can an LLM be a chat?
  A chat?
  >I am not talking about an "AI agent", but really a model that we call in a reproducible manner.
  An LLM is just as deterministic as any other computer program. For identical inputs (which includes the PRNG seed) it produces identical outputs.
  >compressed version of the git kernel
  The git kernel, got it.
  >But would this count as "compression" at all?
  Yes. The decompressor is several tens of gigabytes though.
  1. foldr · 1790077937 · focus · HN ↗
    >An LLM is just as deterministic as any other computer program. For identical inputs (which includes the PRNG seed) it produces identical outputs.
    This is not really true in practice because of multi-threading and out-of-order execution. Mathematically equivalent orderings of operations are not equivalent when dealing with floating point values, so most practical LLM implementations end up being non-deterministic.
    MarkusQ · 1790088510 · focus · HN ↗
    "An LLM is just as deterministic as any other computer program" is not refuted by pointing out hardware limitations that would affect any other computer program implemented at similar scale (weather forecasts, or even just computing the average of a large stream of sensor readings).
    foldr · 1790089005 · focus · HN ↗
    I'm not really trying to 'refute' the original statement. Certainly, an LLM is just doing some calculations that can be done deterministically in principle. However, I think it's worth pointing out that there are practical barriers to doing those particular calculations both deterministically and efficiently. People who worry about LLM output not being reproducible aren't necessarily misunderstanding what an LLM is doing; they are responding to a real feature of most practical LLM implementations.
    eru · 1790146507 · focus · HN ↗
    Agreed.
    If you wanted to and had enough engineering effort to spare, you could run an LLM deterministically at relatively small impacts to performance.
    One approach is to make sure you run things in the same order. Another is to change your operations so that more of them become associative or even commutative.
    See eg the paper 'A Lattice-Based Approach to Deterministic Parallelism' for some interesting ideas on the latter.
nelox · 1790067610 · focus · HN ↗
Only if it can run DOOM.
jll29 · 1790067648 · focus · HN ↗
Yes: you can classify a test file by topic with gzip as follows:
```
  gzip -9 sports.txt   testfile.txt

  gzip -9 politics.txt testfile.txt

  gzip -9 business.txt testfile.txt
```
(ass. sports.txt politics.txt and business.txt are text docs pertaining from the sports, politics and business domains, respectively, and have equal size)
The test file belongs to the topic with the smallest size *.gz file.
Witten's group at Waikato uni were perhaps the first to work on this.
Also check out the Hutter prize if you are interested in this.
1. LPisGood · 1790073275 · focus · HN ↗
  There are some deep connections between machine learning, compression, and cryptography with information theory as a common thread.
  Also, I’ve never seen “ass.” Used to shorten “aside” — I typically use N.B. but perhaps only for important ones.
  1. shoo · 1790074503 · focus · HN ↗
    For anyone wanting an introductory text for information theory & that explores some of these connections & applications, it's worth checking out the late David MacKay's 2003 textbook Information Theory, Inference & Learning Algorithms <a href="https://www.inference.org.uk/itila/" rel="nofollow">https://www.inference.org.uk/itila/
  2. arrowsmith · 1790074634 · focus · HN ↗
    3Blue1Brown has a good video about this: <a href="https://www.youtube.com/watch?v=l6DKRf-fAAM" rel="nofollow">https://www.youtube.com/watch?v=l6DKRf-fAAM
  3. matzf · 1790076082 · focus · HN ↗
    The "ass." more likely stands for "assuming".
    chrisweekly · 1790080295 · focus · HN ↗
    Yeah, I read it as "assuming" too.
2. stingraycharles · 1790073336 · focus · HN ↗
  Back in the day - maybe two decades ago - I implemented language detection like this.
  I seeded gzip compressors’ dictionaries with Wikipedia articles in different languages.
  I would then try to use said dictionaries on any random text, and the one that was best able to compress it, was the correct language.
  Absolutely totally not the best approach, but very fast and super simple to implement.
  1. actionfromafar · 1790084214 · focus · HN ↗
    Sounds like the best approach. :)
  2. ape4 · 1790085725 · focus · HN ↗
    Or maybe make a list of the most used 1000 words in each language. And see which list has the most occurrences.
    wongarsu · 1790087162 · focus · HN ↗
    That requires you to decide what a "word" is, which is not trivial (if you think that ignoring punctuation gets you to a clean "letters surrounded by spaces" you will get lots of issues with various Asian languages)
    Also some languages have a lot of prefixes and suffixes on their verbs or even nouns, which dilutes your list of 1000 words by just adding the same common words over and over again with different suffixes designating grammatical tense, grammatical gender, etc.
    The gzip version sounds more general and more obviously correct
    basilgohar · 1790088048 · focus · HN ↗
    Instead of deciding on words, maybe you can break them up into smaller, subword parts – let's call them "quantums". And these quantums can be the units the quantumizer works on to operate on inputs and outputs. We can then use them to build Expansive Dictionary Models, or EDMs. I suppose we'd need a software library to mak working on this easier, think something speedy, fast, hot, like fire: we can call it PHPFlame...
    ashkankiani · 1790096993 · focus · HN ↗
    I'm guessing this is an allusion to reinventing something that already exists, but do you mind explaining what that is to me, since I don't know?
    BobaFloutist · 1790097578 · focus · HN ↗
    I suspect they're talking about LLM tokens.
    dspillett · 1790097624 · focus · HN ↗
    It is describing essentially how tokenisation is done for LLMs (Large Language Model → Expansive Dictionary Model).
    giancarlostoro · 1790100793 · focus · HN ↗
    You mean Quantumnisation ;)
    kragen · 1790088619 · focus · HN ↗
    You can just use 2-grams of Unicode code points.
    npb · 1790214362 · focus · HN ↗
    Very much agree. Character bigrams will solve this perfectly with tiny amounts of training data (like a couple KB).
    thesz · 1790091939 · focus · HN ↗
    Byte Pair Encoding [1] will be different for different languages. Application of the per-language BPEs to the input text will produce encodings with different lengths.
    [1] <a href="https://en.wikipedia.org/wiki/Byte-pair_encoding" rel="nofollow">https://en.wikipedia.org/wiki/Byte-pair_encoding
    It naturally takes care of common prefixes and suffixes.
    It is easy and fast to apply using radix tree or with finite automata. Even without radix tree, it is possible to have processing speed in the range of hundredths of thousands of bytes per second.
    colejohnson66 · 1790104113 · focus · HN ↗
    Even German has issues with "letters surrounded by spaces". They love to smash words together to get one giant word.
    <a href="https://www.youtube.com/watch?v=m8niIHChc1Y" rel="nofollow">https://www.youtube.com/watch?v=m8niIHChc1Y
    wodenokoto · 1790106768 · focus · HN ↗
    Just collect common bi- and trigrams and a naive bayesian classifier.
    ignoramous · 1790114779 · focus · HN ↗
    [delayed]
3. woadwarrior01 · 1790073684 · focus · HN ↗
  aka Normalized compression distance (NCD). Its close cousin: Normalized Google distance (NGD) is also super interesting!
  <a href="https://en.wikipedia.org/wiki/Normalized_compression_distance" rel="nofollow">https://en.wikipedia.org/wiki/Normalized_compression_distanc...
4. m-hodges · 1790078812 · focus · HN ↗
  [dead]
5. myrmidon · 1790084526 · focus · HN ↗
  Nitpick: Doing it exactly like this is flawed because you let the compressibility of your references taint the result; what you would prefer is the compressed size of testfile given sports.txt/... as a dictionary without accounting for the compressed size of that, no?
  Really interesting approach though.
  1. akoboldfrying · 1790136237 · focus · HN ↗
    True in principle, but usually compressors only make back-references (since allowing forward references makes decompression challenging), meaning that it's difficult for later test data to influence the compression of earlier reference data. (It's not completely impossible: E.g., if the last bytes in the reference are "foofoo" and this would normally get compressed to "2 repetitions of 'foo' found at earlier position 456", then if the first bytes of the test data were "foo", all of this would get compressed to "3 repetitions of 'foo' found at earlier position 456".)
    In practice it would be enough to insert a short, very unlikely constant string in between the reference data and the test data to prevent such "crossovers".
  2. akoboldfrying · 1790136536 · focus · HN ↗
    You're right, you should subtract off the compressed sizes of the respective reference files before comparing. (This suffices if we assume that later input data does not influence the compression of earlier input data, which is true except for certain unusual conditions like a repeated substring at the end of the reference data that also appears at the beginning of the test data.)
6. Lerc · 1790085672 · focus · HN ↗
  I seem to remember it being shown for character recognition via JBIG. Maybe in Managing Gigabytes?
7. ape4 · 1790085848 · focus · HN ↗
  There would be some overlap with business sports analogies - eg team huddle.
8. chris_va · 1790085998 · focus · HN ↗
  We used a similar technique for a class project (N decades ago) to test this:
  <a href="https://en.wikipedia.org/wiki/Baconian_theory_of_Shakespeare_authorship" rel="nofollow">https://en.wikipedia.org/wiki/Baconian_theory_of_Shakespeare...
  By looking at mutual information from different authors on the same topic vs same author on different topics. As I recall, it convincingly disproved the hypothesis.
9. mgaldys4 · 1790086044 · focus · HN ↗
  That's a bit of a stretch. The LZ77 algorithm merely finds repeated items within a sliding window and encodes them into Huffman codes. A basic model should at least be capable of classification and prediction, like very early-stage machine models. The former is a fixed, quantifiable algorithm, whereas the latter becomes difficult to quantify and understand once the scale of neurons grows large. It's not very reasonable to regard the two as the same thing.
10. anthk · 1790106097 · focus · HN ↗
  Is pigz faster?
11. jjtheblunt · 1790106855 · focus · HN ↗
  we were doing this in Qualcomm 20 years ago
12. mrtnmcc · 1790106938 · focus · HN ↗
  You might also want the topic files to be compressed against each other to get a baseline matrix and then multiply any results by the inverse, assuming equal priors on the topics.
relevant_stats · 1790068055 · focus · HN ↗
Yay, another mostly AI authored piece with vibe-coded aesthetics.
Some will say that I should 'judge the idea, not the form'.
But if the author didn't find enough strength to write alone a short ~700 words summary about his work, it means he himself isn't that interested or enthusiastic about it. Why should others bother then? Particularly since low-effort like that signals possibility the whole work is superficial and derivative.
1. marand23 · 1790070738 · focus · HN ↗
  I thought it was interesting.
  1. relevant_stats · 1790083387 · focus · HN ↗
    so did I, that's why I clicked.
2. DonHopkins · 1790079043 · focus · HN ↗
  I'm judging the ideas in your post, not its form.
  Your claim that suspected AI assistance proves the author isn't interested -- and therefore that the work is probably superficial -- is unsupported.
  The article presents a working experiment, explains why naive decoding fails, describes the beam-search fix, and links the code.
  Dismissing all that with presumptuous speculation and banal boilerplate drive-by anti-AI snark adds absolutely nothing to the conversation -- and that is intrinsically poor form.
  You couldn't even find enough strength to criticize anything beyond the form, while your own form is lackluster.
  Ironically, an LLM could have written your comment and improved its form without losing anything distinctive.
  1. relevant_stats · 1790083364 · focus · HN ↗
    > I'm judging the ideas in your post, not its form.
    and simultaneously you write that 'my form is lackluster' and that 'an LLM could have written your comment and improved its form without losing anything distinctive'. We are having ourselves a small contradiction, aren't we.
    Be my guest, enjoy chatbot writing and drowning in slop. But don't encroach upon my freedom to protest it.
    DonHopkins · 1790083860 · focus · HN ↗
    Ironic, isn't it? Yet you continue your lackluster form devoid of ideas. All you can criticise is form, yet my criticism of your ideas still stands:
    Your claim that suspected AI assistance proves the author isn't interested -- and therefore that the work is probably superficial -- is unsupported.
    I also have the freedom to ironically protest the poor form of your inability to criticize ideas, as well as your poorly formulated unsupportable ideas.
    relevant_stats · 1790085783 · focus · HN ↗
    > Ironic, isn't it? Yet you continue your lackluster form devoid any criticism of ideas.
    Pointing out someone's contradiction is now being 'critical of form'? And someone's contradicting themselves is 'ironic'?
    Now I'm not even sure you know the meaning of words you use. EOT from me.
    DonHopkins · 1790139500 · focus · HN ↗
    [delayed]
3. elendilm · 1790090762 · focus · HN ↗
  Say you have been working on a multi year project writing code and putting in a lot of effort.
  If you then use AI to write a product page documentation and proof read it for correctness, would that constitute to signaling that the whole effort is superficial?
  Some work may be left to AI while you focus on the more important aspects of the work.
  Surprisingly people have got it completely backwards where they want AI to generate code and humans to write documentation.
  1. relevant_stats · 1790178250 · focus · HN ↗
    > If you then use AI to write a product page documentation and proof read it for correctness, would that constitute to signaling that the whole effort is superficial?
    It would constitute signalling possibility the whole effort is superficial.
    Look around at the amount of slop that is thrown out there using low-effort methods.
    I haven't order the work that is being presented here, I'm in no need to guarantee its correctness. I'm only bystander whose attention the OP is trying to grab by posting it on the HN. I don't know if his project is multi year or only multi prompt. The burden of proof lays on him to show he is not one of those another guys who hide emptiness behind AI generated text. Earlier the grammatically correct and carefully laid out blog post was in itself a proof that at least some effort was exerted. Now that important signal is worthless and when it's clearly written by AI it becomes an anti-signal.
    At least according to me, because the front page of the last year shows surprisingly big number people love superficial slop.
    I'll reiterate, I don't know if the work here is legit or not (and after seeing that ai documentation I have no desire to check that). Demanding from the reader to carefully look past the generated documentation and meticulously analyse the actual content is more time consuming than before and opens the gates for AI slop.
    And that attitude also makes people who are not experts on the subject more defenceless against hoaxes and deceit. If both frauds and legit createor look superficially identical, because they use the same chatbot, then whom should the member of general public trust, eh?
    elendilm · 1790207599 · focus · HN ↗
    > It would constitute signalling possibility the whole effort is superficial
    Our company has been developing many core projects for years and one among them is our in-house database Dip. Hiring a writer or spending engineer's time on Dip's docs raises operational cost. Mentality like this gets a startup killed.
    Dip's code is pure 100% human written and scrutinized by AI where appropriate. Docs are AI generated and we have it proof-read for correctness. Developers care about documentation being correct and vetted by the people who built the system.
    Bitching about docs being AI slop is nothing but virtue signaling.
    relevant_stats · 1790247290 · focus · HN ↗
    You are completely missing my point.
    We are not working for a company here. The 'gzip as a language model' is not a company project. I agree what you say would be right about inner company workings or also when working with contractors and customers. AI can save significant time, and there are another multiple guardrails in place for guaranteeing code will work - and it doesn't even matter if it's generated or written by hand.
    This experience doesn't translate to Internet news aggregator at all. We are an audience, not a paying customer. There are no guardrails. And the OP is trying to get our eyes to read his post. The only experience most of us will have with it is the text of the post - so it's basically a main thing. I'm fully in right to note the post is generated with low effort.
    Not seeing that is nothing but blindness.
    elendilm · 1790272200 · focus · HN ↗
    Well if you are concerned only about, say, news aggregation then you are entitled to your opinion.
    I enjoy quality content and despise hollow content be it AI or human generated.
    The article's content was not devoid of substance to be easily dismissed irrespective of the "signalling".
fr2029 · 1790068236 · focus · HN ↗
[dead]
modin · 1790069830 · focus · HN ↗
R. Hendricks, D. Chugtai, and J. Dunn, "Lossless compression via optimized middle-out bitstream processing," Pied Piper Inc., Palo Alto, CA, Tech. Rep. 42, Apr. 2014.
adityaathalye · 1790069976 · focus · HN ↗
I think language itself is compression, so the arxiv paper tracks for me.
Viz. if Language is compression (of thought / culture / the tacit je ne sait quois of being-to-being communication etc.), then definitionally, Language Modelling must also be Compression.
Except, language is an arbitrarily lossy compressor, who's "compression-prediction equivalence" is indeterminate and unstable, because Language co-evolves constantly; both as a function of or response to culture, as well as an influencer of culture.
So, the subjective-objective goodness of Language Models (of any kind of language) would be, at best, upper-bounded by the compression-prediction equivalence of the Languages corpus itself. And that is assuming the language corpus is perfect in every way---it captures all knowledge expressible by language and it is always in-sync with live evolution of all language expression and evolution (i.e. LLM training is not a batch job, but a real-time present continuous process).
For example, to my layperson eyes, the mathematical language of proofs actively weeds out ambiguity of subjective interpretation. Ideally, a proof ought to lead to the exact same conclusion on every single reading by any reader who can follow the steps. A proof also holds only if the rest of the formal, explicit, inviolable, internally-consistent set of axioms and results holds.
So it stands to reason that mathematical prose of proofs, being optimised as mechanical procedure of taking an open question to a deterministically closed solution, has better odds of approximating the tacit aspects of mathematical derivation.
Which makes an LLM able to construct a mathematical proof, which is mind-melting to say the least.
However, I wonder, can LLMs dream of mathematical sheep?
1. cestith · 1790084339 · focus · HN ↗
  Language is compression of a sort. A dictionary is a decompressor. You look up one word, and you may get a paragraph about its meaning. An encyclopedia can be thought of as roughly the same with more detail for some nouns.
  It makes a lot of sense why dictionary-based compression is named the way it is. A shorter symbol is used to store information that would take more symbols in the uncompressed corpus, if the shorter symbol hadn't been assigned to represent it. That's in a way just what an actual dictionary on your English professor's shelf does. The big difference is your compressor is coining new short symbols all the time.
  1. IAmBroom · 1790108991 · focus · HN ↗
    I'd say the bigger difference is the lack of homophones in a compressor.
    Languages add new words constantly, albeit slower than a computer does compressing a new file.
networked · 1790070662 · focus · HN ↗
I was curious to see how this would work with bzip2 and zstd. The source is public at <a href="https://github.com/nathanrs/gzipt" rel="nofollow">https://github.com/nathanrs/gzipt, and I asked MiMo-V2.6-Flash to fork and modify it. The answer is that bzip2 produces sequences that don't resemble human language:
```
  gzipt \
      --corpus data&#x2F;tinyshakespeare.txt \  
      --prompt $&#x27;MENENIUS:\n&#x27; \
      --length 200 \
      ;
  
  MENENIUS:
  MtLUMSeptuttyyyxyxyxyxyvyyyxyxyxyxyvyyyxyxyxyxywyvzyxyxyx
  yyxyyyxyxyxyxyxPlyxyxyxyxyxyxyxyxyxtoxzfTUS.zxzzzyzzzvzzz
  vzzzxvzyvyxyxyxyvyxyxyxyvy--,Vdvyxyxyxyxyxyxyxyxyxxy!zFlx
  zzyyxyxyxyvyxyxyxyvyySPffuyuy
```
Line breaks added. This looks roughly optimized for the most repetitive Burrows-Wheeler transform (<a href="https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transform" rel="nofollow">https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transf...). Why is it `yxyxyx` and not `yyyyyy`?
Zstandard produces whitespace with the occasional letter thrown in. To quote MiMo: "As you can see, zstd does not speak Shakespeare. ... zstd encodes a run of one repeated byte as a near-free run-length sequence, and space and newline are the cheapest literals in the corpus: ten newlines cost about the same to append ten bytes of genuine corpus text and less than nonsense does."
1. maxidog · 1790072371 · focus · HN ↗
  Did you check MiMo correctly performed this unfamiliar task before posting this comment?
  1. networked · 1790072682 · focus · HN ↗
    I did, depending on how you define "correctly". I read the code and saw that MiMo had replaced the zlib calls with a similar bz2 and compression.zstd call, removing the unavailable <a href="https://docs.python.org/3/library/zlib.html#zlib.Compress.copy" rel="nofollow">https://docs.python.org/3/library/zlib.html#zlib.Compress.co.... So I made sure the quality MiMo's work matched what I would do for a quick experiment, not that the code was free from subtle bugs.
    jrmg · 1790085425 · focus · HN ↗
    Is the fact that the original did [compress base]+[compress seq] rather than [compress [bytes + seq]] not important?
    (I honestly don’t know is gzip does something different when presented with two chunks as opposed to one, or, if it does, if bz2 has equivalent behaviour - but the difference in the code did stand out to me, and it does seem related to ‘extending the token sequence’)
    networked · 1790092111 · focus · HN ↗
    This difference doesn't matter because of how zlib works. At least by default, zlib divides the input data into its own blocks independent of the caller. If you don't feed it enough data to complete a block, it waits until you feed it more or finish the stream.
    We can test it by going back to zlib:
    def length_for(seq: bytes) -> int: - return len(bz2.compress(context + seq, level)) + return len(zlib.compress(context + seq, level))
    At temperature zero, this outputs the same sample as commit 3734bf6, the most recent commit upstream:
    MENENIUS: 'Though all at once cannq MARCIUS: I'll fight 'Though all at once cannq MARCIUannq MARCIUS: I'll fight 'Though AUFIDIUS: If I fly, Marci AUFIDIUS: If I fly, Marci AUFID AUFIDIUS: If If I fly
    I also tried LZMA for good measure:
    def length_for(seq: bytes) -> int: - return len(bz2.compress(context + seq, level)) + return len(lzma.compress(context + seq))
    The sample at temperature zero:
    MENENIUS: 'Th A carbuncle enti , as big as thou A aa
    This is followed by a lot of whitespace.
    python-lz4 gives you all newlines after the prompt. I tried debugging it, and the compressed length of different candidate seqs is the same.
2. jeremyjh · 1790075004 · focus · HN ↗
  So, you had an AI write code you don't understand, then posted output you don't understand in a comment on the internet for other humans to read?
  1. networked · 1790077713 · focus · HN ↗
    [delayed]
  2. networked · 1790078015 · focus · HN ↗
    [delayed]
  3. Dylan16807 · 1790199184 · focus · HN ↗
    No, they used AI to write code they do understand, then posted interesting results they don't understand for other humans to see.
    jeremyjh · 1790199962 · focus · HN ↗
    Then why attribute the work to Mimo?
elendilm · 1790071038 · focus · HN ↗
Good article.
Compression is a property of language.
A seemingly simple sentence like "I had lunch" has enormous amount of information compressed inside it. The word lunch is a compressed form of "having food at noon" and "noon" in turn is a compressed form of "Sun's position against Earth's rotation" and so on and so forth.
Every sentence has layers of compressed sentences. How many layers one decompress is upto the person.
colinmarc · 1790072510 · focus · HN ↗
Would this work with video compression? Video codecs encode a lot of meaning; they use motion vectors to track the movement of objects on screen, for example.
segmondy · 1790078690 · focus · HN ↗
I think not. it's true that a large language model compresses knowledge and allows us to decompress knowledge. but gzip compresses data and not knowledge. with a LM you can decompress various forms of knowledge from the same data. gzip is a 1 to 1 kind of decompression where as an LM is a 1 to infinity kind of decompression.
DonHopkins · 1790078788 · focus · HN ↗
Will Wright once described the same connection from the opposite direction: compression as procedural content generation.
In the 2023 discussion of "Demoscene accepted as UNESCO cultural heritage in The Netherlands" I posted a transcript from a video of Will Wright discussing the demo scene:
<a href="https://news.ycombinator.com/item?id=36599415">https://news.ycombinator.com/item?id=36599415
Will Wright Discusses the Demoscene:
<a href="https://www.youtube.com/watch?v=m7iuFVmTJus" rel="nofollow">https://www.youtube.com/watch?v=m7iuFVmTJus
>You can take any piece of content in the game, and imagine an algorithmic solution to it. Or also, you know, a way that the player could customize that object of thing.
>There's this group in Europe called the Demoscene that make these very elaborate demos for a computer that fit into very tiny little memory blocks, you know like 64K of memory, and you run the thing, and in fact it algorithmically generates about 100 megabytes worth of data, you know these rich 3D environment, generated music, generated wave files, generated animation.
>And they're developing techniques to generate, you know, huge amounts of interesting data, with very very simple, elegant, compression algorithms.
>And this is a skill that game developers used to have, back in the 8-bit days. That was the only ways to do a game like Karateka(?), was to find all these little tips and tricks to compress things and generate them algorithmically.
>But since the CD-ROM came out, and very cheap hard drives, storage is cheap, so basically we've lost that skill set, and now we attack all those problems with brute force. I think we've lost something by dropping that skill set.
[...]
<a href="https://news.ycombinator.com/item?id=36613058">https://news.ycombinator.com/item?id=36613058
[...] Here's a simple low-tech pre-LLM example that shows the equivalence of compression and procedural content generation:
Take a huge text file of HN postings, and compress it with gzip or compress or some other robust compression algorithm. The better the algorithm, the more the output will look like random noise. Then slice the compressed file in half, and replace the second half with random numbers. Then uncompress it. You'll find that at the point you sliced it, it keeps on writing out almost plausible text for a while, consisting of highly probably snippets of commonly encountered words and phrases, then goes downhill towards incoherence. It's not as coherent or confident as an LLM, but the point is to show how low the bar is for using compression for procedural content generation.
LLMs are essentially a form of compression of the world's knowledge or whatever they're trained on, not just word frequencies or pixel patterns, but also concepts and ideas. [...]
kindkang2024 · 1790080681 · focus · HN ↗
[dead]
mohd_rafay · 1790082852 · focus · HN ↗
[dead]
greengemz · 1790083551 · focus · HN ↗
[dead]
ronfriedhaber · 1790085848 · focus · HN ↗
See also the Hutter Prize [1]. Compression is a subset of Intelligence.
[1] <a href="https://en.wikipedia.org/wiki/Hutter_Prize" rel="nofollow">https://en.wikipedia.org/wiki/Hutter_Prize
logicallee · 1790085969 · focus · HN ↗
It's an interesting experiment, though the examples in the article didn't really seem that coherent, so I'm not sure that the answer to the title is "yes". Along the same lines, a few years ago I made a very tiny transformer demonstration in javascript, also trained on Shakespeare, but it was too small to produce text like Shakespeare. You can try it here:
<a href="https://taonexus.com/mini-transformer-in-js.html" rel="nofollow">https://taonexus.com/mini-transformer-in-js.html
I guess the results are similar to the article, maybe a little more coherent since the tokens are words.
dominotw · 1790086154 · focus · HN ↗
3blue did a series of videos on this
<a href="https://www.youtube.com/watch?v=l6DKRf-fAAM" rel="nofollow">https://www.youtube.com/watch?v=l6DKRf-fAAM
_def · 1790087611 · focus · HN ↗
Fun topic but generated article text and then not even actually using gzip? Rubs me in a weird way.
aghilmort · 1790088452 · focus · HN ↗
zip2zip paper by Geng et al. also exploited LZ/LZW and made the rounds a while back; novel approach that uses zip content as output compression adapter
<a href="https://arxiv.org/abs/2506.01084" rel="nofollow">https://arxiv.org/abs/2506.01084
jcattle · 1790089946 · focus · HN ↗
Can JPEG be a vision model?
kazinator · 1790090223 · focus · HN ↗
[delayed]
northlondoner · 1790090817 · focus · HN ↗
Yes.
northlondoner · 1790090990 · focus · HN ↗
Reminded me Google's earlier paper, "Language Modeling Is Compression" <a href="https://arxiv.org/abs/2309.10668" rel="nofollow">https://arxiv.org/abs/2309.10668
Learning being a compression is also recently proposed as Gibbs compression proposition.
See Gibbs randomness-compression proposition <a href="https://arxiv.org/abs/2505.23869v5" rel="nofollow">https://arxiv.org/abs/2505.23869v5
lotus_uk · 1790091119 · focus · HN ↗
[dead]
corbinvachal · 1790091821 · focus · HN ↗
[dead]
js98 · 1790095561 · focus · HN ↗
“Solving” mnist with gzip: <a href="https://jakobs.dev/solving-mnist-with-gzip/" rel="nofollow">https://jakobs.dev/solving-mnist-with-gzip/
teh64 · 1790104650 · focus · HN ↗
There is an older paper that also looks at gzip for ML: <a href="https://arxiv.org/abs/2212.09410" rel="nofollow">https://arxiv.org/abs/2212.09410
Video from tsoding where he implements the algorithm: <a href="https://www.youtube.com/watch?v=9n39SbRPXKQ" rel="nofollow">https://www.youtube.com/watch?v=9n39SbRPXKQ
adamgordonbell · 1790105622 · focus · HN ↗
Compressing something and next token prediction are very related. Once you understand the connection, things like ts_zip and hutter prize make a lot more sense.
<a href="https://bellard.org/ts_zip/" rel="nofollow">https://bellard.org/ts_zip/ <a href="https://corecursive.com/the-hutter-prize/" rel="nofollow">https://corecursive.com/the-hutter-prize/
aitoolcrux · 1790129277 · focus · HN ↗
[dead]
Klaster_1 · 1790142412 · focus · HN ↗
Can't wait for someone to make Doom run on gzip.
rsrsrs86 · 1790204388 · focus · HN ↗
Funny, this kind of knowledge used to be a common theme in ML/DS roles. Funny to see that people are thinking about it now by talking to AI (the article does look very AI written). Funny that people can enter very deep rabbit holes because they didn’t study something and guess they are making progress on something that is commonly known. Dunning Krueger prime time 2026

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.