Thread · HN Continuity

I was curious to see how this would work with bzip2 and zstd. The source is public at <a href="https://github.com/nathanrs/gzipt" rel="nofollow">https://github.com/nathanrs/gzipt, and I asked MiMo-V2.6-Flash to fork and modify it in a straightforward way. The answer is that bzip2 produces sequences that don't resemble human language:

  gzipt \
      --corpus data&#x2F;tinyshakespeare.txt \  
      --prompt $&#x27;MENENIUS:\n&#x27; \
      --length 200 \
      ;
  
  MENENIUS:
  MtLUMSeptuttyyyxyxyxyxyvyyyxyxyxyxyvyyyxyxyxyxywyvzyxyxyx
  yyxyyyxyxyxyxyxPlyxyxyxyxyxyxyxyxyxtoxzfTUS.zxzzzyzzzvzzz
  vzzzxvzyvyxyxyxyvyxyxyxyvy--,Vdvyxyxyxyxyxyxyxyxyxxy!zFlx
  zzyyxyxyxyvyxyxyxyvyySPffuyuy

Line breaks added. This looks roughly optimized for the most repetitive Burrows-Wheeler transform (<a href="https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transform" rel="nofollow">https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transf...). Why are they runs of alternating symbols and not one symbol?

Zstandard produces whitespace with the occasional letter thrown in. To quote MiMo: "As you can see, zstd does not speak Shakespeare. ... zstd encodes a run of one repeated byte as a near-free run-length sequence, and space and newline are the cheapest literals in the corpus: ten newlines cost about the same to append ten bytes of genuine corpus text and less than nonsense does."