Thread

1. networked · 1790070662 · focus · HN ↗

   I was curious to see how this would work with bzip2 and zstd. The source is public at <a href="https:&#x2F;&#x2F;github.com&#x2F;nathanrs&#x2F;gzipt" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;nathanrs&#x2F;gzipt, and I asked MiMo-V2.6-Flash to fork and modify it in a straightforward way. The answer is that bzip2 produces sequences that don&#x27;t resemble human language:

     gzipt \
         --corpus data/tinyshakespeare.txt \  
         --prompt $'MENENIUS:\n' \
         --length 200 \
         ;
     
     MENENIUS:
     MtLUMSeptuttyyyxyxyxyxyvyyyxyxyxyxyvyyyxyxyxyxywyvzyxyxyx
     yyxyyyxyxyxyxyxPlyxyxyxyxyxyxyxyxyxtoxzfTUS.zxzzzyzzzvzzz
     vzzzxvzyvyxyxyxyvyxyxyxyvy--,Vdvyxyxyxyxyxyxyxyxyxxy!zFlx
     zzyyxyxyxyvyxyxyxyvyySPffuyuy
   

   Line breaks added. This looks roughly optimized for the most repetitive Burrows-Wheeler transform (<a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Burrows%E2%80%93Wheeler_transform" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Burrows%E2%80%93Wheeler_transf...). Why are they runs of alternating symbols and not one symbol?

   Zstandard produces whitespace with the occasional letter thrown in. To quote MiMo: &quot;As you can see, zstd does not speak Shakespeare. ... zstd encodes a run of one repeated byte as a near-free run-length sequence, and space and newline are the cheapest literals in the corpus: ten newlines cost about the same to append ten bytes of genuine corpus text and less than nonsense does.&quot;

   1. maxidog · 1790072371 · focus · HN ↗

      Did you check MiMo correctly performed this unfamiliar task before posting this comment?

      1. networked · 1790072682 · focus · HN ↗

         I did. I read the code to make sure the quality of MiMo&#x27;s work matched mine for a quick experiment, though not that the code was free from subtle bugs.

         This was the main change for bzip2:

           @@ -33,19 +34,16 @@ def candidate_lengths(
                level: int = 9,
                pool: ThreadPoolExecutor | None = None,
            ) -> list[int]:
           -    """Compressed length of ``context + seq`` for each seq, sharing the context.
           +    """Compressed length of ``context + seq`` for each seq.
           
           -    Compresses ``context`` once into a ``compressobj``, then clones its encoder
           -    state per candidate and feeds only that candidate. Identical to
           -    ``len(zlib.compress(context + seq, level))`` for each seq, but the expensive
           -    match search over ``context`` happens a single time.
           +    Unlike ``zlib``'s ``compressobj``, Python's ``BZ2Compressor`` cannot be
           +    snapshotted mid-stream, and bzip2's move-to-front + Huffman stages see the
           +    whole block, so every candidate recompresses the full context. Threads
           +    still scale because ``bz2`` releases the GIL.
                """
           -    base = zlib.compressobj(level)
           -    head = len(base.compress(context))
           
                def length_for(seq: bytes) -> int:
           -        clone = base.copy()
           -        return head + len(clone.compress(seq) + clone.flush(zlib.Z_FINISH))
           +        return len(bz2.compress(context + seq, level))
           
                if pool is not None:
                    return list(pool.map(length_for, sequences))

         1. jrmg · 1790085425 · focus · HN ↗

            Is the fact that the original did [compress base]+[compress seq] rather than [compress [bytes + seq]] not important?

            (I honestly don’t know is gzip does something different when presented with two chunks as opposed to one, or, if it does, if bz2 has equivalent behaviour - but the difference in the code did stand out to me, and it does seem related to ‘extending the token sequence’)

            1. networked · 1790092111 · focus · HN ↗

               This difference doesn&#x27;t matter because of how zlib works. At least by default, zlib divides the input data into its own blocks independent of the caller. If you don&#x27;t feed it enough data to complete a block, it waits until you feed it more or finish the stream.

               We can test it by going back to zlib:

                      def length_for(seq: bytes) -> int:
                 -        return len(bz2.compress(context + seq, level))
                 +        return len(zlib.compress(context + seq, level))
               

               At temperature zero, this outputs the same sample as commit 3734bf6, the most recent commit upstream:

                 MENENIUS:
                 'Though all at once cannq
               
                 MARCIUS:
                 I'll fight
                 'Though all at once cannq
               
                 MARCIUannq
                 
                 MARCIUS:
                 I'll fight
                 'Though
                 
                 AUFIDIUS:
                 If I fly, Marci
                 
                 AUFIDIUS:
                 If I fly, Marci
                 
                 AUFID
                 
                 AUFIDIUS:
                 If
                 If I fly
               

               I also tried LZMA for good measure:

                      def length_for(seq: bytes) -> int:
                 -        return len(bz2.compress(context + seq, level))
                 +        return len(lzma.compress(context + seq))
               

               The sample at temperature zero:

                 MENENIUS:
                 'Th
                 
                 A carbuncle enti
                 
                 , as big as thou
                 
                 
                 A aa
               

               This is followed by a lot of whitespace.

               python-lz4 gives you all newlines after the prompt. I tried debugging it, and the compressed length of different candidate seqs is the same.

   2. jeremyjh · 1790075004 · focus · HN ↗

      So, you had an AI write code you don&#x27;t understand, then posted output you don&#x27;t understand in a comment on the internet for other humans to read?

      1. [deleted] · 1790077713 · focus · HN ↗

         [deleted]

      2. [deleted] · 1790078015 · focus · HN ↗

         [deleted]

      3. Dylan16807 · 1790199184 · focus · HN ↗

         No, they used AI to write code they do understand, then posted interesting results they (partially) don&#x27;t understand for other humans to see.

         1. jeremyjh · 1790199962 · focus · HN ↗

            Then why attribute the work to MiMo? People mostly only attribute the model when they&#x27;ve outsourced the understanding along with the coding. If I understand the code I don&#x27;t bother to mention the tools I used to create it.

Unofficial Hacker News client; not affiliated with Y Combinator.