‹ BackHN Continuity

Thread

Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint

589 points · 200 comments · JonSchneider

  1. simonw · · focus · HN ↗
    If you want to try out out the GGUFs from <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;prism-ml&#x2F;Ternary-Bonsai-2-27B-gguf#these-files-need-our-llamacpp-build" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;prism-ml&#x2F;Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism&#x27;s llama.cpp fork to get them to work, from <a href="https:&#x2F;&#x2F;github.com&#x2F;PrismML-Eng&#x2F;llama.cpp&#x2F;releases&#x2F;tag&#x2F;prism-b10685-7dffb15" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;PrismML-Eng&#x2F;llama.cpp&#x2F;releases&#x2F;tag&#x2F;prism-...

    This should work:

      cd &#x2F;tmp

  # Get the Prism macOS runtime
  curl -fL https:&#x2F;&#x2F;github.com&#x2F;PrismML-Eng&#x2F;llama.cpp&#x2F;releases&#x2F;download&#x2F;prism-b10685-7dffb15&#x2F;llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz
  tar -xzf bonsai-runtime.tar.gz

  # Get the ~5.95 GB GGUF model:
  curl -fL https:&#x2F;&#x2F;huggingface.co&#x2F;prism-ml&#x2F;Ternary-Bonsai-2-27B-gguf&#x2F;resolve&#x2F;main&#x2F;Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf

  # Run the server, I used port 8331
  .&#x2F;llama-prism-b10685-7dffb15&#x2F;llama-server \
    -m Ternary-Bonsai-2-27B-PTQ1_0.gguf \
    --port 8331 -ngl 99 -fa on -c 32768
    Then open http:&#x2F;&#x2F;localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this:

      uvx llm openai endpoint http:&#x2F;&#x2F;127.0.0.1:8331&#x2F;v1 \
    --model bonsai-2-27b --responses hi
    That&#x27;s running at ~20 token&#x2F;second for me on an M5 Pro (after a server restart I got 44 token&#x2F;second, not sure why), but I&#x27;m pretty sure something isn&#x27;t working right, on startup the server said &quot;ggml_metal_device_init: - the tensor API is not supported in this environment - disabling&quot;.
    1. simonw · · focus · HN ↗
      I used that to Generate an SVG of a pelican riding a bicycle:

      <a href="https:&#x2F;&#x2F;tools.simonwillison.net&#x2F;markdown-svg-renderer?url=https%3A%2F%2Fgist.github.com%2Fsimonw%2Fba4cf3a88f4e7dc32994f2672150f770" rel="nofollow">https:&#x2F;&#x2F;tools.simonwillison.net&#x2F;markdown-svg-renderer?url=ht...

      It took 18 minutes 20 seconds. Pretty decent for a 5.5GB model file.

      1. kadoban · · focus · HN ↗
        Honestly looks pretty good except whatever is going on with its booty. Is that an ass helmet? I cannot parse what&#x27;s going on there.
        1. Forgeties79 · · focus · HN ↗
          I think it’s supposed to be a wing
        2. bigwheels · · focus · HN ↗
          I like the lens effect behind the rear tire.
        3. tomcam · · focus · HN ↗
          Like you&#x27;ve never worn an ass helmet
          1. kadoban · · focus · HN ↗
            Only because I hadn&#x27;t previously thought of it xD Step up from the standard ass-hat for sure.
      2. [deleted] · · focus · HN ↗

        [deleted]

      3. rahimnathwani · · focus · HN ↗
        M1 Pro, same prompt, same cli options:

          32,706 tokens
  38min 19s
  14.22 t&#x2F;s
      4. raylad · · focus · HN ↗
        How does that compare with the bf16 version?

        For my &quot;Please recite Jabberwocky&quot; test the bf16 almost passes but the ternary and even fp8 versions fail badly.

        1. ctolsen · · focus · HN ↗
          Definitely a little better.

          <a href="https:&#x2F;&#x2F;gist.github.com&#x2F;ctolsen&#x2F;b2883e7cbf5e4357fa04366019e60bfe" rel="nofollow">https:&#x2F;&#x2F;gist.github.com&#x2F;ctolsen&#x2F;b2883e7cbf5e4357fa04366019e6...

      5. shmoil · · focus · HN ↗
        Can you ask it for an SVG of a bicycle riding a pelican? Thanks.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.