Thread

Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint

589 points · 200 comments · JonSchneider

simonw · 1789683185 · focus · HN ↗
If you want to try out out the GGUFs from <a href="https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#these-files-need-our-llamacpp-build" rel="nofollow">https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism's llama.cpp fork to get them to work, from <a href="https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b10685-7dffb15" rel="nofollow">https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-...
This should work:
```
  cd &#x2F;tmp

  # Get the Prism macOS runtime
  curl -fL https:&#x2F;&#x2F;github.com&#x2F;PrismML-Eng&#x2F;llama.cpp&#x2F;releases&#x2F;download&#x2F;prism-b10685-7dffb15&#x2F;llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz
  tar -xzf bonsai-runtime.tar.gz

  # Get the ~5.95 GB GGUF model:
  curl -fL https:&#x2F;&#x2F;huggingface.co&#x2F;prism-ml&#x2F;Ternary-Bonsai-2-27B-gguf&#x2F;resolve&#x2F;main&#x2F;Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf

  # Run the server, I used port 8331
  .&#x2F;llama-prism-b10685-7dffb15&#x2F;llama-server \
    -m Ternary-Bonsai-2-27B-PTQ1_0.gguf \
    --port 8331 -ngl 99 -fa on -c 32768
```
Then open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this:
```
  uvx llm openai endpoint http:&#x2F;&#x2F;127.0.0.1:8331&#x2F;v1 \
    --model bonsai-2-27b --responses hi
```
That's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said "ggml_metal_device_init: - the tensor API is not supported in this environment - disabling".
1. refibrillator · 1789685765 · focus · HN ↗
  Where did you get these instructions?
  They have a demo repo with a setup.sh script:
  <a href="https://github.com/PrismML-Eng/Bonsai-demo" rel="nofollow">https://github.com/PrismML-Eng/Bonsai-demo
  The release tag and weight file you suggest doesn’t match what they wrote.
  1. simonw · 1789686424 · focus · HN ↗
    I figured them out, starting from the GGUF on Hugging Face.
    If you have found better instructions and they work then use those instead!
    Personally I prefer to download models directly rather than running some `./setup.sh` script where I need to then review what it does first.
    refibrillator · 1789687285 · focus · HN ↗
    Yeah just wanted to mention in case it explains the 2x lower throughout you are seeing on M5. To be fair their documentation is a bit inconsistent in some spots.
    Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.
    fnordpiglet · 1789750719 · focus · HN ↗
    They’re fairly useful as constrain domain classifiers due to the low memory requirements means you can stuff a lot of them into a less expensive GPU farm and get really decent throughout with pretty good results over all. At least that’s my experience. I wouldn’t bother using a tiny model for coding - but the world is full of abductive reasoning tasks that don’t involve coding.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.