Bonsai-demo

# Bonsai Demo

Backend and model format compatibility: [BACKEND-SUPPORT.md](BACKEND-SUPPORT.md).

<p align="center">
  <img src="./assets/bonsai-logo.svg" width="280" alt="Bonsai">
</p>

<p align="center">
  <a href="https://prismml.com"><b>Website</b></a> &nbsp;|&nbsp;
  <a href="https://github.com/PrismML-Eng/Bonsai-demo"><b>GitHub</b></a> &nbsp;|&nbsp;
  <a href="https://discord.gg/prismml"><b>Discord</b></a>
</p>

<p align="center">
  <b>Models:</b>
  <a href="https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf">Bonsai 2 27B GGUF</a> ·
  <a href="https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-mlx-2bit">Bonsai 2 27B MLX</a>
</p>

---

Run **Bonsai 2 2B, 4B, 9B, and 27B** locally on Mac (Metal), Linux/Windows (CUDA, Vulkan, ROCm), or CPU.

For the earlier **Bonsai 1-bit and Ternary-Bonsai** models (27B, 8B, 4B, 1.7B), see
[Bonsai1_README.md](Bonsai1_README.md). For troubleshooting, see [FAQ.md](FAQ.md).

## 🌱 Bonsai 2: 2B, 4B, 9B, and 27B

**Bonsai 2 now includes 2B, 4B, and 9B models alongside 27B.** Pick a smaller model
for a lighter local setup; 27B remains the demo's default. GGUF and MLX checkpoints
are available for each size.

The following benchmark and memory highlights describe **27B**; see each smaller
model's card for its specifications:

- **98.2% of FP16 intelligence retained** at roughly a ninth of the size, with the reasoning core
  intact: math within half a point of full precision, coding level with the baseline.
- **Vision:** send photos, screenshots and PDFs and ask about them, on both llama.cpp and MLX
  (see [VISION.md](VISION.md)).
- **Agentic tool calling:** native OpenAI-style `tool_calls` with full round-trips, plus MCP servers
  in both demo UIs (see [TOOLS.md](TOOLS.md)).
- **Thinking:** a reasoning model; pick the reasoning effort per chat in the UI or budget it per request.
- **262K-token context**, kept practical on-device by the hybrid-attention backbone.
- **1.75 bits per weight** in the `PTQ1_0` packing. A second packing, `PQ2_0`, trades 1.3 GB for
  faster prompt processing and is what this demo downloads by default.
  See [MODEL-FORMATS.md](MODEL-FORMATS.md).

**Official `Q2_0` works with upstream llama.cpp [b11556](https://github.com/ggml-org/llama.cpp/releases/tag/b11556) or newer.**
`PQ2_0` and `PTQ1_0` still require the [PrismML fork](https://github.com/PrismML-Eng/llama.cpp).
This demo downloads `PQ2_0` and the matching fork binaries by default.
All three packings contain the same model weights, with different memory/speed trade-offs.

Quick Start below gets you there in two commands: `./setup.sh` downloads Bonsai 2 27B, then
`./scripts/start_llama_server.sh` gives you chat, vision and tools at http://localhost:8080.

### Choose a smaller model

```bash
export BONSAI_MODEL=4B    # 2B, 4B, 9B, or 27B
./setup.sh
./scripts/start_llama_server.sh
```

On Windows, set `$env:BONSAI_MODEL = "4B"` before setup and launch. For Apple Silicon,
use `./scripts/run_mlx.sh` or `./scripts/start_mlx_server.sh`