How to Run Bonsai 2 27B Locally With GGUF or MLX

01

Overview

Bonsai 2 27B is small enough on disk to make local use unusually practical for a 27B-class model, but installing it is not the same as downloading an ordinary GGUF and opening it in any llama.cpp app.

The key compatibility detail is simple: PrismML currently says the main Bonsai 2 low-bit builds need its modified runtime. Until the required activation transform is fully upstreamed, using the wrong llama.cpp build can fail — or, for one development format, appear to load while producing invalid output.

This guide follows PrismML's current official repositories rather than assuming generic GGUF compatibility.

02

Choose GGUF or MLX first

PrismML currently provides two practical local paths.

GGUF

Use the official prism-ml/Ternary-Bonsai-2-27B-gguf release if you want PrismML's llama.cpp-based route across systems such as Linux, Windows or macOS and supported CPU/GPU backends.

The principal packages documented by PrismML are approximately:

• PTQ1_0: 5.95 GB.
• PQ2_0: 7.21 GB.

For image input, download the separate vision projection file as documented by the model repository.

MLX

Apple Silicon users can use prism-ml/Ternary-Bonsai-2-27B-mlx-2bit. Hugging Face currently lists the MLX package at about 8.6 GB.

Do not compare that number directly with the 5.95 GB GGUF as though one listing must be wrong. They use different packaging and low-bit representations.

03

Recommended setup: PrismML's Bonsai demo

PrismML maintains PrismML-Eng/Bonsai-demo specifically to set up and run its Bonsai family locally. The repository currently makes Bonsai 2 27B the default model.

This is the safest starting point because the setup scripts install the runtime version PrismML expects rather than relying on whatever llama.cpp build happens to be installed on your computer.

On a Unix-like system, follow the current commands in the repository's README. Do not copy an old command from a third-party guide without checking the repo first because runtime compatibility is actively changing.

The demo documents local operation on Mac with Metal and on Linux/Windows with backends including CUDA, Vulkan, ROCm or CPU where supported.

04

Why stock llama.cpp is a problem right now

Bonsai 2 uses a matching activation transform for its ternary weights. PrismML's current demo documentation says the main Bonsai 2 GGUF and MLX bands need its fork for now.

There is an especially useful warning in PrismML's Ternary-Bonsai-2-27B-gguf-dev model card. A testing Q2_0 file uses a type and architecture that stock llama.cpp recognizes, so stock llama.cpp can load it. But because the necessary activation transform is missing, PrismML says the result is gibberish without a warning.

The production PTQ1_0 and PQ2_0 packages use types stock llama.cpp does not recognize, which is safer because an unsupported runtime should stop instead of silently returning bad inference.

The practical rule is: use the runtime linked by PrismML and recheck upstream support before changing to stock llama.cpp.

05

Running the text model

Once the PrismML demo/runtime is installed, select Bonsai 2 27B and let its setup process download the appropriate model package. The exact invocation can evolve as PrismML upstreams support, so treat the current repository README as the command-level source of truth.

For text-only work you do not need the optional vision projection. This keeps the download and runtime footprint lower.

Typical uses include local chat, coding assistance, reasoning tasks and agent/tool-call experiments. Tool support also depends on the surrounding application and chat template; the model alone does not magically provide external tools.

06

Enabling vision

Bonsai 2 is multimodal. The official GGUF card lists an optional Q8_0 vision projection package of roughly 0.63 GB.

Load that component only if your runtime and interface support the model's vision path. PrismML's demo documentation describes sending photos, screenshots and PDFs with both its llama.cpp and MLX workflows.

A PDF being accepted by the demo does not necessarily mean the language model directly consumes a PDF file format internally; the application can preprocess pages into supported visual inputs. Follow the runtime's documented workflow.

07

How much RAM or VRAM do you need?

There is no trustworthy universal minimum that can be calculated from the 5.95 GB download alone.

The model file is only one part of runtime memory. You also need capacity for runtime overhead, context/cache, activations, the operating system and — if enabled — vision components. Longer contexts increase memory pressure.

PrismML's public material demonstrates operation on consumer hardware and publishes measurements for devices including Apple Silicon and NVIDIA GPUs, but those examples are not a promise that every machine with a particular RAM number will deliver the same speed or context capacity.

A cautious approach is to:

1. Use the smaller PTQ1_0 build when footprint is the priority.

2. Begin with a modest context length rather than immediately requesting the full 262K window.

3. Close memory-heavy applications during initial testing.

4. Test text-only inference before adding the vision projection.

5. Increase context and workload after confirming stable operation.

Treat any exact minimum-RAM or VRAM figure from an unofficial guide as a configuration-specific estimate unless PrismML documents the same requirement.

08

What about the 262K context window?

The official model card lists 262K tokens inherited from Qwen3.8 27B. That is a model capability, not a guarantee that the complete window is practical on every local device.

Context memory, backend implementation and workload all matter. A laptop that comfortably runs short prompts may not comfortably run a near-maximum context.

09

Mac / Apple Silicon

Apple Silicon has two routes: PrismML's llama.cpp/Metal setup and the dedicated MLX package. The MLX build is the more platform-specific option.

PrismML reports local Apple performance in its own tests, but speed varies by chip, memory bandwidth, context and selected package. Use those figures as vendor measurements, not guaranteed throughput.

10

Windows and Linux

PrismML's demo lists Windows/Linux support through its llama.cpp workflow with backends such as CUDA, Vulkan, ROCm and CPU. GPU compatibility depends on the backend and hardware.

Again, the important part is not simply having llama.cpp installed: use a build that contains Bonsai 2's required runtime changes.

11

Common failure: the model loads but output is nonsense

If you are experimenting with the development Q2_0 build and get nonsensical output, verify that you are not running it with stock llama.cpp. PrismML explicitly documents this failure mode.

For the main PTQ1_0/PQ2_0 packages, an unsupported runtime is more likely to reject the model format instead.

12

Common failure: a GGUF app says the format is unsupported

That can be expected while Bonsai 2 support is not fully upstream. A program supporting ordinary GGUF files does not automatically support every custom quantization/packing type.

Use PrismML's documented runtime or wait until your chosen application confirms Bonsai 2 support.

13

Which Bonsai 2 package should you use?

Choose PTQ1_0 GGUF when the smallest footprint is your main goal and PrismML's llama.cpp route fits your system.

Choose PQ2_0 GGUF when you want PrismML's alternative 2-bit-slot representation and its larger footprint is acceptable.

Choose MLX 2-bit when you specifically want the Apple Silicon MLX workflow.

Add the vision projection only when you need image/document visual input.

14

Before you install: recheck compatibility

Bonsai 2's runtime situation is moving quickly. PrismML says related ternary formats are being upstreamed into mainline llama.cpp, while Bonsai 2's current formats still need its fork.

Before installation, check the latest Bonsai demo README and official model card. Once upstream support lands, the best setup path may become much simpler.

15

FAQ

Can Ollama or LM Studio run Bonsai 2 27B?

Do not assume so merely because the model is distributed as GGUF. Support depends on whether the app's bundled runtime includes Bonsai 2's required ternary implementation. Check the application's current release notes and PrismML's compatibility guidance.

Does Bonsai 2 require an NVIDIA GPU?

No. PrismML documents CPU, Metal and multiple GPU backends, and publishes an MLX build for Apple Silicon. Actual performance varies significantly.

Can I use the full 262K context on a laptop?

The model supports a 262K context, but whether the maximum is practical depends on available memory and runtime behavior. Start smaller and measure.

Do I need the vision file?

Only for multimodal/image workflows. Text-only inference does not need the separate vision projection.

Is 5.95 GB the RAM requirement?

No. It is approximately the PTQ1_0 language-model file size. Total runtime memory is higher and workload-dependent.