How to Run K2 Horizon Locally

01

Start by choosing a practical model size

For local experiments, 3.7B and 7B are the clearest starting points in the current official documentation. The 32B and 36B-A4B models move toward workstation or server-class use, while 375B-A23B is an enterprise-scale serving target rather than a normal desktop model.

Do not convert those deployment classes into fixed VRAM numbers. Precision, quantization, context length, KV cache, runtime overhead and CPU/GPU offloading all change the real requirement.

02

Option 1: Transformers for direct Python inference

The official 3.7B and 7B cards include Transformers examples using trust_remote_code and automatic device mapping. This is the simplest path when you want to inspect model behavior directly in Python without running a separate API server.

The current cards document a validated release stack including Transformers 5.15.0, PyTorch 2.13.0 and Safetensors 0.8.0. Treat those as validated release-state versions, not a guarantee that every future dependency combination is equivalent.

03

Option 2: vLLM for an OpenAI-compatible API

IFM’s 7B quick start shows vLLM serving with BF16, trust_remote_code, a K2 Horizon reasoning parser, automatic tool choice and the K2 Horizon tool-call parser.

Once served, applications can call the local OpenAI-compatible chat-completions endpoint. This is a practical route for agent frameworks and internal tools that already support a configurable OpenAI-style base URL.

04

Option 3: SGLang for validated serving

IFM also documents SGLang recipes for Horizon models. The current 7B card points to a validated BF16 configuration using the K2 Horizon reasoning parser and FlashAttention-based serving.

Use the live model card or linked cookbook when deploying rather than freezing a launch-week revision forever. Serving flags, tested revisions and backend recommendations can change quickly after a new model launch.

05

Option 4: GGUF and llama.cpp-compatible workflows

IFM publishes official GGUF repositories for several K2 Horizon sizes. The 7B repository says its files are intended for llama.cpp and include tokenizer metadata plus a llama.cpp-compatible chat template.

06

Reasoning and context settings

The current 3.7B and 7B cards recommend high reasoning effort, temperature 1.0, top_p 0.95 and at least 32,768 output tokens for reproducing reported reasoning evaluations. Production workloads can trade that depth for latency.

Both cards document a native 524,288-token context window, but the 7B vLLM example configures 131,072 tokens. Longer context increases memory pressure, especially through the KV cache, so begin with the context your workload actually needs.

07

A practical local workflow

1. Start with 3.7B or 7B unless you have a clear reason to move larger.
2. Use Transformers for direct experimentation.
3. Use vLLM or SGLang when you want a local API or higher-throughput serving.
4. Use an official GGUF only after verifying current runtime architecture support.
5. Start below the maximum context length and measure memory and latency.
6. Pin revisions when reproducibility matters.

The main K2 Horizon guide explains licensing, model families, API availability and benchmark boundaries. Use it alongside this setup guide when deciding whether local inference or a hosted partner is the better fit.