How to Run Hy4 Preview Locally with vLLM or SGLang

01

Before you start

Tencent's Hy4 Preview is open-weight, but “run locally” needs a qualifier: it has 770B total parameters with 49B active per token. Practical deployment is aimed at multi-GPU infrastructure.

Start with the official tencent/Hy4-preview or tencent/Hy4-preview-FP8 checkpoints. Tencent licenses both under Apache 2.0 and publishes model-specific reasoning and tool-call parsers.

02

Option 1: vLLM

Tencent documents vLLM as an official serving path. Use the compatible version in the current repository, download the checkpoint and launch it with tensor parallelism sized for your GPU fleet. The resulting server exposes an OpenAI-compatible API.

03

Option 2: SGLang

Tencent's current SGLang recipe uses the FP8 checkpoint, Hy4-specific tool and reasoning parsers, EAGLE-style speculative decoding and the native MTP layer.

The official example uses tensor parallelism across eight devices. That proves eight-way serving is a supported example configuration; it is not a universal minimum-hardware claim.

04

FP8, context length and hardware sizing

The official FP8 checkpoint is the clearest starting point for reducing memory pressure. It remains extremely large, so consumer single-GPU claims should be treated cautiously unless they include a reproducible conversion and measured setup.

Tencent does not publish one universal minimum-VRAM figure. Precision, tensor parallelism, KV cache, speculative decoding, concurrency and requested context all matter. Attempting a 1M-token context can increase memory requirements sharply compared with short-context tests.

Tencent points to AngelSlim for compression and quantization. Any community conversion should be labelled community-produced rather than presented as an official Tencent checkpoint.

05

Practical deployment checklist

1. Choose the official full or FP8 checkpoint.
2. Confirm current Transformers and runtime versions in Tencent's repository.
3. Select vLLM or SGLang and size tensor parallelism for the actual GPU fleet.
4. Start with a shorter context before testing extreme context lengths.
5. Verify the Hy4 tool and reasoning parsers.
6. Benchmark latency, throughput and memory on the production workload.

Self-host when data control, dedicated capacity or custom infrastructure justify it. Hosted access is much simpler for experimentation with a model at this scale.