How to Run Qwen3.8 Flash Next Locally: GGUF, Hardware and Benchmarks

01

Before you try to run it locally

Qwen3.8-Flash-Next activates about 6B parameters per token, but the released model still contains a 125B main network, 51B N-gram embeddings and a 4B MTP component. Low active compute does not make the complete weight set fit like a 6B model.

The official Hugging Face repository is roughly 360 GB in its original weight format. Community GGUF conversions range from about 72.5 GB at 1-bit to 188 GB at 8-bit, before runtime overhead and context memory.

02

Official weights and license

Qwen publishes the official post-trained weights at Qwen/Qwen3.8-Flash-Next on Hugging Face. The repository contains Transformers-format Safetensors and configuration files and is the authoritative source for the model artifacts.

The model uses the Qwen Community License 1.0. It permits broad use, modification and deployment, but it is not Apache-2.0 and includes conditions for very large commercial products plus separate-license requirements for certain commercial Model-as-a-Service and AI work-assistant uses. Review the actual license before commercial deployment.

03

Supported runtimes

Qwen's official model card lists Hugging Face Transformers, vLLM, SGLang and TokenSpeed as compatible with the original weights. For production or high-throughput serving, Qwen specifically recommends current versions of SGLang, KTransformers or vLLM.

For quantized local use, community GGUF builds target llama.cpp and compatible apps such as Ollama and LM Studio. Support can move quickly after a new architecture lands, so use a current runtime build and check the conversion's model card before downloading more than 100 GB of files.

04

Is there an official Qwen3.8-Flash-Next GGUF?

Qwen's official repository currently provides the original Transformers weights and an official FP8 variant. It does not publish an official Qwen GGUF repository.

GGUF repositories such as unsloth/Qwen3.8-Flash-Next-GGUF are community-produced conversions derived from the Qwen model. They can be useful, but they should not be described as official Qwen releases or assumed to reproduce the original weights exactly.

05

GGUF quantization options and file sizes

The current Unsloth community repository lists these complete artifact sizes. They are download sizes, not guaranteed peak RAM or VRAM requirements:

• 1-bit UD-IQ1 variants: about 72.5–74.5 GB
• 2-bit UD-Q2_K_XL: about 78.9 GB
• 3-bit variants: about 82–90 GB
• 4-bit UD-IQ4_XS: about 93.7 GB; UD-Q4_K_XL: about 111 GB
• 5-bit UD-Q5_K_XL: about 158 GB
• 6-bit UD-Q6_K_XL: about 169 GB
• 8-bit Q8_0: about 188 GB
• BF16 conversion: about 354 GB

06

Realistic RAM and VRAM tiers

• 64 GB total memory: no complete GGUF listed above fits; this is not a realistic full-model target.
• 96 GB total or unified memory: only the smallest 1–3-bit files fit on paper, with limited room for runtime overhead and context.
• 128 GB unified or system memory: the first practical single-machine tier for a roughly 94 GB 4-bit build; the 111 GB Q4_K_XL remains tight once context and overhead are included.
• 192 GB total memory: allows higher-quality 5-, 6- or 8-bit files with more working room, although speed still depends heavily on bandwidth and offload.
• Multi-GPU servers: size the aggregate VRAM above the chosen weights and leave headroom for KV cache and runtime allocations. Qwen does not publish one universal minimum GPU count for every engine and context length.

A two-GPU system with 48 GB cards has 96 GB aggregate VRAM, but that does not automatically make a 94 GB artifact practical. Sharding, non-weight allocations and context still consume memory. Two 80 GB GPUs or four 48 GB GPUs offer more realistic headroom for a 4-bit deployment.

07

CPU, GPU and unified-memory trade-offs

CPU-only inference is possible with a GGUF runtime when system RAM is large enough, but generation speed is likely to be constrained by memory bandwidth. The MoE design reduces compute per token, yet the runtime still has to move data from a very large weight set.

GPU offload improves throughput when the selected layers and weights fit in VRAM. Unified-memory systems avoid a hard CPU/GPU split, but they still need enough total memory and bandwidth. Hybrid CPU/GPU offload can make a model run without making it fast.

08

Why context length changes memory use

Qwen documents a native context length of 262,144 tokens and extension to 1 million with YaRN. That is a model capability, not a sensible default for a memory-limited local machine.

KV-cache and runtime allocations grow as context increases. Start with 8K or 16K, confirm stable generation, then raise context while watching the runtime's reported memory use. Do not configure 262K or 1M merely because the architecture supports it.

09

Run the official weights with vLLM or SGLang

On sufficiently large GPU infrastructure, Qwen's simplest official vLLM path is:

The official SGLang path is pip install sglang followed by python3 -m sglang.launch_server --model-path Qwen/Qwen3.8-Flash-Next --host 0.0.0.0 --port 30000. Both expose an OpenAI-compatible local endpoint in the documented examples.

Use current framework recipes for tensor parallelism, quantized formats and long-context overrides. The one-line commands demonstrate compatibility; they do not guarantee the unquantized model will fit on a single GPU.

10

Run a community GGUF with llama.cpp

After installing a recent llama.cpp build, the Unsloth community card gives this direct server command for its Q4_K_XL conversion:

The equivalent Ollama path published on the community model card is ollama run hf.co/unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL. Start with a modest context, verify the downloaded variant and inspect startup logs for offload and memory allocation.

11

Benchmarks: capability is not local speed

Qwen reports strong capability scores for the original model, including 62.5 on SWE-bench Pro and 91.9 on LiveCodeBench v6. These are vendor-reported evaluation results and do not tell you how many tokens per second a particular GGUF will generate on your machine.

No official, reproducible consumer-hardware throughput table was published in the sources reviewed for this guide. Rather than invent a number, benchmark the exact runtime, quant, context and offload configuration you plan to use.

1. Record prompt-processing and generation tokens per second separately.
2. Keep the prompt, context length, quant and sampling settings fixed.
3. Report CPU, RAM channels, GPU model, VRAM and offload layers.
4. Run multiple passes after model loading and compare median results.
5. Test output quality as well as speed before choosing an extreme low-bit quant.

12

The practical recommendation

Use the official Qwen weights with vLLM or SGLang when you have server-class GPU capacity and need the authoritative model artifacts. Use a clearly labelled community GGUF when local experimentation and CPU, unified-memory or hybrid offload matter more than matching the original format.

For a single machine, 128 GB is the realistic starting tier for a useful 4-bit attempt, and 192 GB provides healthier headroom. Anyone with less memory should consider a smaller Qwen model or a managed endpoint rather than relying on an extreme quant that barely fits.