GLM-5.2

Deploy GLM-5.2 with SGLang — Z.ai’s DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X.

Deployment

Install SGLang

For all methods and hardware platforms, see the official SGLang installation guide. The two paths below match the Python / Docker toggle in the command panel.

Python (pip / uv)

Command

pip install --upgrade pip
pip install uv
uv pip install sglang

Then run the Python output of the command panel below in that environment.

Docker

Command

docker pull lmsysorg/sglang:latest

For how to launch the image, see Install → Method 3: Using Docker. Substitute the inner sglang serve ... with what the command generator below produces.

Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:

• Low-Latency — fastest reply for a single user. Pick for chat.
• Balanced — good speed with several users at once. Use for typical multi-user serving.
• High-Throughput — most tokens per second across many users. Best for batch jobs.

All recipes here run the DSA indexer top-k on the default --dsa-topk-backend sgl-kernel. Other top-k backend choices have not been fully validated on GLM-5.2.

Speed numbers are measured with --random-range-ratio 1.0, --flush-cache, on main @ 09ca4fc. Spec cells pin the EAGLE acceptance length via the serve env SGLANG_SIMULATE_ACC_LEN (low-latency 5-1-6 = 3.5, balanced 2-1-3 = 2); high-throughput has no spec.

Playground

The Playground is where you experiment with SGLang features beyond the verified matrix. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

1. Model Introduction

GLM-5.2 is Z.ai’s flagship Mixture-of-Experts model built on DeepSeek Sparse Attention (DSA): a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It ships in two precisions — FP8 (zai-org/GLM-5.2-FP8) and full BF16 (zai-org/GLM-5.2) — both with 78 transformer layers, 256 routed experts (8 active per token), a 1M-token context window, and a single MTP (Multi-Token Prediction) layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, NVIDIA also publishes an NVFP4 build (nvidia/GLM-5.2-NVFP4) that quantizes only the MoE experts’ linear weights and activations to 4-bit (the shared expert stays unquantized), holding accuracy within ~1 point of the FP8 baseline on GPQA Diamond, SciCode, and IFBench.

Model	Architecture	Context
GLM-5.2-FP8	MoE · DSA · 256 experts (top-8) · MTP · FP8	1,048,576
GLM-5.2	MoE · DSA · 256 experts (top-8) · MTP · BF16	1,048,576
GLM-5.2-NVFP4	MoE · DSA · 256 experts (top-8) · MTP · NVFP4	1,048,576

Recommended generation: temperature=1.0, top_p=0.95 (the checkpoint’s generation_config.json defaults; informational — do not hardcode in client code).

Resources: GLM-5.2-FP8 · GLM-5.2 (BF16) · GLM-5.2-NVFP4.

2. Configuration Tips

• DeepSeek Sparse Attention (DSA). GLM-5.2 uses the glm_moe_dsa architecture; SGLang auto-selects the DSA attention backends (flashmla_sparse prefill, fa3 decode, sgl-kernel indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — fp8_e4m3 on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and bf16 on Hopper (H200) — so no --kv-cache-dtype flag is required.

• MTP / speculative decoding. The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 for low-latency; 1-1-2 for balanced). The config’s index_share_for_mtp_iteration reuses the DSA indexer’s topk across draft steps (effective only at --speculative-eagle-topk 1). Tune the draft length to the accept length. GLM-5.2’s MTP head is strong — accept length runs high (4+ in many workloads, near-saturating at 5–6 in low-latency runs). Watch the server’s reported accept length and adjust --speculative-num-steps / --speculative-num-draft-tokens accordingly: while accept length stays close to the draft-token count there is headroom to push them higher (more accepted tokens per step); if it falls well below, lower them — every rejected draft token is wasted verification compute.

• Memory. The FP8 weights are large (MoE total, not active params). Start around --mem-fraction-static 0.8 on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4).

• DP-Attention + DeepEP for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP.

• BF16 weights need more GPUs. The full-precision build (zai-org/GLM-5.2, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on 8×B300 (TP8, ~2.1 TB HBM) — verified; on the smaller GPUs it needs a multi-node layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8), and those multi-node BF16 recipes are still proposed/inferred (verified: false). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. On B300, BF16 low-latency matches FP8 (the sm103 FP8 path is not yet optimized), but FP8 wins at the balanced/high-throughput points.

• PD Disaggregation (prefill/decode). GLM-5.2 is a DSA model and runs under prefill/decode disaggregation — toggle the PD Disagg card in the Playground above (pick a Prefill/Decode role + transfer backend, then front the roles with sglang_router.launch_router --pd-disaggregation). The Mooncake backend auto-detects the InfiniBand HCA, so no device flag is needed by default; only add --disaggregation-ib-device mlx5_0 (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (--privileged --ulimit memlock=-1, or --device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK) — without IB exposure Mooncake silently falls back to TCP.

• Chunked-prefill size is regime-dependent. At long input (8K+) the default --chunked-prefill-size 2048 is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to --chunked-prefill-size 32768 on the balanced recipe gave roughly +34–78% output throughput and −39–59% TTFT on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is neutral for high-throughput (decode-bound there) — keep the default. --max-running-requests tracks KV capacity, not a tuning free-for-all: ~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near --max-running-requests 80 and let high-throughput run wider.

• AMD GPUs (MI300X / MI325X / MI355X). FP8 (zai-org/GLM-5.2-FP8) runs single-node at tp=8 on all three. BF16 (zai-org/GLM-5.2, ~1.51 TB) only fits single-node on MI325X (2 TB HBM) and MI355X (2.3 TB); MI300X (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there (or a multi-node BF16 layout once validated). Use the DSA tilelang backend (--dsa-prefill-backend tilelang --dsa-decode-backend tilelang) and add --chunked-prefill-size 131072 plus --watchdog-timeout 1200 (20 min for weight loading). FP8 uses about half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). GLM-5.2 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the DeepSeek-V3.2 cookbook.

gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (v0.5.13.post1-rocm720-mi35x-20260618). Earlier SGLang ROCm images miscompiled AMD aiter’s gemm_a8w8_blockscale_bpreshuffle GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — GSM8K ≈ 0.96 (0% invalid) and 15/15 needle-in-haystack retrieval to ~118K tokens. MI300X / MI325X (gfx942) were never affected. If you must run an older image, treat gfx950 FP8 output as unverified. Background: sgl-project/sglang#28685 (analysis) and the upstream CK fix ROCm/rocm-libraries#8639 (scalar FMA + accumulator anchor; restores correctness and determinism at -O3).

• MTP / EAGLE speculative decoding is disabled for AMD in the Deploy panel. The block-FP8 accuracy bug that previously degraded it is now fixed (see note above), but MTP on gfx950 still depends on the spec-decode draft kernel, which is not yet validated on this hardware (and at --speculative-num-steps > 3 hits a separate build issue). Until MTP is validated on gfx950, omit the --speculative-* flags and serve without MTP.

3. Advanced Usage

3.1 Reasoning

GLM-5.2 is a hybrid-reasoning model. Enable the glm45 reasoning parser (toggle Reasoning Parser in the Parsers card of the Playground above) to separate thinking from the final answer — thinking lands in message.reasoning_content, the answer in message.content. Thinking is on by default; turn it off with chat_template_kwargs: {"enable_thinking": False} (the template variable is enable_thinking, not thinking).

Reasoning effort. Pass chat_template_kwargs: {"reasoning_effort": ...} to inject a Reasoning Effort: <level> system line (only while thinking is on). The template wires only two effective levels — Max and High — and if you don’t pass reasoning_effort at all you get Max, the highest. "high" is the only value that lowers effort; every other value (including "low" and "medium") falls through to Max:

reasoning_effort	Injected system line	Effect
(not passed / unset)	Reasoning Effort: Max	default — highest reasoning
"high"	Reasoning Effort: High	dials reasoning down
"low", "medium", any other value	Reasoning Effort: Max	falls through to Max (not a distinct level)

Reasoning Example (Python)

Example

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="zai-org/GLM-5.2-FP8",
    messages=[{"role": "user", "content": "What is 15% of 240?"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "high"}},
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Answer:", msg.content)

Example Output

Output

Reasoning: 1.  **Identify the core question:** The user wants to find 15% of 240.
2.  **Convert the percentage to a decimal:** 15% = 0.15
3.  **Multiply by the total:** 0.15 * 240 = 36
    (Quick mental math: 10% of 240 = 24; 5% = 12; 24 + 12 = 36.)

Answer: 15% of 240 is **36**.

Here is how you can calculate it:
0.15 × 240 = 36

3.2 Tool Calling

Enable the glm47 tool-call parser (toggle Tool Call Parser in the Parsers card of the Playground above) to surface structured tool calls via message.tool_calls. GLM-5.2 emits the newer <tool_call>…<arg_key>…<arg_value>… format, so it needs the glm47 parser — the older glm45 parser does not parse it (the call would be left as raw text in content). On thinking mode the turn also fills reasoning_content, so print both fields.

Tool Calling Example (Python)

Example

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
resp = client.chat.completions.create(
    model="zai-org/GLM-5.2-FP8",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Tool calls:", msg.tool_calls)

Example Output

Output

Reasoning: The user wants to know the weather in Paris. I'll call the get_weather function with "Paris" as the city.

Tool calls: [
  {
    "id": "call_13fcd52146934b7781d06d4a",
    "type": "function",
    "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}
  }
]

3.3 HiCache (Hierarchical KV Caching)

For long-context, prefix-heavy workloads, enable hierarchical KV caching to spill cold KV blocks to host memory (toggle the Hierarchical KV Cache card in the Playground above). Useful given GLM-5.2’s 1M-token window; pair --hicache-ratio with a write policy that matches your reuse pattern.

3.4 Claude Code Integration

GLM-5.2’s strong reasoning + tool-calling makes it a good backend for Claude Code, Anthropic’s agentic CLI. SGLang exposes the Anthropic-compatible /v1/messages endpoint on every server, so Claude Code can talk to a GLM-5.2 server with only environment variables — no code change. Launch the server with --reasoning-parser glm45 --tool-call-parser glm47 (any recipe from the Deployment panel above works), then:

Command

export ANTHROPIC_BASE_URL="http://127.0.0.1:30000"
export ANTHROPIC_AUTH_TOKEN="dummy"
export API_TIMEOUT_MS="3000000"
export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000"
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
export CLAUDE_CODE_ATTRIBUTION_HEADER=0
export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]"
export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]"
export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]"
claude

Two of these matter specifically for GLM-5.2:

• CLAUDE_CODE_ATTRIBUTION_HEADER=0 — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.2’s chat template renders tools before system, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse.
• glm-5.2[1m] as the model name — the [1m] suffix is the client-side hint that enables Claude Code’s 1M-context beta, matching GLM-5.2’s 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the model field, so any name is accepted server-side.

For the full setup (streaming, tool-use, count_tokens, persisting env in ~/.claude/settings.json, troubleshooting), see Anthropic-Compatible API.

3.5 Context Parallelism

Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.2, please append the following arguments:

--attn-cp-size 8 \
--enable-prefill-cp \
--cp-strategy interleave \

which splits the sequence equally across --attn-cp-size ranks during attention forward. The trade off for prefill CP is that it will introduce extra all-gather operation before indexer-topk and attention kernels, so it will increase latency for decode (in unified deployment) or short prefill.

When deploying with PD Disaggregation, the prefill node can choose to enable LayerSplit technique with

--enable-dsa-cache-layer-split \
--attn-cp-size 8 \
--cp-strategy interleave \

With LayerSplit, the kv cache on each rank can be sharded over the CP attention group, and prefetched when necessary. This can reduce kv cache memory by up to 75%, thus increasing the throughput on prefill side.