# PaddleOCR-VL Source: https://docs.sglang.io/cookbook/autoregressive/Baidu/PaddleOCR-VL Deploy PaddleOCR-VL 1.6 / 1.5 / 0.9B with SGLang — Baidu's 0.9B NaViT + ERNIE-4.5 document-parsing VLM for OCR, tables, formulas and charts in 109 languages, on a single H100, H200 or B200. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick a release and your hardware to generate the launch command. The model is 0.9B and single-GPU, so there is one serving recipe per platform; the axis that actually moves cost is **Page Resolution**, which caps how many image tokens one page is worth. ## Playground Use the Playground to layer tensor parallelism on top of the selected deployment cell. At this size TP is a latency knob, not a capacity one — the weights fit on one GPU. ## 1. Model Introduction **PaddleOCR-VL** is Baidu's compact document-parsing vision-language model: a NaViT-style dynamic-resolution SigLIP vision encoder feeding an **ERNIE-4.5-0.3B** language backbone, 0.9B parameters in total, released under **Apache 2.0**. It recognizes text, tables, formulas, charts and seals across **109 languages**, and is small enough that a single GPU serves it comfortably. SGLang serves **this model**, which is the recognition stage of PaddleOCR's document-parsing pipeline — not the pipeline itself. Layout and region detection, cropping, reading order and the assembly of markdown or JSON stay in PaddleOCR, which calls a model endpoint once per region. That split is why the model is prompt-conditioned rather than instruction-following: the caller decides whether a crop is a table, a formula or a chart (§3.1), and SGLang is the backend that answers. All three releases share an identical `config.json` (same `PaddleOCRVLForConditionalGeneration` architecture, same tower and backbone dimensions), so one SGLang recipe serves every variant and only the model path changes.
Variant Total params Use
PaddleOCR-VL-1.6 0.9B Latest. Best tables, Chinese characters and seals; drop-in for 1.5.
PaddleOCR-VL-1.5 0.9B Previous generation; pin it if you have calibrated against its output.
PaddleOCR-VL 0.9B The original 0.9B release.
**Recommended generation:** greedy decoding (`temperature=0`) with a per-page `max_tokens` budget — the model card uses 512 for a single region and the reference server allows more for a full page. These are informational; do not hardcode them in library code. **Resources:** [Hugging Face](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6) · [PaddleOCR on GitHub](https://github.com/PaddlePaddle/PaddleOCR) ## 2. Configuration Tips * **Page resolution is the main cost knob.** The vision tower and the prefill both scale with the patch count of a page. `max_pixels` is expressed in 28x28 units (patch size 14 with a 2x2 merge), so `max_pixels / 784` is the image-token budget per page. The checkpoint's own default is 1280 tokens; the **Page Resolution** selector in the Deploy panel emits the corresponding `--mm-process-config` value. Lower it for clean born-digital PDFs, raise it for dense scans and small print. * **Prompt selects the task.** PaddleOCR-VL is prompt-conditioned rather than instruction-following — use the exact task strings in §3.1. A free-form question will not behave like a chat model. * **Leave `--trust-remote-code` off.** The checkpoints ship their own `configuration_paddleocr_vl.py` / `processing_paddleocr_vl.py`, but `transformers` 5.12 supports `paddleocr_vl` natively — and the bundled remote image processor is the slower of the two implementations (measured 87.4 ms vs 39.1 ms per 1080p page). Passing the flag pins SGLang to the remote copy. Serving without it produced byte-identical OCR output on every page we checked and about 5% more requests per second at 32-way concurrency. * **Preprocessing is parallelized for you.** A full-resolution page costs tens of milliseconds of CPU to resize, normalize and patchify, which caps throughput long before the GPU saturates, so this model runs the image processor across several workers by default. `--mm-processor-worker-num` overrides the count; raising it past the default did not help in our measurements. * **Keep the radix cache on for repeated pages.** Unlike whole-document batch OCR over unique scans, a workload that re-asks about the same page (different task prompts on one image) reuses the image prefix. Add `--disable-radix-cache` only if every request carries a different page. * **The saturated-throughput flags earn their place.** A page is \~2700 tokens, so the default 8192-token prefill budget packs only three of them into a forward. Raising it to 16384 and letting decode ride along in the same batch (`--enable-mixed-chunk`, `--num-continuous-decode-steps 2`) measured +11% requests per second at 32-way concurrency and cut queued TTFT by 23%, with single-stream latency unchanged. Measured on an H200; on a smaller card lower `--chunked-prefill-size` until it fits. * **Prefill CUDA graph is on for this model.** SGLang normally switches the breakable prefill graph off for every multimodal architecture; PaddleOCR-VL is allowlisted back in, which is worth 16.1 ms → 11.5 ms of single-stream TTFT on text-only prompts. Image-carrying batches are rejected at graph replay and run eager, so this helps mixed and text traffic, not pure page parsing. No flag needed. * **Tensor parallelism is optional.** The weights are under 2 GB in BF16; TP>1 only shortens the vision-encoder and prefill critical path, at the cost of a collective per layer. Measure before adopting it. * **Context length.** The backbone advertises 131072 positions, but a parsed page rarely needs more than a few thousand tokens. The recipe pins `--context-length 16384` so the KV pool stays small and concurrency stays high; raise it only if you batch many pages into one request. ### Measured on one H200 One 1080p page (\~2700 image tokens) in, 128 tokens out, prefix cache disabled, median TTFT:
Configuration TTFT, 1 stream req/s at 32 concurrent
With --trust-remote-code (remote image processor) 219 ms 10.9
Recipe above (native image processor) 114 ms 11.3
Throughput at saturation is bound by the vision tower, which runs full attention over every patch of the page — so the **Page Resolution** selector is the lever that moves it, not tensor parallelism. ## 3. Advanced Usage ### 3.1 Task prompts PaddleOCR-VL exposes its capabilities through a small set of fixed prompts. Send the prompt as the text part and the page as the image part of the same user turn.
Prompt Task
OCR: Plain text recognition.
Table Recognition: Table structure and cell contents.
Formula Recognition: Mathematical expressions.
Chart Recognition: Chart contents.
Spotting: Text with locations. Benefits from the high-detail resolution setting.
Seal Recognition: Seals and stamps (1.6).
Structured tasks answer in the model's own markup, not HTML: `Table Recognition:` returns OTSL-style cell tokens (`` per cell, `` per row), so a caller that wants HTML or markdown converts them itself. ```text Output theme={null} MethodsRPFFPSSegLink [26]70.086.077.08.9PixelLink [4]73.283.077.8-TextSnake [18]73.983.278.31.1 ... (one per cell, one per row, to the end of the table) ``` ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="PaddlePaddle/PaddleOCR-VL-1.6", messages=[ { "role": "user", "content": [ {"type": "text", "text": "OCR:"}, { "type": "image_url", "image_url": { "url": "https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/release/2.9/doc/imgs_en/img_12.jpg" }, }, ], } ], max_tokens=2048, ) print(response.choices[0].message.content) ``` ```text Output theme={null} ACKNOWLEDGEMENTS We would like to thank all the designers and contributors who have been involved in the production of this book; their contributions have been indispensable to its creation. We would also like to express our gratitude to all the producers for their invaluable opinions and assistance throughout this project. And to the many others whose names are not credited but have made specific input in this book, we thank you for your continuous support. ``` ### 3.2 Parsing a multi-page document The model parses one page per request. Render each page to an image, then fan the pages out concurrently — SGLang batches the vision encoders of in-flight requests into a single forward, so concurrency is what keeps the GPU busy on a model this small. ```python Example theme={null} import base64 from concurrent.futures import ThreadPoolExecutor import pymupdf from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") def render(page, dpi=200): pixmap = page.get_pixmap(dpi=dpi) return base64.b64encode(pixmap.tobytes("png")).decode("ascii") def parse(page_png_b64): response = client.chat.completions.create( model="PaddlePaddle/PaddleOCR-VL-1.6", messages=[ { "role": "user", "content": [ {"type": "text", "text": "OCR:"}, { "type": "image_url", "image_url": { "url": f"data:image/png;base64,{page_png_b64}" }, }, ], } ], max_tokens=2048, ) return response.choices[0].message.content document = pymupdf.open("your_document.pdf") pages = [render(page) for page in document] with ThreadPoolExecutor(max_workers=16) as pool: for index, text in enumerate(pool.map(parse, pages)): print(f"--- page {index + 1} ---") print(text) ``` ```text Output theme={null} --- page 1 --- (a) Total-Text (b) Total-Text (c) CTW1500 (d) CTW1500 Figure 8. Visual experimental results. The blue contours are boundary proposals, and the green contours are final detection boundaries. Table 6. Experimental results on CTW-1500. Methods Ext R P F FPS TextSnake [18] Syn 85.3 67.9 75.6 - ... (page continues) --- page 2 --- ... (one block per page, in page order) ``` # Unlimited-OCR Source: https://docs.sglang.io/cookbook/autoregressive/Baidu/Unlimited-OCR Deploy Baidu Unlimited-OCR with SGLang for long document OCR using prefill-aware sliding-window attention. ## Deployment Unlimited-OCR support is in [SGLang PR #29186](https://github.com/sgl-project/sglang/pull/29186). Until that PR is included in a tagged SGLang release, install from a build that contains the PR. ```bash Command theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate git clone https://github.com/sgl-project/sglang.git cd sglang git fetch origin pull/29186/head && git checkout FETCH_HEAD uv pip install --prerelease=allow -e python ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware to generate the launch command. The recipe uses FlashAttention-3 with `--page-size 1`, which is required by the current prefill-aware sliding-window attention path. It also disables radix cache by default, which is the better fit for batch OCR workloads where each request usually contains a different image. ## Playground Use the Playground to adjust tensor parallelism on top of the selected deployment cell. ## 1. Model Introduction [Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) is Baidu's multimodal OCR model for document parsing. It uses a sliding-window language backbone, but SGLang serves it with a prefill-aware sliding-window path so image and prompt tokens remain visible during long decode. The SGLang integration loads the standalone Unlimited-OCR architecture with SAM and CLIP vision encoders plus a DeepSeek-style language backbone. It supports OpenAI-compatible image requests and model-specific image processing options through `images_config`. **Resources:** [Hugging Face](https://huggingface.co/baidu/Unlimited-OCR) · [SGLang PR #29186](https://github.com/sgl-project/sglang/pull/29186) ## 2. Configuration Tips * **Attention backend**: use `--attention-backend fa3 --page-size 1`. The prefill-aware SWA page table is built with token-level locations, so page size 1 is required. * **Radix cache**: keep `--disable-radix-cache` for batch OCR over different documents. If your workload repeatedly asks about the same image and prompt, remove this flag to allow prefix reuse through `PureSWARadixCache`. * **Long OCR generations**: keep the default prefill-aware SWA path enabled. It retains prompt and image KV while still applying a sliding window to generated text. * **Custom logit processor**: keep `--enable-custom-logit-processor` in the launch command. * **Image modes**: pass `images_config.image_mode` per request. Supported modes are `tiny`, `small`, `base`, `large`, and `gundam`. Multiple images are supported only for `tiny`, `small`, and `base`. * **Default image mode**: when `images_config.image_mode` is omitted, SGLang uses `gundam`. ## 3. Advanced Usage ### 3.1 OCR request ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="baidu/Unlimited-OCR", messages=[ { "role": "user", "content": [ {"type": "text", "text": "document parsing."}, { "type": "image_url", "image_url": { "url": "https://example.com/your_document.png" }, }, ], } ], max_tokens=2048, temperature=0, extra_body={"images_config": {"image_mode": "gundam"}}, ) print(response.choices[0].message.content) ``` ### 3.2 Choosing an image mode Use lower modes to reduce prefill cost for simple images, and use `gundam` for high-detail document parsing.
Mode Use Multiple images
tiny Lowest prefill cost. Yes
small Lightweight OCR requests. Yes
base Balanced quality and cost. Yes
large Higher resolution single-image OCR. No
gundam Default high-detail document parsing mode. No
# Clef Source: https://docs.sglang.io/cookbook/autoregressive/Cloudflare/clef Deploy Cloudflare's Clef 27B and Clef Flash 9B decision models with SGLang. Select either variant for BF16 text and image decisions on one NVIDIA B200, B300, or H200. ## Deployment Clef support merged after v0.5.21 in [#42721](https://github.com/sgl-project/sglang/pull/42721). Use the pinned nightly wheel or Docker image below. See the [installation guide](/docs/get-started/install) for other methods. This command selects the wheel from [nightly build 37ae292e6f](https://github.com/sgl-project/sglang/actions/runs/37897861964), which includes Clef support. It targets Python 3.10 on Linux x86\_64 with glibc 2.39 or newer. ```bash Command theme={null} pip install --upgrade pip pip install uv uv venv --python 3.10 source .venv/bin/activate uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ "sglang==0.5.22.dev20261009+g37ae292e6f" ``` Run the command from the wizard’s Python tab in this environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev-clef ``` The [Clef image build](https://github.com/sgl-project/sglang/actions/runs/37893990016) produces this image from the #42721 merge commit. Run the command from the wizard’s Docker tab. Both checkpoints are public; no Hugging Face token is required. Choose your GPU, then Clef or Clef Flash. The wizard updates the model in the launch command and sample request. Both models run on one GPU in BF16 with TP=1 using the default settings; the commands need only the model path, host, and port. ## 1. Model introduction Cloudflare post-trained Clef from Qwen3.8-27B and Clef Flash from Qwen3.5-9B, then released both under the Apache-2.0 license. Each model keeps its backbone and vision encoder and adds a joint schema head. The head reads the backbone’s final hidden states, routes state evidence to each option, and lets questions attend to one another. It returns one logit per allowed option in a single prefill, with no text generation.
Variant Checkpoint Backbone
Clef (27B) Cloudflare/clef Qwen3.8-27B
Clef Flash (9B) Cloudflare/clef-flash Qwen3.5-9B
Pass a text string, JSON object, or JSON array as the state, with optional images. `/v1/systemone` returns one answer per question: a `choice` over allowed options, a `noul` probability of true, or a `score` over ordered levels. ## 2. Configuration tips * **Route.** Send requests to `/v1/systemone`, or set your System One client’s base URL to the server. This includes TypeSafe SDK clients. See the [API reference](/docs/supported-models/decision_models#system-one-compatible-api) for request and response fields. Clef refuses `/v1/decisions`, chat, and generation requests. * **One GPU, one prefill per request.** Keep TP=1 and PP=1. All questions share one prompt, so the model reads the state once. SGLang detects `joint_head_config.json`, loads `joint_head.safetensors`, and enables embedding mode. It disables radix caching, chunked prefill, and CUDA graphs. No reasoning parser, tool-call parser, or chat-template override is needed. * **Answers.** `probabilities` are the softmax of the head’s logits for each question. `confidence` uses the System One adapter formulas. It can differ from the checkpoint’s reference `systemone()`, which reports the top option’s probability as confidence. Answers omit `x_label_mass` because the head scores only the allowed options. * **Long states.** Prompts, including images, are capped at 16,384 tokens, the reference `encode_record` function’s default `max_length`. The effective limit can be lower, depending on the server context, KV token pool, and `--max-prefill-tokens`. The server truncates the state to leave room for the questions and images. If those do not fit even without the state, it rejects the request. * **Images.** Send each image in `images` as a data URL, such as `data:image/jpeg;base64,...`, or an HTTP(S) URL. Keep the data-URL prefix for JPEGs: bare base64 starts with `/`, which the loader treats as a file path, causing the request to fail. Images precede the state. ## 3. Advanced usage ### 3.1 Text decisions Set `MODEL` to the checkpoint you selected in the wizard. The request works with either variant. ```python Example theme={null} import requests MODEL = "Cloudflare/clef-flash" # Or "Cloudflare/clef". response = requests.post( "http://localhost:30000/v1/systemone", json={ "model": MODEL, "state": "My Stripe integration keeps failing. I am losing sales. Please help ASAP.", "questions": { "routing": { "type": "choice", "instructions": "Which team should handle this request?", "criteria": { "billing": "Charges and refunds", "technical_support": "Integration errors", "sales": "Questions about buying a product", }, }, "urgency": {"type": "noul", "instructions": "Does this message express urgency?"}, "frustration": { "type": "score", "instructions": "How frustrated is the customer?", "criteria": ["Calm", "Frustrated but civil", "Very angry"], }, }, }, timeout=60, ) response.raise_for_status() answers = response.json()["answers"] print("Routing:", answers["routing"]["choice"]) print("Urgency:", answers["urgency"]["noul"]) print("Frustration:", answers["frustration"]["score"]) print("Usage:", response.json()["usage"]) ``` Captured with Clef Flash on one H200. Probabilities can vary with the model variant, hardware, and numerical backend. ```text Output theme={null} Routing: technical_support Urgency: 0.9492946352021694 Frustration: 1.7297646970642042 Usage: {'input_tokens': 334, 'output_tokens': 0} ``` ## 4. GSM8K accuracy [Decision Index 0.2.1](https://github.com/apolinario/decision-index/blob/9eb2dbe2a358004c8782c66e40a83ac07b953fec/docs/suite.md) evaluates all 1,319 GSM8K test problems with both four-choice and ten-choice questions, for 2,638 requests per configuration. The table reports accuracy across both formats through `/v1/systemone`. | Model | [Reported](https://huggingface.co/Cloudflare/clef#results) | H200 | B200 | B300 | | - | -: | -: | -: | -: | | Clef | 80.8% | 80.71% | 80.59% | 80.59% | | Clef Flash | 67.3% | 67.32% | 67.36% | 67.36% | All runs use BF16 and TP=1 on one GPU. # Ornith-1.0 Source: https://docs.sglang.io/cookbook/autoregressive/DeepReinforce/Ornith-1.0 Deploy DeepReinforce Ornith-1.0 with SGLang - a self-improving agentic-coding model family with 397B, 35B, and 9B checkpoints plus FP8 and GGUF variants. ## Deployment Ornith-1.0 model cards recommend SGLang `>=0.5.9`. The Deploy panel below emits the base serve command; the reasoning and tool-call parsers from the model-card quickstarts (`--reasoning-parser qwen3` for `...` traces, `--tool-call-parser qwen3_coder` for Qwen-style XML tool calls) are added on top via the [Playground](#playground). ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow "sglang>=0.5.9" ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install -> Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick an Ornith checkpoint to generate the launch command. The non-FP8 397B recipe requires an H200 single node in this matrix. The 397B-FP8 recipe is available on H100 and H200 with TP=8; H100 also supports the 35B and 9B checkpoints. The 35B recipes use tensor parallelism 2 in this matrix. The 9B checkpoint is dense and serves on a single GPU by default; the command panel makes that default explicit with `--tp 1`. ## Playground The Playground layers SGLang features on top of whichever cell the Deploy panel is showing — only your overrides change, and any change flips the badge to **Not Verified** until the new configuration is run end-to-end. For Ornith-1.0 the knobs are the reasoning and tool-call parsers: * **Reasoning Parser** appends `--reasoning-parser qwen3`. Ornith emits `...` traces; with this on, SGLang surfaces them as `message.reasoning_content` instead of leaving the tags inline in `content`. * **Tool Call Parser** appends `--tool-call-parser qwen3_coder`, so Qwen-style XML tool calls are returned as OpenAI-compatible `tool_calls`. ## 1. Model Introduction [Ornith-1.0](https://huggingface.co/collections/deepreinforce-ai/ornith-10) is DeepReinforce's self-improving open-source model family for agentic coding. The model cards describe the family as post-trained on top of Gemma 4 and Qwen 3.5, and the collection currently includes 397B, 35B, and 9B repos plus FP8 and GGUF variants. The model cards report results on Terminal-Bench 2.1, SWE-Bench, NL2Repo, ClawEval, and SWE Atlas benchmarks. **Key Features:** * **Agentic coding specialization**: the model cards describe Ornith-1.0 as specialized for agentic coding and report coding-agent benchmark results. * **Self-improving training**: the model cards state that Ornith-1.0 uses reinforcement learning to optimize both solution rollouts and the scaffold that drives those rollouts. * **Reasoning model behavior**: assistant responses begin with a `...` reasoning block before the final answer; enable the `--reasoning-parser qwen3` toggle in the [Playground](#playground) to split it into `reasoning_content`. * **Tool calling**: emits Qwen-style XML tool calls; enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground). * **Long context**: model-card recipes use `--context-length 262144`. * **MIT license**: the Hugging Face repos are released under MIT. **Available Models:**
Model Format Deploy Panel Notes
deepreinforce-ai/Ornith-1.0-397B BF16 H200 only Flagship 397B MoE checkpoint; model-card baseline uses TP=8 on an H200 single node.
deepreinforce-ai/Ornith-1.0-397B-FP8 FP8 H100 / H200 FP8 repo in the collection; the deploy command uses this repo id with TP=8.
deepreinforce-ai/Ornith-1.0-35B BF16 H100 / H200 35B MoE checkpoint; the deploy command uses TP=2.
deepreinforce-ai/Ornith-1.0-35B-FP8 FP8 H100 / H200 FP8 repo in the collection; the deploy command uses this repo id with TP=2.
deepreinforce-ai/Ornith-1.0-9B BF16 H100 / H200 Dense 9B checkpoint; the model card describes it as designed for efficient single-GPU deployment.
deepreinforce-ai/Ornith-1.0-35B-GGUF GGUF No Listed for completeness; GGUF targets llama.cpp-style local inference, not the SGLang server recipe here.
deepreinforce-ai/Ornith-1.0-9B-GGUF GGUF No Listed for completeness; the model card shows llama.cpp and Ollama examples for the GGUF build.
**License:** [MIT](https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B/blob/main/LICENSE) **Resources:** [Hugging Face collection](https://huggingface.co/collections/deepreinforce-ai/ornith-10) · [Ornith blog](https://deep-reinforce.com/ornith_1_0.html) ## 2. Configuration Tips * **Reasoning parser**: Ornith responses include `...`. Enable the `--reasoning-parser qwen3` toggle in the [Playground](#playground) so OpenAI-compatible responses expose the reasoning trace as `message.reasoning_content`. * **Tool-call parser**: enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground) so `` blocks are returned as OpenAI-compatible tool calls. * **Context length**: the model-card SGLang recipes use `--context-length 262144`. Lower it if you need more memory headroom. * **Tensor parallelism**: the 397B model-card recipes use `--tp 8`; in this single-node matrix, non-FP8 397B is H200-only, while 397B-FP8 is available on both H100 and H200. The 35B deploy commands use `--tp 2`. The 9B model-card recipe is single-GPU by default; the command panel makes that explicit with `--tp 1`. Adjust TP to match your node and memory budget. * **Sampling**: model cards recommend `temperature=0.6`, `top_p=0.95`, and `top_k=20` for normal use. Their reported benchmark setup may use different task-specific sampling parameters. * **Benchmarks**: benchmark numbers in the model cards are reported by DeepReinforce. They are useful for context, but the command panel leaves recipes unverified until exact runs are signed off. ## 3. Usage Examples ### 3.1 Basic Chat Completion `message.reasoning_content` is only populated when the server was launched with the `--reasoning-parser qwen3` toggle (see the [Playground](#playground)); otherwise the `...` trace stays inline in `message.content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="Ornith-1.0-9B", messages=[ {"role": "user", "content": "Write a compact Python function is_prime(n)."} ], temperature=0.6, top_p=0.95, max_tokens=1024, extra_body={"top_k": 20}, ) message = response.choices[0].message print("=============== Reasoning ===============") print(message.reasoning_content) print("=============== Answer ==================") print(message.content) ``` ### 3.2 Tool Calling Enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground) and launch with the resulting command. Then use the standard OpenAI-compatible `tools` field: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "run_tests", "description": "Run the project's test suite.", "parameters": { "type": "object", "properties": { "target": {"type": "string", "description": "Test target or command"} }, "required": ["target"], }, }, }] response = client.chat.completions.create( model="Ornith-1.0-9B", messages=[{"role": "user", "content": "Run the unit tests for the parser module."}], tools=tools, tool_choice="auto", temperature=0.6, top_p=0.95, max_tokens=2048, ) print(response.choices[0].message.tool_calls) ``` # DeepSeek-Math-V2 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-Math-V2 ## 1. Model Introduction [DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) is DeepSeek's advanced mathematical reasoning model with strong theorem-proving capabilities. The model demonstrates exceptional performance on mathematical competitions, achieving gold-level scores on IMO 2025 and CMO 2024, and a near-perfect 118/120 on Putnam 2024 with scaled test-time compute. **Key Features:** * **Strong Theorem-Proving**: Gold-level performance on IMO 2025 and CMO 2024 * **Self-Verifiable Reasoning**: Implements self-verifiable mathematical reasoning for improved accuracy * **Competition-Level Math**: Near-perfect score (118/120) on Putnam 2024 * **Large MoE Model**: \~671B total parameters, requires high-memory GPUs (B200 183GB or B300 275GB) **Available Models:** * **BF16 (Full Weights)**: [deepseek-ai/DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) - Full precision weights **License:** To use DeepSeek-Math-V2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2/blob/main/LICENSE) for details. ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. DeepSeek-Math-V2 is built on DeepSeek-V3.2 and uses DSA sparse attention. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on this model. ### 3.2 Configuration Tips **Hardware Requirements:** * **B200 (183GB)**: BF16 tp=8 * **B300 (275GB)**: BF16 tp=8 **DP Attention:** * Enable DP attention for high-throughput scenarios * The `--attn-dp-size` value commonly matches the `--tp` value * Trade-off: Higher throughput at the cost of slightly increased latency ## 4. Model Invocation ### 4.1 Deployment Command Deploy the model using the command generated above. Example for B200: ```shell Command theme={null} sglang serve --model-path deepseek-ai/DeepSeek-Math-V2 \ --tp 8 \ --ep 8 \ --reasoning-parser deepseek-r1 \ --host 0.0.0.0 \ --port 30000 ``` ### 4.2 Mathematical Reasoning DeepSeek-Math-V2 excels at mathematical problem-solving with step-by-step reasoning. **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Mathematical reasoning problem response = client.chat.completions.create( model="deepseek-ai/DeepSeek-Math-V2", messages=[ {"role": "user", "content": "Prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2"} ], max_tokens=4096, stream=True ) # Process the stream thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= We need to prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2. This is a classic formula for the sum of the first n natural numbers. We can prove by induction. Base case: n=1, LHS = 1, RHS = 1*(1+1)/2 = 1*2/2 = 1. Holds. Inductive step: Assume true for n = k, i.e., 1 + 2 + ... + k = k(k+1)/2. Then for n = k+1, sum = 1 + 2 + ... + k + (k+1) = [k(k+1)/2] + (k+1) = (k(k+1) + 2(k+1))/2 = (k+1)(k+2)/2 = (k+1 )((k+1)+1)/2. So holds for k+1. By induction, holds for all positive integers n. ... =============== Content ================= We can prove the well-known formula for the sum of the first \(n\) positive integers in several ways. Two of the most elementary are presented below. --- ### 1. Proof by mathematical induction **Base case (\(n=1\))**: \[ 1 = \frac{1\cdot(1+1)}{2}= \frac{1\cdot2}{2}=1, \] so the formula holds for \(n=1\). **Inductive hypothesis:** Assume that for some positive integer \(k\) the formula is true, i.e. \[ 1+2+\dots+k = \frac{k(k+1)}{2}. \] **Inductive step (\(k \to k+1\))**: Consider the sum up to \(k+1\): \[ \begin{aligned} 1+2+\dots+k+(k+1) &= \bigl(1+2+\dots+k\bigr) + (k+1) \\[4pt] &= \frac{k(k+1)}{2} + (k+1) \qquad\text{(by the induction hypothesis)}\\[4pt] &= (k+1)\left(\frac{k}{2}+1\right)\\[4pt] &= (k+1)\frac{k+2}{2}\\[4pt] &= \frac{(k+1)(k+2)}{2}\\[4pt] &= \frac{(k+1)\bigl((k+1)+1\bigr)}{2}. \end{aligned} \] Thus the formula also holds for \(n=k+1\). By the principle of mathematical induction, \[ 1+2+3+\dots+n = \frac{n(n+1)}{2} \] for every positive integer \(n\). --- ### 2. Proof by pairing (Gauss’s trick) Let \[ S = 1 + 2 + 3 + \dots + n. \] Write the same sum in reverse order: \[ S = n + (n-1) + (n-2) + \dots + 1. \] Add the two equalities term‑by‑term: \[ \begin{aligned} 2S &= (1+n) + \bigl(2+(n-1)\bigr) + \bigl(3+(n-2)\bigr) + \dots + (n+1)\\ &= \underbrace{(n+1)+(n+1)+\dots+(n+1)}_{n\ \text{times}}\\ &= n\,(n+1). \end{aligned} \] Therefore \[ S = \frac{n(n+1)}{2}. \] Both proofs are rigorous and show that the formula holds for all positive integers \(n\). ``` ### 4.3 Competition-Level Problems **Example: IMO-style Problem:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # IMO-style problem response = client.chat.completions.create( model="deepseek-ai/DeepSeek-Math-V2", messages=[ {"role": "user", "content": "Let a, b, c be positive real numbers such that abc = 1. Prove that (a-1+1/b)(b-1+1/c)(c-1+1/a) <= 1."} ], max_tokens=8192, stream=True ) # Process the stream thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= We need to prove that for positive real numbers a,b,c with abc = 1, we have: \[ (a - 1 + \frac{1}{b})(b - 1 + \frac{1}{c})(c - 1 + \frac{1}{a}) \le 1. \] We can rewrite the expressions: Since abc=1, we have 1/b = ac, 1/c = ab, 1/a = bc. Wait careful: abc=1 => 1/b = ac? Actually 1/b = ac? Let's check: abc=1 => ac = 1/b? Multiply both sides by something: abc=1 => (ac) b = 1 => ac = 1/b. Yes, because (ac) * b = 1 => ac = 1/b. Similarly, ab = 1/c, bc = 1/a. So we can rewrite: ... =============== Content ================= We are given positive real numbers \(a,b,c\) with \(abc=1\). We must prove \[ \Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 . \] --- ### 1. A convenient substitution Because \(abc=1\), we can write \[ a=\frac{x}{y},\qquad b=\frac{y}{z},\qquad c=\frac{z}{x} \] with positive numbers \(x,y,z\). (For instance, take \(x=1,\;y=\frac1a,\;z=\frac1{ab}\); then indeed \(a=\frac{x}{y},\;b=\frac{y}{z}\) and, using \(abc=1\), we obtain \(c=\frac{z}{x}=\frac1{ab}=c\).) --- ### 2. Rewriting the factors \[ \begin{aligned} a-1+\frac1b &=\frac{x}{y}-1+\frac{z}{y}= \frac{x+z-y}{y},\\[2mm] b-1+\frac1c &=\frac{y}{z}-1+\frac{x}{z}= \frac{x+y-z}{z},\\[2mm] c-1+\frac1a &=\frac{z}{x}-1+\frac{y}{x}= \frac{y+z-x}{x}. \end{aligned} \] Hence the product becomes \[ P=\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr) =\frac{(x+z-y)(x+y-z)(y+z-x)}{xyz}. \] --- ### 3. Reducing to a known inequality We have to show \(P\le1\), i.e. \[ (x+z-y)(x+y-z)(y+z-x)\le xyz . \tag{1} \] Set \[ p=x+y+z,\qquad q=xy+yz+zx,\qquad r=xyz . \] Notice that \[ x+z-y=p-2y,\quad x+y-z=p-2z,\quad y+z-x=p-2x . \] Therefore \[ \begin{aligned} (x+z-y)(x+y-z)(y+z-x) &=(p-2x)(p-2y)(p-2z)\\ &=p^{3}-2p^{2}(x+y+z)+4p(xy+yz+zx)-8xyz\\ &=-p^{3}+4pq-8r . \end{aligned} \] Inequality (1) is thus equivalent to \[ -p^{3}+4pq-8r\le r\quad\Longleftrightarrow\quad 4pq-p^{3}\le 9r . \tag{2} \] --- ### 4. Applying Schur’s inequality Schur’s inequality of third degree states that for any non‑negative \(x,y,z\) \[ p^{3}+9r\ge 4pq . \] Rearranged, this is exactly \(4pq-p^{3}\le 9r\), which is (2). Since our \(x,y,z\) are positive, Schur’s inequality applies and (2) holds. Consequently (1) is true, and we obtain \(P\le1\). --- ### 5. Equality case Equality in Schur’s inequality for positive numbers occurs only when \(x=y=z\). Then \(a=b=c=1\), and indeed the product equals \(1\). --- Thus for all positive \(a,b,c\) with \(abc=1\), \[ \Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 . \] ∎ ``` ## 5. Benchmark ### 5.1 Accuracy Benchmark #### 5.1.1 GSM8K Benchmark **Benchmark Command:** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. **Test Results:** ```text Output theme={null} Accuracy: 0.975 Invalid: 0.000 Latency: 34.358 s Output throughput: 540.162 token/s ``` ### 5.2 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (8x, 183GB each) * Model: DeepSeek-Math-V2 * Tensor Parallelism: 8 * SGLang Version: 0.5.8 #### 5.2.1 Latency Benchmark **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model deepseek-ai/DeepSeek-Math-V2 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 53.34 Total input tokens: 1972 Total input text tokens: 1972 Total generated tokens: 2784 Total generated tokens (retokenized): 2778 Request throughput (req/s): 0.19 Input token throughput (tok/s): 36.97 Output token throughput (tok/s): 52.19 Peak output token throughput (tok/s): 56.00 Peak concurrent requests: 3 Total token throughput (tok/s): 89.16 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5330.72 Median E2E Latency (ms): 5879.28 P90 E2E Latency (ms): 8320.33 P99 E2E Latency (ms): 9921.29 ---------------Time to First Token---------------- Mean TTFT (ms): 183.38 Median TTFT (ms): 177.92 P99 TTFT (ms): 217.64 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 17.96 Median TPOT (ms): 18.39 P99 TPOT (ms): 19.03 ---------------Inter-Token Latency---------------- Mean ITL (ms): 18.57 Median ITL (ms): 18.63 P95 ITL (ms): 19.26 P99 ITL (ms): 19.48 Max ITL (ms): 24.93 ================================================== ``` #### 5.2.2 Throughput Benchmark **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model deepseek-ai/DeepSeek-Math-V2 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 217.36 Total input tokens: 301701 Total input text tokens: 301701 Total generated tokens: 188375 Total generated tokens (retokenized): 187456 Request throughput (req/s): 4.60 Input token throughput (tok/s): 1388.05 Output token throughput (tok/s): 866.67 Peak output token throughput (tok/s): 2589.00 Peak concurrent requests: 109 Total token throughput (tok/s): 2254.72 Concurrency: 89.81 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 19521.73 Median E2E Latency (ms): 12076.76 P90 E2E Latency (ms): 47248.87 P99 E2E Latency (ms): 86862.79 ---------------Time to First Token---------------- Mean TTFT (ms): 790.40 Median TTFT (ms): 456.81 P99 TTFT (ms): 4223.33 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 106.52 Median TPOT (ms): 107.24 P99 TPOT (ms): 238.33 ---------------Inter-Token Latency---------------- Mean ITL (ms): 100.29 Median ITL (ms): 38.34 P95 ITL (ms): 237.00 P99 ITL (ms): 347.49 Max ITL (ms): 3642.56 ================================================== ``` # DeepSeek-OCR Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-OCR ## 1. Model Introduction [DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) is DeepSeek's advanced OCR (Optical Character Recognition) model designed for high-accuracy text extraction from images. The model is optimized for various document processing and image-to-text conversion tasks. **Key Features:** * **Advanced OCR**: High-accuracy text recognition from images and documents * **Multi-Modality**: Supports various image formats and document types **Available Models:** * **Base Model**: [deepseek-ai/DeepSeek-OCR](https://huggingface.co/deepseek-ai/DeepSeek-OCR) - Recommended for OCR tasks **License:** To use DeepSeek-OCR, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR/blob/main/LICENSE) for details. For more details, please refer to the [official DeepSeek-OCR repository](https://github.com/deepseek-ai/DeepSeek-OCR). ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 OCR-Specific Prompts DeepSeek-OCR accepts recommended prompts from the model card: ```text theme={null} <|grounding|>Convert the document to markdown. ``` ```text theme={null} Free OCR. ``` **OpenAI-compatible image request example:** ```python Example theme={null} import requests url = "http://localhost:30000/v1/chat/completions" data = { "model": "deepseek-ai/DeepSeek-OCR", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "\n<|grounding|>Convert the document to markdown." }, { "type": "image_url", "image_url": {"url": "https://example.com/your_image.jpg"} }, ], } ], "max_tokens": 512, } response = requests.post(url, json=data) print(response.text) ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X GPU (1x) * Model: DeepSeek-OCR * Tensor Parallelism: 1 * sglang version: 0.5.7 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-OCR \ --tp 1 \ --dtype float16 \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 8000 \ --model deepseek-ai/DeepSeek-OCR \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 4.45 Total input tokens: 1972 Total input text tokens: 1972 Total input vision tokens: 0 Total generated tokens: 2784 Total generated tokens (retokenized): 2770 Request throughput (req/s): 2.25 Input token throughput (tok/s): 442.89 Output token throughput (tok/s): 625.26 Peak output token throughput (tok/s): 635.00 Peak concurrent requests: 4 Total token throughput (tok/s): 1068.16 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 443.32 Median E2E Latency (ms): 493.29 ---------------Time to First Token---------------- Mean TTFT (ms): 21.59 Median TTFT (ms): 20.89 P99 TTFT (ms): 24.81 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1.47 Median TPOT (ms): 1.52 P99 TPOT (ms): 1.53 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.52 Median ITL (ms): 1.51 P95 ITL (ms): 1.76 P99 ITL (ms): 1.93 Max ITL (ms): 8.28 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-OCR \ --tp 1 \ --ep 1 \ --dtype float16 \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 8000 \ --model deepseek-ai/DeepSeek-OCR \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 16.24 Total input tokens: 301698 Total input text tokens: 301698 Total input vision tokens: 0 Total generated tokens: 188375 Total generated tokens (retokenized): 186927 Request throughput (req/s): 61.59 Input token throughput (tok/s): 18582.90 Output token throughput (tok/s): 11602.84 Peak output token throughput (tok/s): 15479.00 Peak concurrent requests: 179 Total token throughput (tok/s): 30185.75 Concurrency: 85.53 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1388.60 Median E2E Latency (ms): 901.43 ---------------Time to First Token---------------- Mean TTFT (ms): 73.36 Median TTFT (ms): 50.21 P99 TTFT (ms): 349.53 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.42 Median TPOT (ms): 7.31 P99 TPOT (ms): 27.99 ---------------Inter-Token Latency---------------- Mean ITL (ms): 7.04 Median ITL (ms): 4.62 P95 ITL (ms): 21.11 P99 ITL (ms): 36.92 Max ITL (ms): 172.15 ================================================== ``` # DeepSeek-OCR-2 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2 ## 1. Model Introduction [DeepSeek-OCR-2](https://github.com/deepseek-ai/DeepSeek-OCR-2) is DeepSeek's next-generation OCR (Optical Character Recognition) model, building on DeepSeek-OCR with improved accuracy and broader document understanding capabilities. The model is optimized for high-accuracy text extraction from images across a wide variety of document types and formats. **Key Features:** * **Semantic-Aware Visual Encoding (DeepEncoder V2)**: DeepSeek-OCR-2 introduces DeepEncoder V2, which models document reading order in a more human-like, semantic-driven manner rather than relying on fixed raster scanning. This significantly improves logical reading flow in complex layouts (e.g., multi-column documents). * **Stronger Layout and Structural Understanding**: DeepSeek-OCR-2 demonstrates improved performance on structured documents such as tables, forms, and dense multi-column pages. It reduces reading-order errors and improves overall document parsing robustness compared to the original version. * **Improved Accuracy While Maintaining Token Efficiency**: The original DeepSeek-OCR emphasized aggressive visual token compression. OCR-2 maintains high token efficiency while delivering higher benchmark performance, particularly on document-level understanding tasks. * **Better Generalization Across Complex Document Tasks**: DeepSeek-OCR-2 performs more consistently across multilingual documents, structured data extraction, and visually complex content, making it more suitable for real-world document intelligence scenarios beyond plain text OCR. **Available Models:** * **Base Model**: [deepseek-ai/DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) - Recommended for OCR tasks **License:** To use DeepSeek-OCR-2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2/blob/main/LICENSE.txt) for details. For more details, please refer to the [official DeepSeek-OCR-2 repository](https://github.com/deepseek-ai/DeepSeek-OCR-2). ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The DeepSeek-OCR-2 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, and Intel Arc Pro B-Series GPUs, as well as Intel Xeon CPUs. **Note**: DeepSeek-OCR-2 has \~3B parameters and easily fits on a single modern GPU. For low-latency serving, no model parallelism is needed. For high-throughput requirements, consider using data parallelism with the SGLang Model Gateway — see [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway) for more details. ### 3.2 Configuration Tips * **Single GPU Deployment:** DeepSeek-OCR-2 (\~3B parameters) fits on a single modern GPU — no tensor parallelism required for low-latency serving. * **High Throughput:** For high-throughput scenarios, use data parallelism with the SGLang Model Gateway. See [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway). * **NCCL timeout:** If model loading is slow, increase `--dist-timeout 3600`. * **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage **OpenAI-compatible request example** ```python Example theme={null} import requests url = "http://localhost:30000/v1/chat/completions" data = { "model": "deepseek-ai/DeepSeek-OCR-2", "messages": [ { "role": "user", "content": [ {"type": "text", "text": "\n<|grounding|>Convert the document to markdown."}, {"type": "image_url", "image_url": {"url": "https://example.com/your_image.jpg"}}, ], } ], "max_tokens": 512, } response = requests.post(url, json=data) print(response.text) ``` **Reference** * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Recommended Prompts The following prompts are recommended by the [official model card](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2#main-prompts). **Structured document conversion** — extracts text while preserving layout: ```text Example theme={null} <|grounding|>Convert the document to markdown. ``` **Free-form OCR** — extracts without layouts: ```text Example theme={null} Free OCR. ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA H200 GPU (1x) * Model: DeepSeek-OCR-2 * Tensor Parallelism: 1 * sglang version: 0.0.0.dev1+g93fca0bbc We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. For more details on how to perform evaluation, see [Evaluating New Models with SGLang](../../../docs/developer_guide/evaluating_new_models). #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-OCR-2 \ --enable-multimodal \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 0.0.0.0 \ --port 30000 \ --model deepseek-ai/DeepSeek-OCR-2 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 3.54 Total input tokens: 1972 Total input text tokens: 1972 Total generated tokens: 2784 Total generated tokens (retokenized): 2710 Request throughput (req/s): 2.83 Input token throughput (tok/s): 557.53 Output token throughput (tok/s): 787.10 Peak output token throughput (tok/s): 818.00 Peak concurrent requests: 5 Total token throughput (tok/s): 1344.63 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 352.69 Median E2E Latency (ms): 392.34 P90 E2E Latency (ms): 540.64 P99 E2E Latency (ms): 639.01 ---------------Time to First Token---------------- Mean TTFT (ms): 18.08 Median TTFT (ms): 16.57 P99 TTFT (ms): 25.67 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1.18 Median TPOT (ms): 1.21 P99 TPOT (ms): 1.22 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.21 Median ITL (ms): 1.21 P95 ITL (ms): 1.28 P99 ITL (ms): 1.44 Max ITL (ms): 4.32 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-OCR-2 \ --enable-multimodal \ --tp 1 \ --ep 1 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 0.0.0.0 \ --port 30000 \ --model deepseek-ai/DeepSeek-OCR-2 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 14.79 Total input tokens: 301698 Total input text tokens: 301698 Total generated tokens: 188375 Total generated tokens (retokenized): 185236 Request throughput (req/s): 67.63 Input token throughput (tok/s): 20402.54 Output token throughput (tok/s): 12738.99 Peak output token throughput (tok/s): 17508.00 Peak concurrent requests: 187 Total token throughput (tok/s): 33141.53 Concurrency: 86.87 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1284.50 Median E2E Latency (ms): 866.07 P90 E2E Latency (ms): 3027.32 P99 E2E Latency (ms): 5490.63 ---------------Time to First Token---------------- Mean TTFT (ms): 86.08 Median TTFT (ms): 50.09 P99 TTFT (ms): 613.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.79 Median TPOT (ms): 6.54 P99 TPOT (ms): 50.10 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.42 Median ITL (ms): 4.64 P95 ITL (ms): 23.65 P99 ITL (ms): 39.62 Max ITL (ms): 452.65 ================================================== ``` # DeepSeek-R1 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-R1 ## 1. Model Introduction [DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1) is DeepSeek's advanced reasoning model that combines powerful language understanding with step-by-step reasoning capabilities. The model is available in multiple quantization formats optimized for different hardware platforms. **Key Features:** * **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving * **Multiple Quantizations**: FP8 and FP4 variants for different performance/memory trade-offs * **Hardware Optimization**: Specifically tuned for NVIDIA B200 (Blackwell) and H200 (Hopper) GPUs, AMD MI300X, MI325X and MI355X GPUs, as well as Intel Xeon CPUs * **High Performance**: Optimized for both throughput and latency scenarios **Available Models:** * **FP8 (8-bit quantized)**: [deepseek-ai/DeepSeek-R1-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) - Recommended for H200 and MI300X * **FP4 (4-bit quantized)**: [nvidia/DeepSeek-R1-0528-FP4-v2](https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2) - Recommended for B200 and MI355X * **BF16 (upcast from FP8)**: [unsloth/DeepSeek-R1-0528-BF16](https://huggingface.co/unsloth/DeepSeek-R1-0528-BF16) * **INT8 (channel-wise)**: [meituan/DeepSeek-R1-Channel-INT8](https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8) * **W4A8**: [novita/Deepseek-R1-0528-W4AFP8](https://huggingface.co/novita/Deepseek-R1-0528-W4AFP8) * **AWQ (4-bit)**: [QuixiAI/DeepSeek-R1-0528-AWQ](https://huggingface.co/QuixiAI/DeepSeek-R1-0528-AWQ) * **MXFP4**: [amd/DeepSeek-R1-MXFP4](https://huggingface.co/amd/DeepSeek-R1-MXFP4) **License:** To use DeepSeek-R1, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/blob/main/LICENSE) for details. For more details, please refer to the [official DeepSeek-R1 repository](https://github.com/deepseek-ai/DeepSeek-R1). ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate a basic deployment command for your hardware platform, quantization method, and deployment strategy. ### 3.2 Optimal Configurations Pareto-optimal configurations for B200, H200, MI300X, MI325X, and MI355X hardware. ### 3.3 Configuration Tips DeepSeek-R1 shares the same MoE architecture as DeepSeek-V3, so the same hardware and optimization recommendations apply. **Recommended GPU configurations by weight type:**
Weight Type Supported Hardware
FP8 (recommended) 8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20
BF16 (upcast from FP8) 2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800
INT8 16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Ascend A3 Series
W4A8 / AWQ / MXFP4 / NVFP4 8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200
> The official DeepSeek-R1 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it. **DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels to avoid JIT overhead (\~10 min): ```bash theme={null} python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-R1 --tp 8 --trust-remote-code ``` **Data Parallelism Attention (`--attn-dp-size`):** Recommended for high-throughput scenarios. Use `--tp 8 --attn-dp-size 8` on a single 8-GPU node. **NCCL timeout:** If model loading is slow, increase: `--dist-timeout 3600`. **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser DeepSeek-R1 supports advanced reasoning capabilities with built-in thinking process. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-0528 \ --reasoning-parser deepseek-r1 \ --tp 8 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-0528", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling DeepSeek-R1 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-0528 \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --chat-template examples/chat_template/tool_chat_template_deepseekr1.jinja \ --tp 8 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-0528", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"🔧 Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= 🔧 Tool Call: get_weather Arguments: 🔧 Tool Call: None Arguments: {"location": "Beijing"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-0528", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding) DeepSeek-R1 supports EAGLE-based Multi-Token Prediction (MTP), the same mechanism as DeepSeek-V3. Refer to [DeepSeek-V3 §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-3-multi-token-prediction-eagle-speculative-decoding) for the complete launch command, flag reference, tuning guidance (`--speculative-num-steps`, `--speculative-eagle-topk`, `--max-running-requests`), and `bench_speculative.py` link. R1's speed benchmark commands that include `--speculative-*` flags use this mechanism. #### 4.2.4 Thinking Budget Limit the model's thinking token budget using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`: ```shell Command theme={null} python3 -m sglang.launch_server \ --model deepseek-ai/DeepSeek-R1 \ --tp 8 \ --port 30000 \ --reasoning-parser deepseek-r1 \ --enable-custom-logit-processor ``` ```python Example theme={null} import openai from sglang.srt.sampling.custom_logit_processor import DeepSeekR1ThinkingBudgetLogitProcessor client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*") response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1", messages=[{"role": "user", "content": "Is Paris the Capital of France?"}], max_tokens=1024, extra_body={ "custom_logit_processor": DeepSeekR1ThinkingBudgetLogitProcessor().to_str(), "custom_params": {"thinking_budget": 512}, }, ) print(response) ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: B200 GPU (8x) * Model: DeepSeek-R1-0528 * Tensor Parallelism: 8 * SGLang Version: 0.5.6.post1 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Test Scenarios Three core scenarios reflect real-world usage patterns:
Scenario Input Length Output Length Use Case
**Chat** 1K 1K Most common conversational AI workload
**Reasoning** 1K 8K Long-form generation, complex reasoning tasks
**Summarization** 8K 1K Document summarization, RAG retrieval
#### 5.1.2 Concurrency Levels Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off: * **Low Concurrency**: `--max-concurrency 1` (Latency-optimized) * **Medium Concurrency**: `--max-concurrency 16` (Balanced) * **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) #### 5.1.3 Number of Prompts For each concurrency level, configure `num_prompts` to simulate realistic user loads: * **Quick Test**: `num_prompts = concurrency × 1` (minimal test) * **Recommended**: `num_prompts = concurrency × 5` (standard benchmark) * **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade) *** #### 5.1.4 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-0528 \ --tp 8 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 40.00 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4210 Total generated tokens (retokenized): 4205 Request throughput (req/s): 0.25 Input token throughput (tok/s): 152.52 Output token throughput (tok/s): 105.24 Peak output token throughput (tok/s): 110.00 Peak concurrent requests: 2 Total token throughput (tok/s): 257.76 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3998.40 Median E2E Latency (ms): 3207.53 ---------------Time to First Token---------------- Mean TTFT (ms): 153.00 Median TTFT (ms): 140.76 P99 TTFT (ms): 214.66 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.16 Median TPOT (ms): 9.15 P99 TPOT (ms): 9.21 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.16 Median ITL (ms): 9.15 P95 ITL (ms): 9.47 P99 ITL (ms): 9.63 Max ITL (ms): 15.45 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 51.21 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40725 Total generated tokens (retokenized): 40458 Request throughput (req/s): 1.56 Input token throughput (tok/s): 774.66 Output token throughput (tok/s): 795.30 Peak output token throughput (tok/s): 1088.00 Peak concurrent requests: 21 Total token throughput (tok/s): 1569.96 Concurrency: 13.93 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8918.33 Median E2E Latency (ms): 9466.16 ---------------Time to First Token---------------- Mean TTFT (ms): 273.51 Median TTFT (ms): 131.71 P99 TTFT (ms): 839.57 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 17.56 Median TPOT (ms): 17.46 P99 TPOT (ms): 28.68 ---------------Inter-Token Latency---------------- Mean ITL (ms): 17.02 Median ITL (ms): 14.70 P95 ITL (ms): 16.41 P99 ITL (ms): 112.38 Max ITL (ms): 461.90 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 110.46 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252162 Total generated tokens (retokenized): 251441 Request throughput (req/s): 4.53 Input token throughput (tok/s): 2261.80 Output token throughput (tok/s): 2282.90 Peak output token throughput (tok/s): 3900.00 Peak concurrent requests: 109 Total token throughput (tok/s): 4544.71 Concurrency: 92.26 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 20380.71 Median E2E Latency (ms): 19391.65 ---------------Time to First Token---------------- Mean TTFT (ms): 563.14 Median TTFT (ms): 147.62 P99 TTFT (ms): 2632.11 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 40.11 Median TPOT (ms): 41.98 P99 TPOT (ms): 50.10 ---------------Inter-Token Latency---------------- Mean ITL (ms): 39.37 Median ITL (ms): 26.36 P95 ITL (ms): 98.16 P99 ITL (ms): 150.08 Max ITL (ms): 2052.85 ================================================== ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 411.34 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 44452 Total generated tokens (retokenized): 44390 Request throughput (req/s): 0.02 Input token throughput (tok/s): 14.83 Output token throughput (tok/s): 108.07 Peak output token throughput (tok/s): 110.00 Peak concurrent requests: 2 Total token throughput (tok/s): 122.90 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 41132.04 Median E2E Latency (ms): 44288.71 ---------------Time to First Token---------------- Mean TTFT (ms): 125.76 Median TTFT (ms): 126.19 P99 TTFT (ms): 137.69 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.21 Median TPOT (ms): 9.20 P99 TPOT (ms): 9.27 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.23 Median ITL (ms): 9.22 P95 ITL (ms): 9.64 P99 ITL (ms): 9.86 Max ITL (ms): 15.18 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 348.93 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 318226 Total generated tokens (retokenized): 317630 Request throughput (req/s): 0.23 Input token throughput (tok/s): 113.69 Output token throughput (tok/s): 912.02 Peak output token throughput (tok/s): 1088.00 Peak concurrent requests: 19 Total token throughput (tok/s): 1025.70 Concurrency: 14.07 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 61360.70 Median E2E Latency (ms): 62071.20 ---------------Time to First Token---------------- Mean TTFT (ms): 176.02 Median TTFT (ms): 153.75 P99 TTFT (ms): 268.44 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 15.42 Median TPOT (ms): 15.59 P99 TPOT (ms): 16.07 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.39 Median ITL (ms): 15.17 P95 ITL (ms): 16.62 P99 ITL (ms): 18.13 Max ITL (ms): 226.59 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 589.31 Total input tokens: 158939 Total input text tokens: 158939 Total input vision tokens: 0 Total generated tokens: 1300705 Total generated tokens (retokenized): 1297658 Request throughput (req/s): 0.54 Input token throughput (tok/s): 269.70 Output token throughput (tok/s): 2207.16 Peak output token throughput (tok/s): 2944.00 Peak concurrent requests: 68 Total token throughput (tok/s): 2476.86 Concurrency: 57.03 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 105032.36 Median E2E Latency (ms): 108229.09 ---------------Time to First Token---------------- Mean TTFT (ms): 223.91 Median TTFT (ms): 158.15 P99 TTFT (ms): 474.86 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 25.94 Median TPOT (ms): 26.72 P99 TPOT (ms): 27.99 ---------------Inter-Token Latency---------------- Mean ITL (ms): 25.79 Median ITL (ms): 25.37 P95 ITL (ms): 26.70 P99 ITL (ms): 105.49 Max ITL (ms): 237.91 ================================================== ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 40.65 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4210 Total generated tokens (retokenized): 4195 Request throughput (req/s): 0.25 Input token throughput (tok/s): 1031.65 Output token throughput (tok/s): 103.56 Peak output token throughput (tok/s): 110.00 Peak concurrent requests: 2 Total token throughput (tok/s): 1135.20 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4063.62 Median E2E Latency (ms): 3296.13 ---------------Time to First Token---------------- Mean TTFT (ms): 165.91 Median TTFT (ms): 154.96 P99 TTFT (ms): 240.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.26 Median TPOT (ms): 9.27 P99 TPOT (ms): 9.42 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.28 Median ITL (ms): 9.28 P95 ITL (ms): 9.66 P99 ITL (ms): 9.83 Max ITL (ms): 14.06 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 56.71 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41589 Total generated tokens (retokenized): 41490 Request throughput (req/s): 1.41 Input token throughput (tok/s): 5290.75 Output token throughput (tok/s): 733.41 Peak output token throughput (tok/s): 1024.00 Peak concurrent requests: 20 Total token throughput (tok/s): 6024.16 Concurrency: 14.25 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 10098.99 Median E2E Latency (ms): 10623.46 ---------------Time to First Token---------------- Mean TTFT (ms): 486.80 Median TTFT (ms): 189.59 P99 TTFT (ms): 2138.73 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 19.06 Median TPOT (ms): 19.23 P99 TPOT (ms): 30.69 ---------------Inter-Token Latency---------------- Mean ITL (ms): 18.53 Median ITL (ms): 15.63 P95 ITL (ms): 16.64 P99 ITL (ms): 109.71 Max ITL (ms): 1471.36 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 115.55 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 169680 Total generated tokens (retokenized): 169275 Request throughput (req/s): 2.77 Input token throughput (tok/s): 11024.93 Output token throughput (tok/s): 1468.50 Peak output token throughput (tok/s): 2254.00 Peak concurrent requests: 70 Total token throughput (tok/s): 12493.43 Concurrency: 59.45 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 21465.98 Median E2E Latency (ms): 20686.26 ---------------Time to First Token---------------- Mean TTFT (ms): 913.93 Median TTFT (ms): 224.92 P99 TTFT (ms): 6257.83 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 39.93 Median TPOT (ms): 40.99 P99 TPOT (ms): 60.91 ---------------Inter-Token Latency---------------- Mean ITL (ms): 38.83 Median ITL (ms): 26.29 P95 ITL (ms): 113.81 P99 ITL (ms): 176.94 Max ITL (ms): 5521.53 ================================================== ``` #### 5.1.5 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 1319 \ --num-threads 1316 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. **Test Results:** ```text Output theme={null} Accuracy: 0.959 Invalid: 0.000 Latency: 29.185 s Output throughput: 4854.672 token/s ``` # DeepSeek-V3 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V3 ## 1. Model Introduction [DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3) is a large-scale Mixture-of-Experts (MoE) language model developed by DeepSeek, designed to deliver strong general-purpose reasoning, coding, and tool-augmented capabilities with high training and inference efficiency. As the latest generation in the DeepSeek model family, DeepSeek V3 introduces systematic architectural and training innovations that significantly improve performance across reasoning, mathematics, coding, and long-context understanding, while maintaining a competitive compute cost. Key highlights include: * **Efficient MoE architecture**: DeepSeek V3 adopts a fine-grained Mixture-of-Experts design with a large number of experts and sparse activation, enabling high model capacity while keeping inference and training costs manageable. * **Advanced reasoning and coding**: The model demonstrates strong performance on mathematical reasoning, logical inference, and real-world coding benchmarks, benefiting from improved data curation and training strategies. * **Long-context capability**: DeepSeek V3 supports extended context lengths, allowing it to handle long documents, complex multi-step reasoning, and agent-style workflows more effectively. * **Tool use and function calling**: The model is trained to support structured outputs and tool invocation, enabling seamless integration with external tools and agent frameworks during inference. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips **Recommended GPU configurations by weight type:**
Weight Type Supported Hardware
FP8 (recommended) 8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20
BF16 (upcast from FP8) 2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800
INT8 16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Ascend A3 Series
W4A8 / AWQ / MXFP4 / NVFP4 8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200
> The official DeepSeek-V3 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it. **DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels before the first server run to avoid JIT overhead (\~10 min): ```bash theme={null} python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3 --tp 8 --trust-remote-code ``` DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`. **Data Parallelism Attention (`--attn-dp-size`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--tp 8 --attn-dp-size 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads. **NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`. **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [Basic API Usage](../../../docs/get-started/quickstart) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser DeepSeek-V3 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model deepseek-ai/DeepSeek-V3 \ --reasoning-parser deepseek-v3 \ --tp 8 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, extra_body = {"chat_template_kwargs": {"thinking": True}}, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To determine 15% of a number, follow these steps: **Step 1: Understand the Problem** You need to find 15% of a given number. Let's assume the number is 240 for this example. **Step 2: Convert the Percentage to a Decimal** To work with percentages in calculations, convert the percentage to its decimal form. To do this, divide the percentage by 100. \[ 15\% = \frac{15}{100} = 0.15 \] **Step 3: Multiply the Decimal by the Number** Now, multiply the decimal form of the percentage by the number you want to find the percentage of. \[ 0.15 \times 240 \] **Step 4: Perform the Multiplication** Calculate the product: \[ 0.15 \times 240 = 36 \] **Step 5: Conclusion** Therefore, 15% of 240 is: \boxed{36} The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling DeepSeek-V3 supports tool calling capabilities. Enable the tool call parser: **Deployment Command:** ```shell Command theme={null} python -m sglang.launch_server \ --model deepseek-ai/DeepSeek-V3 \ --tool-call-parser deepseekv3 \ --reasoning-parser deepseek-v3 \ --chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` **Quick Test (curl):** ```shell Command theme={null} curl "http://127.0.0.1:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ -d '{ "temperature": 0, "max_tokens": 100, "model": "deepseek-ai/DeepSeek-V3", "tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}], "messages": [{"role": "user", "content": "How'\''s the weather in Beijing today?"}] }' ``` Use a low `temperature` (e.g. `0`) for more consistent tool call results. The `--chat-template` flag above provides an improved unified prompt for tool use. **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, extra_body = {"chat_template_kwargs": {"thinking": True}}, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** Please attach the code blocks below to the previous Python script. ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding) SGLang implements DeepSeek V3 Multi-Token Prediction (MTP) based on [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding). With this optimization, decoding speed improves by up to **1.8×** at batch size 1 and **1.5×** at batch size 32 on H200 TP8. **Enable with:** ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --speculative-algorithm EAGLE \ --trust-remote-code \ --tp 8 ``` The default configuration is `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. Find the best values for your workload with [bench\_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). The minimum viable config is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`. For large batch sizes (>48), increase `--max-running-requests` beyond the default of 48 for MTP. Also set `--cuda-graph-bs-decode` to include your target batch sizes (default captured sizes for speculative decoding: 48). The spec-v2 overlap scheduler is enabled by default. It improves performance by overlapping draft and verification stages. Pass `--disable-overlap-schedule` to disable. #### 4.2.4 MLA Optimizations DeepSeek V3 uses [Multi-head Latent Attention (MLA)](https://arxiv.org/pdf/2405.04434), an attention mechanism that improves inference efficiency. SGLang implements several optimizations: * **Weight Absorption:** Reorders matrix multiplications to improve decoding phase efficiency. * **MLA Attention Backends:** FA3, Flashinfer, FlashMLA, TRTLLM MLA (Blackwell), and Triton. FA3 is the default. * **FP8 Quantization:** W8A8 FP8 and KV Cache FP8, with BMM operators for weight-absorbed MLA in FP8. * **CUDA Graph & Torch.compile:** Both MLA and MoE support CUDA Graph and Torch.compile for reduced decoding latency. * **Chunked Prefix Cache:** Increases throughput for long-sequence chunked prefill (FlashAttention3 backend only). Overall, these optimizations achieve up to **7×** output throughput improvement vs. the baseline. **Reference:** See [SGLang v0.3 blog](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#deepseek-multi-head-latent-attention-mla-throughput-optimizations) and [Slides](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/lmsys_1st_meetup_deepseek_mla.pdf) for details. #### 4.2.5 Multi-Node Deployment For multi-node serving and hardware-specific examples: * [8× H200 / 4–8× B200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#using-docker-recommended) * [8× MI300X](../../../docs/hardware-platforms/amd_gpu#running-deepseek-v3) * [2×8× H200 with Docker](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h2008-nodes-and-docker) * [4×8× A100](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-four-a1008-nodes) * [8× A100 AWQ](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-8-a100a800-with-awq-quantization) * [16× A100 INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-16-a100a800-with-int8-quantization) * [32× L40S INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-32-l40s-with-int8-quantization) * [Xeon 6980P CPU](../../../docs/hardware-platforms/cpu_server#example-running-deepseek-v3-1-terminus) * [4× Ascend A3 Series (int8)](../../../docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1#multi-node-pd-disaggregation-deployment) **Blog references for large-scale deployment:** * [Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP](https://lmsys.org/blog/2025-06-16-gb200-part-1/) ([Part I](https://lmsys.org/blog/2025-06-16-gb200-part-1/), [Part II](https://lmsys.org/blog/2025-09-25-gb200-part-2/)) * [PD Disaggregation and Large-Scale Expert Parallelism on 96× H100](https://lmsys.org/blog/2025-05-05-large-scale-ep/) * [Best Practices for Serving DeepSeek-R1 on H20](https://lmsys.org/blog/2025-09-26-sglang-ant-group/) ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X GPU (8x) * Model: DeepSeek-V3 * Tensor Parallelism: 8 * sglang version: 0.5.7 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3 \ --tp 8 \ --attn-dp-size 8 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 8000 \ --model deepseek-ai/DeepSeek-V3 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 81.27 Total input tokens: 1972 Total input text tokens: 1972 Total input vision tokens: 0 Total generated tokens: 2784 Total generated tokens (retokenized): 2774 Request throughput (req/s): 0.12 Input token throughput (tok/s): 24.27 Output token throughput (tok/s): 34.26 Peak output token throughput (tok/s): 65.00 Peak concurrent requests: 2 Total token throughput (tok/s): 58.52 Concurrency: 1.00 Accept length: 2.61 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8123.17 Median E2E Latency (ms): 7982.65 ---------------Time to First Token---------------- Mean TTFT (ms): 1080.76 Median TTFT (ms): 1248.82 P99 TTFT (ms): 1896.37 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 25.04 Median TPOT (ms): 24.76 P99 TPOT (ms): 32.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 25.41 Median ITL (ms): 20.14 P95 ITL (ms): 60.28 P99 ITL (ms): 60.99 Max ITL (ms): 61.49 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3 \ --tp 8 \ --ep 8 \ --attn-dp-size 8 \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 8000 \ --model deepseek-ai/DeepSeek-V3 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 406.16 Total input tokens: 301701 Total input text tokens: 301701 Total input vision tokens: 0 Total generated tokens: 188375 Total generated tokens (retokenized): 187542 Request throughput (req/s): 2.46 Input token throughput (tok/s): 742.81 Output token throughput (tok/s): 463.80 Peak output token throughput (tok/s): 1299.00 Peak concurrent requests: 109 Total token throughput (tok/s): 1206.61 Concurrency: 87.53 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 35552.98 Median E2E Latency (ms): 21466.07 ---------------Time to First Token---------------- Mean TTFT (ms): 1521.51 Median TTFT (ms): 476.80 P99 TTFT (ms): 8329.50 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 214.73 Median TPOT (ms): 152.00 P99 TPOT (ms): 1155.85 ---------------Inter-Token Latency---------------- Mean ITL (ms): 182.10 Median ITL (ms): 79.18 P95 ITL (ms): 398.60 P99 ITL (ms): 1488.96 Max ITL (ms): 43465.60 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:8000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * **Test Results**: * DeepSeek-V3 ```text Output theme={null} Accuracy: 0.960 Invalid: 0.000 Latency: 32.450 s Output throughput: 614.211 token/s ``` #### 5.2.2 MMLU Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run mmlu \ --base-url http://127.0.0.1:8000/v1 \ --num-threads 64 --max-tokens 2048 ``` The command evaluates all 57 subjects. Historical results below use the removed 10-subject harness and are not measurements of the sgl-eval command above. * **Test Results**: * DeepSeek-V3 ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.800 subject: anatomy, #q:135, acc: 0.874 subject: astronomy, #q:152, acc: 0.928 subject: business_ethics, #q:100, acc: 0.880 subject: clinical_knowledge, #q:265, acc: 0.928 subject: college_biology, #q:144, acc: 0.965 subject: college_chemistry, #q:100, acc: 0.670 subject: college_computer_science, #q:100, acc: 0.840 subject: college_mathematics, #q:100, acc: 0.800 subject: college_medicine, #q:173, acc: 0.861 Total latency: 58.339 Average accuracy: 0.871 ``` # DeepSeek-V3.1 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V3_1 ## 1. Model Introduction [DeepSeek V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) is an advanced Mixture-of-Experts (MoE) large language model developed by DeepSeek, representing a major capability and usability upgrade over DeepSeek V3. As a refined iteration in the DeepSeek V3 family, DeepSeek V3.1 introduces a hybrid reasoning paradigm that supports both fast non-thinking responses and explicit multi-step reasoning, alongside significantly improved tool calling and agentic behavior. The model demonstrates strong performance across reasoning, mathematics, coding, long-context understanding, and real-world agent workflows, benefiting from continued training, alignment optimization, and inference-time refinements. DeepSeek V3.1 is designed to serve as a robust general-purpose foundation model, well suited for conversational AI, structured tool invocation, search-augmented generation, and complex multi-step tasks, while maintaining high efficiency through its sparse MoE architecture. **[DeepSeek-V3.1-Terminus](https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus)** is an experimental version designed for general conversations and long-context processing. It features hybrid thinking capabilities, allowing you to toggle between "Think" mode for deliberate reasoning and "Non-Think" mode for faster responses. Recommended for general conversations, long-context processing, and experimental use cases. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips DeepSeek-V3.1 shares the same model architecture as DeepSeek-V3, so the same hardware and optimization recommendations apply. **Recommended GPU configurations by weight type:**
Weight Type Supported Hardware
FP8 (recommended) 8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20
BF16 (upcast from FP8) 2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800
INT8 16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Ascend A3 Series
W4A8 / AWQ / MXFP4 / NVFP4 8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200
> The official DeepSeek-V3.1 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it. **DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels before the first server run to avoid JIT overhead (\~10 min): ```bash theme={null} python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3.1 --tp 8 --trust-remote-code ``` DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`. **Data Parallelism Attention (`--attn-dp-size`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--tp 8 --attn-dp-size 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads. **NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`. **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [Basic API Usage](../../../docs/get-started/quickstart) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser DeepSeek-V3.1 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model deepseek-ai/DeepSeek-V3.1-Terminus \ --reasoning-parser deepseek-v3 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.1-Terminus", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, extra_body = {"chat_template_kwargs": {"thinking": True}}, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= First, the problem is asking for 15% of 240. Percent means per hundred, so 15% is the same as 15 out of 100, or 15/100. To find a percentage of a number, I can multiply the number by the percentage expressed as a decimal. So, I need to convert 15% to a decimal. To do that, I divide 15 by 100, which gives me 0.15. Now, I multiply 0.15 by 240. So, the calculation is 0.15 × 240. I can compute this step by step. First, I know that 15% of 100 is 15, but since 240 is larger, I need to adjust. Alternatively, I can think of 10% of 240, which is easy because 10% is just 240 divided by 10, which is 24. Then, 5% is half of 10%, so half of 24 is 12. Therefore, 15% is 10% plus 5%, so 24 plus 12, which equals 36. I should also do the multiplication to confirm. 0.15 × 240. I can break it down: 0.15 × 200 = 30, and 0.15 × 40 = 6, so 30 + 6 = 36. Same answer. So, 15% of 240 is 36. The problem says "step by step," so I should present it clearly. =============== Content ================= To find 15% of 240, follow these steps: 1. Understand that "percent" means "per hundred," so 15% is equivalent to \( \frac{15}{100} \). 2. Convert 15% to a decimal by dividing by 100: \( 15\% = \frac{15}{100} = 0.15 \). 3. Multiply the decimal by 240: \( 0.15 \times 240 \). 4. Perform the multiplication: - \( 0.15 \times 200 = 30 \) - \( 0.15 \times 40 = 6 \) - Add the results: \( 30 + 6 = 36 \). Alternatively, you can find 15% by breaking it into parts: - 10% of 240 is \( \frac{10}{100} \times 240 = 0.10 \times 240 = 24 \). - 5% of 240 is half of 10%, so \( \frac{24}{2} = 12 \). - Add 10% and 5%: \( 24 + 12 = 36 \). Thus, 15% of 240 is 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling DeepSeek-V3.1 and DeepSeek-V3.1-Terminus support tool calling capabilities. Enable the tool call parser: **Deployment Command:** ```shell Command theme={null} python -m sglang.launch_server \ --model deepseek-ai/DeepSeek-V3.1-Terminus \ --tool-call-parser deepseekv31 \ --reasoning-parser deepseek-v3 \ --chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` For DeepSeek-V3.1, use `--tool-call-parser deepseekv31` as well. **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.1-Terminus", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, extra_body = {"chat_template_kwargs": {"thinking": True}}, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= Hmm, the user is asking for the weather in Beijing. This is a straightforward request that matches exactly what the weather tool can provide. I need to call the get_weather function with Beijing as the location parameter. The user didn't specify a temperature unit, so I'll default to Celsius since that's commonly used in most parts of the world. The tool call format needs to be precise - just the city name and unit selection. Once I get the weather data back, I'll present it clearly to the user.I'll check the weather in Beijing for you. =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** Please attach the code blocks below to the previous Python script. ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.1-Terminus", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "Currently, it is **22°C and sunny** in Beijing." ``` #### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding) DeepSeek-V3.1 shares the same architecture as DeepSeek-V3 and supports the same EAGLE-based MTP speculative decoding path. Refer to [DeepSeek-V3 §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-3-multi-token-prediction-eagle-speculative-decoding) for the full configuration, tuning guidance, and `bench_speculative.py` reference. The `--speculative-num-steps`, `--speculative-eagle-topk`, and `--max-running-requests` recommendations apply equally to V3.1. ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X GPU (8x) * Model: DeepSeek-V3.1-Terminus * Tensor Parallelism: 8 * sglang version: 0.5.7 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Test Scenarios Three core scenarios reflect real-world usage patterns:
Scenario Input Length Output Length Use Case
**Chat** 1K 1K Most common conversational AI workload
**Reasoning** 1K 8K Long-form generation, complex reasoning tasks
**Summarization** 8K 1K Document summarization, RAG retrieval
#### 5.1.2 Concurrency Levels Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off: * **Low Concurrency**: `--max-concurrency 1` (Latency-optimized) * **Medium Concurrency**: `--max-concurrency 16` (Balanced) * **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) #### 5.1.3 Number of Prompts For each concurrency level, configure `num_prompts` to simulate realistic user loads: * **Quick Test**: `num_prompts = concurrency × 1` (minimal test) * **Recommended**: `num_prompts = concurrency × 5` (standard benchmark) * **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade) *** #### 5.1.4 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3.1 \ --tp 8 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 106.24 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4201 Request throughput (req/s): 0.09 Input token throughput (tok/s): 57.43 Output token throughput (tok/s): 39.72 Peak output token throughput (tok/s): 43.00 Peak concurrent requests: 2 Total token throughput (tok/s): 97.15 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 10620.29 Median E2E Latency (ms): 8868.09 ---------------Time to First Token---------------- Mean TTFT (ms): 557.85 Median TTFT (ms): 213.58 P99 TTFT (ms): 1625.28 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 23.84 Median TPOT (ms): 23.90 P99 TPOT (ms): 24.03 ---------------Inter-Token Latency---------------- Mean ITL (ms): 23.90 Median ITL (ms): 23.92 P95 ITL (ms): 24.15 P99 ITL (ms): 24.25 Max ITL (ms): 25.44 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 107.71 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40625 Request throughput (req/s): 0.74 Input token throughput (tok/s): 368.28 Output token throughput (tok/s): 378.84 Peak output token throughput (tok/s): 508.00 Peak concurrent requests: 19 Total token throughput (tok/s): 747.12 Concurrency: 13.72 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 18473.65 Median E2E Latency (ms): 19558.42 ---------------Time to First Token---------------- Mean TTFT (ms): 607.91 Median TTFT (ms): 191.32 P99 TTFT (ms): 2135.13 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 35.50 Median TPOT (ms): 35.99 P99 TPOT (ms): 43.62 ---------------Inter-Token Latency---------------- Mean ITL (ms): 35.10 Median ITL (ms): 32.18 P95 ITL (ms): 33.03 P99 ITL (ms): 159.99 Max ITL (ms): 453.99 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 207.65 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 251238 Request throughput (req/s): 2.41 Input token throughput (tok/s): 1203.15 Output token throughput (tok/s): 1216.79 Peak output token throughput (tok/s): 2100.00 Peak concurrent requests: 106 Total token throughput (tok/s): 2419.94 Concurrency: 91.02 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 37800.20 Median E2E Latency (ms): 35921.56 ---------------Time to First Token---------------- Mean TTFT (ms): 835.15 Median TTFT (ms): 236.88 P99 TTFT (ms): 2868.52 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 73.33 Median TPOT (ms): 76.35 P99 TPOT (ms): 97.63 ---------------Inter-Token Latency---------------- Mean ITL (ms): 73.30 Median ITL (ms): 50.82 P95 ITL (ms): 180.67 P99 ITL (ms): 186.83 Max ITL (ms): 1661.39 ================================================== ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 1097.29 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 44462 Total generated tokens (retokenized): 44313 Request throughput (req/s): 0.01 Input token throughput (tok/s): 5.56 Output token throughput (tok/s): 40.52 Peak output token throughput (tok/s): 43.00 Peak concurrent requests: 2 Total token throughput (tok/s): 46.08 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 109725.52 Median E2E Latency (ms): 117748.67 ---------------Time to First Token---------------- Mean TTFT (ms): 156.67 Median TTFT (ms): 156.19 P99 TTFT (ms): 159.87 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 24.41 Median TPOT (ms): 24.51 P99 TPOT (ms): 24.96 ---------------Inter-Token Latency---------------- Mean ITL (ms): 24.65 Median ITL (ms): 24.58 P95 ITL (ms): 25.68 P99 ITL (ms): 25.93 Max ITL (ms): 29.80 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 775.02 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 318306 Total generated tokens (retokenized): 317426 Request throughput (req/s): 0.10 Input token throughput (tok/s): 51.18 Output token throughput (tok/s): 410.70 Peak output token throughput (tok/s): 512.00 Peak concurrent requests: 18 Total token throughput (tok/s): 461.89 Concurrency: 13.86 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 134236.65 Median E2E Latency (ms): 135181.28 ---------------Time to First Token---------------- Mean TTFT (ms): 214.35 Median TTFT (ms): 194.12 P99 TTFT (ms): 300.27 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 33.72 Median TPOT (ms): 34.00 P99 TPOT (ms): 34.75 ---------------Inter-Token Latency---------------- Mean ITL (ms): 33.69 Median ITL (ms): 33.71 P95 ITL (ms): 34.50 P99 ITL (ms): 34.92 Max ITL (ms): 164.76 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 1231.97 Total input tokens: 158939 Total input text tokens: 158939 Total input vision tokens: 0 Total generated tokens: 1301025 Total generated tokens (retokenized): 1296845 Request throughput (req/s): 0.26 Input token throughput (tok/s): 129.01 Output token throughput (tok/s): 1056.05 Peak output token throughput (tok/s): 1472.00 Peak concurrent requests: 67 Total token throughput (tok/s): 1185.07 Concurrency: 56.17 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 216256.25 Median E2E Latency (ms): 224192.84 ---------------Time to First Token---------------- Mean TTFT (ms): 317.68 Median TTFT (ms): 235.28 P99 TTFT (ms): 649.39 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 53.30 Median TPOT (ms): 55.10 P99 TPOT (ms): 56.58 ---------------Inter-Token Latency---------------- Mean ITL (ms): 53.13 Median ITL (ms): 52.95 P95 ITL (ms): 56.23 P99 ITL (ms): 181.04 Max ITL (ms): 208.61 ================================================== ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 114.47 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4194 Request throughput (req/s): 0.09 Input token throughput (tok/s): 366.39 Output token throughput (tok/s): 36.87 Peak output token throughput (tok/s): 42.00 Peak concurrent requests: 2 Total token throughput (tok/s): 403.26 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 11442.86 Median E2E Latency (ms): 9508.87 ---------------Time to First Token---------------- Mean TTFT (ms): 883.78 Median TTFT (ms): 481.38 P99 TTFT (ms): 2217.45 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 24.93 Median TPOT (ms): 25.05 P99 TPOT (ms): 26.11 ---------------Inter-Token Latency---------------- Mean ITL (ms): 25.08 Median ITL (ms): 25.08 P95 ITL (ms): 26.18 P99 ITL (ms): 26.28 Max ITL (ms): 27.41 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 162.33 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41669 Total generated tokens (retokenized): 41443 Request throughput (req/s): 0.49 Input token throughput (tok/s): 1848.27 Output token throughput (tok/s): 256.70 Peak output token throughput (tok/s): 467.00 Peak concurrent requests: 19 Total token throughput (tok/s): 2104.97 Concurrency: 14.52 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 29456.89 Median E2E Latency (ms): 27628.16 ---------------Time to First Token---------------- Mean TTFT (ms): 1784.30 Median TTFT (ms): 1347.21 P99 TTFT (ms): 5384.54 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 53.65 Median TPOT (ms): 52.09 P99 TPOT (ms): 74.39 ---------------Inter-Token Latency---------------- Mean ITL (ms): 53.23 Median ITL (ms): 34.52 P95 ITL (ms): 35.81 P99 ITL (ms): 513.25 Max ITL (ms): 2865.73 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 282.55 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 170000 Total generated tokens (retokenized): 169081 Request throughput (req/s): 1.13 Input token throughput (tok/s): 4508.6 Output token throughput (tok/s): 601.67 Peak output token throughput (tok/s): 1216 Peak concurrent requests: 68 Total token throughput (tok/s): 5110.27 Concurrency: 59.81 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 52810.32 Median E2E Latency (ms): 50981.81 ---------------Time to First Token---------------- Mean TTFT (ms): 786.69 Median TTFT (ms): 499.38 P99 TTFT (ms): 2925.98 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 97.93 Median TPOT (ms): 103.45 P99 TPOT (ms): 157.84 ---------------Inter-Token Latency---------------- Mean ITL (ms): 98.11 Median ITL (ms): 55.7 P95 ITL (ms): 240.71 P99 ITL (ms): 1114.36 ================================================== ``` #### 5.1.5 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 1319 \ --num-threads 1316 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. **Test Results:** ```text Output theme={null} Accuracy: 0.959 Invalid: 0.000 Latency: 29.185 s Output throughput: 4854.672 token/s ``` # DeepSeek-V3.2 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2 ## 1. Model Introduction The DeepSeek-V3.2 series includes three model variants, each optimized for different use cases: **[DeepSeek-V3.2-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp)** is an upgraded version of DeepSeek-V3.1-Terminus, introducing the DeepSeek Sparse Attention (DSA) mechanism through continued training. DSA is a fine-grained sparse attention mechanism powered by a lightning indexer, enabling DeepSeek-V3.2-Exp to achieve significant efficiency improvements in long-context scenarios. Recommended for general conversations, long-context processing, and efficient inference. **[DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)** is the standard version suitable for general tasks and conversational scenarios. For local deployment, we recommend setting the sampling parameters to temperature = 1.0, top\_p = 0.95. Recommended for standard conversations and general tasks. **[DeepSeek-V3.2-Speciale](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Speciale)** is a special variant designed exclusively for deep reasoning tasks. This model is specifically optimized for scenarios requiring complex logical reasoning and deep thinking. However this model does not support tool calls (see below). For local deployment, we recommend setting the sampling parameters to temperature = 1.0, top\_p = 0.95. Recommended for deep reasoning tasks, complex logical problems, and mathematical reasoning. **[DeepSeek-V3.2-NVFP4](https://huggingface.co/nvidia/DeepSeek-V3.2-NVFP4)** is an NVIDIA-optimized NVFP4-quantized variant of DeepSeek-V3.2 for Blackwell devices. It uses ModelOpt FP4 quantization with a choice of MoE runner backends (`flashinfer_trtllm` (recommended), `flashinfer_cutlass`, or `flashinfer_cutedsl`), enabling efficient deployment with lower tensor parallelism (TP=4). It supports the same features as DeepSeek-V3.2 including tool calling, reasoning, and speculative decoding (MTP). **[DeepSeek-V3.2-MXFP4](https://huggingface.co/amd/DeepSeek-V3.2-mxfp4)** is an OCP-MXFP4 optimized variant for DeepSeek-V3.2 for AMD MI300X/MI355X devices. It uses OCP MXFP4 quantization with a triton mxfp4 backend (the same backend for gptoss-120B), enabling efficient deployment with lower tensor parallelism (TP=8) in a single node. It includes the same features as DeepSeek-V3.2 including tool calling, reasoning, fp8-kv, CP, TP and speculative decoding MTP. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ### 2.1 Docker Images Pre-built Docker images are available for different hardware platforms: ```bash Command theme={null} # NVIDIA H200 / B200 docker pull lmsysorg/sglang:latest # AMD MI350 / MI355X docker pull lmsysorg/sglang:v0.5.8-rocm700-mi35x # AMD MI300X # Note: v0.5.8-rocm700-mi30x does not include PR #17504. # Prefer the newest MI30x ROCm image tag from Docker Hub when available, or build from source. docker pull lmsysorg/sglang:v0.5.8-rocm700-mi30x # Ascend NPU (A2/A3 Series) docker pull lmsysorg/sglang:dsv32-a2 docker pull lmsysorg/sglang:dsv32-a3 ``` ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. SGLang supports serving DeepSeek V3.2 on NVIDIA H200, B200, and AMD MI300X/MI355X GPUs. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on DeepSeek-V3.2. ### 3.2 Configuration Tips * **Short-sequence MHA prefill (adaptive):** For prefill sequences shorter than 2048 tokens (default threshold), the DSA backend automatically switches to standard MHA (using FlashAttention variable-length on SM90, TRT-LLM ragged MHA on SM100). To extend this to longer sequences set env var `SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD` to a larger value (potential minor accuracy trade-off). * **DSA prefill/decode attention kernels (`--dsa-prefill-backend`, `--dsa-decode-backend`):** The `dsa` backend is automatically selected for DeepSeek-V3.2. Available kernels: `flashmla_sparse`, `flashmla_sparse_q8` (native FP8 e4m3 sparse prefill — no fp8→bf16 dequantization round-trip; Hopper SM90 + `--kv-cache-dtype fp8_e4m3` only, prefill only), `flashmla_kv`, `flashmla_auto`, `fa3` (Hopper only), `tilelang` (GPU/HPU/NPU), `aiter` (AMD, decode only), `trtllm` (Blackwell only). Defaults: Hopper BF16 KV → `flashmla_sparse` prefill / `fa3` decode; Hopper FP8 KV → `flashmla_kv` both; Blackwell BF16 → `flashmla_sparse` / `trtllm`; Blackwell FP8 → `trtllm` both. * **Index Cache:** Reuses indexer results across layers for efficiency at negligible accuracy cost. For **GLM-5** specifically, append `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for a better speed/accuracy tradeoff. * **HiSparse (experimental):** Reduces per-request GPU memory during long-context decode by offloading KV data to CPU pinned memory. Requires PD disaggregation mode (decode instance only). See [HiSparse Guide](../../../docs/advanced_features/hisparse_guide). * **NVFP4 on Blackwell:** Specify `--quantization modelopt_fp4` and `--moe-runner-backend flashinfer_trtllm` (recommended) / `flashinfer_cutlass` / `flashinfer_cutedsl`. Full example: ```bash theme={null} python -m sglang.launch_server --model nvidia/DeepSeek-V3.2-NVFP4 --tp 4 \ --quantization modelopt_fp4 --moe-runner-backend flashinfer_trtllm \ --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 ``` * **NCCL timeout:** Slow model loading → add `--dist-timeout 3600`. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [Basic API Usage](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser DeepSeek-V3.2 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-V3.2-Exp \ --reasoning-parser deepseek-v3 \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.2-Exp", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, extra_body = {"chat_template_kwargs": {"thinking": True}}, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling DeepSeek-V3.2 and DeepSeek-V3.2-Exp support tool calling capabilities. But they use different parameters. Enable the tool call parser: **Note:** DeepSeek-V3.2-Speciale does **NOT** support tool calling. Launch it with reasoning parser only: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3.2-Speciale \ --trust-remote-code \ --tp-size 8 --attn-dp-size 8 \ --reasoning-parser deepseek-v3 ``` **Deployment Command:** For DeepSeek-V3.2-Exp: ```shell Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-V3.2-Exp \ --tool-call-parser deepseekv31 \ --reasoning-parser deepseek-v3 \ --chat-template ./examples/chat_template/tool_chat_template_deepseekv32.jinja \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` For DeepSeek-V3.2, use `--tool-call-parser deepseekv32` and remove `--chat-template`. **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.2-Exp", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, extra_body = {"chat_template_kwargs": {"thinking": True}}, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.2-Exp", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding) SGLang implements Multi-Token Prediction (MTP) for DeepSeek V3.2 based on [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding). This optimization significantly improves decoding speed for small batch sizes. **With DP Attention:** ```bash Command theme={null} python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --attn-dp-size 8 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 ``` **With Pure TP:** ```bash Command theme={null} python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 ``` Find optimal values for your workload with [bench\_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). The minimum viable config is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`. `--max-running-requests` defaults to 48 for MTP. Increase it for larger batch sizes. The spec-v2 overlap scheduler is enabled by default. Pass `--disable-overlap-schedule` to disable. #### 4.2.4 PD Disaggregation Prefill-Decode (PD) disaggregation separates prefill and decode stages onto different instances, improving GPU utilization for mixed workloads. **Prefill command:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3.2-Exp \ --disaggregation-mode prefill \ --host $LOCAL_IP \ --port $PORT \ --tp 8 \ --attn-dp-size 8 \ --dist-init-addr ${HOST}:${DIST_PORT} \ --trust-remote-code \ --disaggregation-bootstrap-port 8998 \ --mem-fraction-static 0.9 ``` **Decode command:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3.2-Exp \ --disaggregation-mode decode \ --host $LOCAL_IP \ --port $PORT \ --tp 8 \ --attn-dp-size 8 \ --dist-init-addr ${HOST}:${DIST_PORT} \ --trust-remote-code \ --mem-fraction-static 0.9 ``` **Router command:** ```bash Command theme={null} python -m sglang_router.launch_router --pd-disaggregation \ --prefill $PREFILL_ADDR 8998 \ --decode $DECODE_ADDR \ --host 127.0.0.1 \ --port 30000 ``` For production deployments (RBG / LWS-based, DeepEP EP parallelism), see [multi\_node\_deployment docs](../../../docs/references/multi_node_deployment/rbg_pd/deepseekv32_pd). #### 4.2.5 DSA Long-Sequence Context Parallel and PP/CP Enable prefill context parallelism (CP) on CUDA with `--enable-prefill-cp --cp-strategy interleave` for long-sequence workloads. Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for DeepSeek V3.2, GLM-5, GLM-5.1, GLM-5.2, and GLM-5.3. Use `interleave` for these models. Attention DP is supported, with `tp_size / attn_dp_size` CP ranks in each DP group. Dense FFNs retain their configured TP width; `--moe-dense-tp-size 1` selects local dense computation. **Interleave** (`--cp-strategy interleave`): Distributes tokens by `token_idx % cp_size`. It supports fused MoE, FP8 KV cache, and multi-batch prefill. ```bash Command theme={null} # Interleave — FusedMoE + CP8 python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp \ --tp 8 --enable-prefill-cp --attn-cp-size 8 \ --cp-strategy interleave --max-running-requests 32 ``` **PP + CP (multi-node):** Combines Pipeline Parallelism and Context Parallelism for cross-node scaling. The production-optimized configurations below have been verified on Hopper: We suggested `DP2` + `MTP` for local deployment of agentic workflow with DeepSeek V3.2 on Hopper platform: ```shell Command theme={null} export SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS=32 export SGLANG_SET_CPU_AFFINITY=1 # Test workload ISL/OSL=1k/1k, raw tap : 4948.16 toks/sec, MAX ITL 5970 # dp 2 : 5019.54 toks/sec, MAX ITL 7233 # dp 4 : 4942.82 toks/sec, MAX ITL 35654 # dp 2 + mtp : 6842.51 toks/sec, MAX ITL 3081 sglang_args=$(echo serve \ --model-path $MAPPED_MODEL_PATH \ --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 16 \ --attn-dp-size 2 --page-size 64 \ --trust-remote-code --host "0.0.0.0" --port 30000 \ --log-requests \ --context-length 65536 --max-running-requests 128 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \ --allow-auto-truncate --enable-metrics \ --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \ --served-model-name DeepSeek-V3.2-Opt-dp2-mtp ) sglang_args=($sglang_args) sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log ``` **CP + PP + EP** `CP` is currently enabled with `PP=2` on Hopper platform and we can reduce TP=16 to TP=8 from standalone deployment: ```shell Command theme={null} # verified on Hopper platform sglang_args=$(echo serve \ --model-path $MAPPED_MODEL_PATH \ --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 8 --pp-size 2 \ --moe-a2a-backend deepep --ep-size 16 \ --page-size 128 \ --chunked-prefill-size 16384 \ --attention-backend dsa \ --dsa-prefill-backend flashmla_sparse \ --dsa-decode-backend flashmla_sparse \ --enable-prefill-cp \ --cp-strategy interleave \ --cuda-graph-max-bs-decode 128 \ --max-running-requests 128 \ --trust-remote-code --host "0.0.0.0" --port 30000 \ --log-requests \ --context-length 65536 \ --allow-auto-truncate --enable-metrics \ --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \ --served-model-name DeepSeek-V3.2-dsa-pp-cp-ep ) sglang_args=($sglang_args) sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log ``` **fp8 KV + CP + PP** FP8 KV reduces the memory footprint. For DeepSeek V3.2, combine it with interleave CP and PP, without attention DP: ```shell Command theme={null} # verified in Hopper platform cp_config=" \ --enable-prefill-cp \ --cp-strategy interleave \ " # see discussion : https://github.com/sgl-project/sglang/pull/12065 sglang_args=$(echo serve \ --model-path $MAPPED_MODEL_PATH \ --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 8 --pp-size 2 --pp-async-batch-depth 1 \ --trust-remote-code --host "0.0.0.0" --port 30000 \ --log-requests \ --context-length 65536 --max-running-requests 128 \ $cp_config \ --kv-cache-dtype fp8_e4m3 \ --allow-auto-truncate --enable-metrics \ --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \ --served-model-name DeepSeek-V3.2-Opt-fp8kv-pp2-cp4 ) sglang_args=($sglang_args) sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log ``` ## 5. Benchmark ### 5.1 Speed Benchmark on Blackwell **Test Environment:** * Hardware: NVIDIA B200 GPU (8x) * Model: DeepSeek-V3.2-Exp * Tensor Parallelism: 8 * sglang version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-V3.2-Exp \ --tp 8 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model deepseek-ai/DeepSeek-V3.2-Exp \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 29.11 Total input tokens: 1972 Total input text tokens: 1972 Total input vision tokens: 0 Total generated tokens: 2784 Total generated tokens (retokenized): 2777 Request throughput (req/s): 0.34 Input token throughput (tok/s): 67.73 Output token throughput (tok/s): 95.62 Peak output token throughput (tok/s): 157.00 Peak concurrent requests: 3 Total token throughput (tok/s): 163.36 Concurrency: 1.00 Accept length: 2.46 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2909.74 Median E2E Latency (ms): 3088.27 P90 E2E Latency (ms): 4200.62 P99 E2E Latency (ms): 5588.52 ---------------Time to First Token---------------- Mean TTFT (ms): 317.58 Median TTFT (ms): 191.31 P99 TTFT (ms): 740.79 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.09 Median TPOT (ms): 9.25 P99 TPOT (ms): 11.73 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.35 Median ITL (ms): 7.64 P95 ITL (ms): 22.81 P99 ITL (ms): 23.33 Max ITL (ms): 31.45 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-V3.2-Exp \ --tp 8 \ --ep 8 \ --attn-dp-size 8 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model deepseek-ai/DeepSeek-V3.2-Exp \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 219.09 Total input tokens: 301701 Total input text tokens: 301701 Total input vision tokens: 0 Total generated tokens: 188375 Total generated tokens (retokenized): 187443 Request throughput (req/s): 4.56 Input token throughput (tok/s): 1377.06 Output token throughput (tok/s): 859.80 Peak output token throughput (tok/s): 2465.00 Peak concurrent requests: 109 Total token throughput (tok/s): 2236.86 Concurrency: 88.05 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 19291.23 Median E2E Latency (ms): 11927.39 ---------------Time to First Token---------------- Mean TTFT (ms): 530.36 Median TTFT (ms): 444.00 P99 TTFT (ms): 1504.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 106.16 Median TPOT (ms): 106.69 P99 TPOT (ms): 221.12 ---------------Inter-Token Latency---------------- Mean ITL (ms): 100.46 Median ITL (ms): 41.73 P95 ITL (ms): 225.67 P99 ITL (ms): 392.37 Max ITL (ms): 975.03 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * **Test Results**: * DeepSeek-V3.2-Exp ``` Accuracy: 0.980 Invalid: 0.000 Latency: 19.128 s Output throughput: 965.919 token/s ``` * **Full GSM8K (1319 questions)** — evaluate the complete test set with sgl-eval: ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 1319 \ --num-threads 1319 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. * 8-shot: ``` Accuracy: 0.956 Invalid: 0.000 Latency: 25.109 s Output throughput: 5226.235 token/s ``` * 20-shot (long-context; stays close to the 8-shot result): ``` Accuracy: 0.956 Invalid: 0.000 Latency: 29.545 s Output throughput: 4418.617 token/s ``` #### 5.2.2 MMLU Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run mmlu \ --base-url http://127.0.0.1:30000/v1 \ --num-threads 64 --max-tokens 2048 ``` The command evaluates all 57 subjects. Historical results below use the removed 10-subject harness and are not measurements of the sgl-eval command above. * **Test Results**: * DeepSeek-V3.2-Exp ``` subject: abstract_algebra, #q:100, acc: 0.780 subject: anatomy, #q:135, acc: 0.874 subject: astronomy, #q:152, acc: 0.961 subject: business_ethics, #q:100, acc: 0.860 subject: clinical_knowledge, #q:265, acc: 0.925 subject: college_biology, #q:144, acc: 0.972 subject: college_chemistry, #q:100, acc: 0.660 subject: college_computer_science, #q:100, acc: 0.880 subject: college_mathematics, #q:100, acc: 0.840 subject: college_medicine, #q:173, acc: 0.879 Total latency: 7.961 Average accuracy: 0.879 ``` #### 5.2.3 GPQA-Diamond Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run gpqa --base-url http://127.0.0.1:30000/v1 --num-examples 198 --max-tokens 128000 --n-repeats 8 --thinking ``` * **Historical Test Results** (model: `deepseek-ai/DeepSeek-V3.2-Exp`, 8×B200; measured with the earlier `sglang.test.run_eval` GPQA command): * Default (`temperature=0`): mean **0.797** over 8 runs — closely matches the official GPQA-Diamond score of **79.9** for DeepSeek-V3.2-Exp reported in its [model card](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp) * With `temperature=1.0, top_p=0.95` (as recommended by DeepSeek): ```shell Command theme={null} sgl-eval run gpqa --base-url http://127.0.0.1:30000/v1 --num-examples 198 --max-tokens 128000 --n-repeats 8 --top-p 0.95 --temperature 1.0 --thinking ``` ``` Repeat: 8, mean: 0.840 Scores: ['0.848', '0.808', '0.848', '0.838', '0.879', '0.813', '0.838', '0.848'] ``` #### 5.2.4 AIME 2025 Benchmark Results on AIME 2025 (8×B200), evaluated with [sgl-eval](https://github.com/sgl-project/sgl-eval):
Model pass\@1 avg-of-4 majority\@4 pass\@4
DeepSeek-V3.2-Exp 87.50% ± 1.67% 90.00% 90.00%
DeepSeek-V3.2 92.50% ± 1.67% 94.71% 96.67%
DeepSeek-V3.2-Speciale 95.00% ± 1.92% 95.83% 100.00%
**Reproduction.** Install [sgl-eval](https://github.com/sgl-project/sgl-eval), launch the server with the tool-call and reasoning parsers, then run `sgl-eval run`. The AIME 2025 dataset ships with sgl-eval, and thinking is on by default for it: ```bash Command theme={null} pip install sgl-eval sgl-eval run aime25 \ --base-url http://localhost:30000/v1 \ --model deepseek-ai/DeepSeek-V3.2-Exp \ --n-repeats 4 \ --temperature 1.0 \ --top-p 0.95 \ --max-tokens 64000 # Use --max-tokens 120000 for the DeepSeek-V3.2-Speciale model ``` ### 5.3 Speed Benchmark on Hopper **Test Environment:** * Hardware: NVIDIA H800 GPU (16x) * Model: DeepSeek-V3.2 * Tensor Parallelism: 16 * sglang version: 0.5.9 #### 5.3.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} export SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS=32 export SGLANG_SET_CPU_AFFINITY=1 # Test workload ISL/OSL=1k/1k, raw tap : 4948.16 toks/sec, MAX ITL 5970 # dp 2 : 5019.54 toks/sec, MAX ITL 7233 # dp 4 : 4942.82 toks/sec, MAX ITL 35654 # dp 2 + mtp : 6842.51 toks/sec, MAX ITL 3081 sglang_args=$(echo serve \ --model-path $MAPPED_MODEL_PATH \ --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 16 \ --attn-dp-size 2 --page-size 64 \ --trust-remote-code --host "0.0.0.0" --port 30000 \ --log-requests \ --context-length 65536 --max-running-requests 128 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \ --allow-auto-truncate --enable-metrics \ --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \ --served-model-name DeepSeek-V3.2-Opt-dp2-mtp ) sglang_args=($sglang_args) sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host $MASTER_ADDR \ --port 30000 \ --model deepseek-ai/DeepSeek-V3.2 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 64.0 Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 48.96 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4217 Request throughput (req/s): 0.20 Input token throughput (tok/s): 124.62 Output token throughput (tok/s): 86.20 Peak output token throughput (tok/s): 113.00 Peak concurrent requests: 2 Total token throughput (tok/s): 210.81 Concurrency: 1.00 Accept length: 3.27 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4893.12 Median E2E Latency (ms): 3742.47 P90 E2E Latency (ms): 8877.37 P99 E2E Latency (ms): 10769.85 ---------------Time to First Token---------------- Mean TTFT (ms): 199.88 Median TTFT (ms): 176.15 P99 TTFT (ms): 272.49 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.99 Median TPOT (ms): 10.88 P99 TPOT (ms): 13.93 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.15 Median ITL (ms): 8.86 P95 ITL (ms): 17.29 P99 ITL (ms): 33.71 Max ITL (ms): 36.84 ================================================== ``` #### 5.3.2 Throughput-Sensitive Benchmark We simply use the same deployment method and vary the throughput by maximizing concurrencies: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host $MASTER_ADDR \ --port 30000 \ --model deepseek-ai/DeepSeek-V3.2 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 2048 \ --max-concurrency 1024 # see picture below why we use 1024 for concurrency, hence num prompts 2048 ``` DeepSeek 3.2 can steadily support concurrency up to `1024` and when concurrency is greater than `128`, the TTFT increase sharply: ![DeepSeek V3.2 Concurrency ISL/OSL=1024/128](https://github.com/user-attachments/assets/d5c9c9fb-44f3-4793-a0fd-f8fa954546f5) Performance record: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 64.0 Max request concurrency: 1024 Successful requests: 2048 Benchmark duration (s): 408.09 Total input tokens: 1048992 Total input text tokens: 1048992 Total generated tokens: 1032734 Total generated tokens (retokenized): 1031817 Request throughput (req/s): 5.02 Input token throughput (tok/s): 2570.50 Output token throughput (tok/s): 2530.66 Peak output token throughput (tok/s): 5092.00 Peak concurrent requests: 1035 Total token throughput (tok/s): 5101.16 Concurrency: 763.41 Accept length: 3.26 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 152117.70 Median E2E Latency (ms): 181704.84 P90 E2E Latency (ms): 215924.77 P99 E2E Latency (ms): 231679.59 ---------------Time to First Token---------------- Mean TTFT (ms): 127729.28 Median TTFT (ms): 170098.94 P99 TTFT (ms): 185705.73 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 49.18 Median TPOT (ms): 48.48 P99 TPOT (ms): 77.24 ---------------Inter-Token Latency---------------- Mean ITL (ms): 48.46 Median ITL (ms): 52.11 P95 ITL (ms): 110.26 P99 ITL (ms): 200.63 Max ITL (ms): 2666.37 ================================================== ``` By adding `--random-range-ratio 1`, we could get even higher statistical numbers: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 64.0 Max request concurrency: 1024 Successful requests: 2048 Benchmark duration (s): 612.87 Total input tokens: 2097152 Total input text tokens: 2097152 Total generated tokens: 2097152 Total generated tokens (retokenized): 2096201 Request throughput (req/s): 3.34 Input token throughput (tok/s): 3421.84 Output token throughput (tok/s): 3421.84 Peak output token throughput (tok/s): 9077.00 Peak concurrent requests: 1039 Total token throughput (tok/s): 6843.68 Concurrency: 772.66 Accept length: 3.26 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 231222.27 Median E2E Latency (ms): 289846.24 P90 E2E Latency (ms): 314480.41 P99 E2E Latency (ms): 320392.27 ---------------Time to First Token---------------- Mean TTFT (ms): 194081.02 Median TTFT (ms): 252945.22 P99 TTFT (ms): 279637.50 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 36.31 Median TPOT (ms): 36.73 P99 TPOT (ms): 46.33 ---------------Inter-Token Latency---------------- Mean ITL (ms): 36.31 Median ITL (ms): 23.18 P95 ITL (ms): 96.79 P99 ITL (ms): 135.81 Max ITL (ms): 3121.00 ================================================== ``` # DeepSeek-V4 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4 Deploy DeepSeek-V4 with SGLang — verified launch commands, benchmarks, and tuning for Flash Official (0731), Flash, Flash Vision (Exp), Pro, and Pro Official (0813). ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces): **NVIDIA GPUs** A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000). The one exception is **Flash Vision (Exp)**, whose support has not shipped in a release yet: its cells use the preview image `lmsysorg/sglang:dev-dsv4-flash-vision` (the command panel picks it automatically — see the [Flash Vision notes](#vision-note)). **DGX Spark** is the other exception: its three cells (Flash Official FP4, Flash Official NVFP4, Flash Vision FP4) use the DGX Spark–only preview image `lmsysorg/sglang:dev-v4f-2dgx-v2` (the command panel picks it automatically — see the [DGX Spark notes](#spark-note)); do not use that image on any other hardware. ```bash Command theme={null} docker pull lmsysorg/sglang:latest docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:latest \ sglang serve ``` **AMD GPUs (ROCm)** AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version. For example: * **MI355X** → `lmsysorg/sglang-rocm:v0.5.21-rocm720-mi35x-20261007` * **MI300X** → `lmsysorg/sglang-rocm:v0.5.21-rocm720-mi30x-20261007` ```bash Command theme={null} docker pull lmsysorg/sglang-rocm:v0.5.21-rocm720-{mi35x,mi30x}-20261007 docker run \ --device=/dev/kfd --device=/dev/dri \ --group-add video \ --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \ --shm-size 32g --ipc=host \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ lmsysorg/sglang-rocm:v0.5.21-rocm720-{mi35x,mi30x}-20261007 \ sglang serve ``` For **MI355X → Pro / Pro Official → FP4 → Multi-Nodes**, the panel generates a **1P1D deployment**: one prefill node and one decode node, each running an independent worker. Select **Router**, **Prefill**, or **Decode** to copy its command. Start the [UMBP tier](#3-8-unified-cache-external-linker-umbp) on the prefill node, then both workers, then the router. Send client requests to the router on port `8000`. The MI355X multi-node strategies follow the [InferenceX AgentX submission](https://github.com/SemiAnalysisAI/InferenceX/pull/3664): **Low-Latency**, **Balanced**, and **High-Throughput** use fixed concurrency presets of **16**, **48**, and **256**, respectively. **Pro** uses EAGLE/MTP; **Pro Official (0813)** uses DSpark, matching the InferenceX checkpoint. Both enable UMBP on prefill by default. The Playground inherits the selected worker, including its speculative decoder and cache settings. Run these recipes' **Python** commands inside the ROCm image shown in the **Docker** tab, where Aiter is already enabled. Add explicit parser and network-device settings through the Playground's **Parsers** and **PD Disagg → IB Device** controls. Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points: * **Low-Latency** — fastest reply for a single user. Pick for chat. * **Balanced** — good speed with several users at once. Use for typical multi-user serving. * **High-Throughput** — most tokens per second across many users. Best for batch jobs. The **Flash Vision (Exp)** variant runs on a dedicated preview Docker image, `lmsysorg/sglang:dev-dsv4-flash-vision` — its support ([sgl-project/sglang#37253](https://github.com/sgl-project/sglang/pull/37253)) has not shipped in a release yet. The command panel's Docker mode emits that image automatically for Flash Vision cells; see the [Flash Vision notes](#vision-note). For a runnable end-to-end example, see the [DeepSeek-V4-Flash demo notebook](https://github.com/sgl-project/sglang/blob/main/docs/demo/deepseek_v4_flash.ipynb).

Panel controls (top of the command box):

  • Python / Docker — bare sglang serve … for an existing SGLang env, or a docker run … sglang serve … wrap against the per-hardware image from the Install SGLang panel above.
  • ⧉ Copy — copies the current command (with whichever framing is active) to your clipboard.
  • \$ cURL — a sample request against localhost:30000 to confirm the server is up.
  • ⚙ Env — edits the placeholders (HOST\_IP, PORT, HF\_TOKEN, NODE\_RANK, NODE0\_IP) the command and cURL share. Persists in localStorage across cookbooks.
  • Verified / Not Verified badge — green when the (hw, variant, quant, strategy, nodes) combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.
## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change. The knobs come in two flavors: * **Built-in SGLang features** — parallelism overrides (TP / CP / DP-Attention — DP-Attention's value is the DP degree, with `off` to disable), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, HiCache tiers, and HiSparse hierarchical sparse attention (decode-role only — the card appears once PD-Disagg mode is set to decode). * **DeepSeek-V4 specific features** — MegaMoE W4A8 / W4A4 fused kernel (Blackwell only; Hopper SM90 uses a separate all-FP8 MegaMoE path — see Configuration Tips below). Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.

Panel controls reuse Python / Docker · ⧉ Copy · \$ cURL · ⚙ Env from the Deploy panel, plus one extra:

  • Submit ↗ — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says Not Verified; click it once you've actually run the command on your hardware and confirmed it works.
## Agentic Long-Context Deployment Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order. * **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks). * **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`. * **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image. ## 1. Model Introduction **DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. The 0731 Flash and 0813 Pro refreshes add checkpoints with a bundled DSpark draft head, and the experimental Flash Vision checkpoint builds image understanding on top of the 0731 Flash base:
Variant Total params Active (MoE) Use
DeepSeek-V4-Flash 284B 13B single-node serving on B200 / B300 / GB200 / GB300 / H200 (TP=4); RTX PRO 6000 (TP=2); H100 (TP=8)
DeepSeek-V4-Flash-0731 304 13B Flash Official (0731), with a bundled DSpark draft head; verified on 8×B200, 4×GB300, and 4×H200
DeepSeek-V4-Flash-Vision-Exp 305B 13B Flash Vision (Exp) — experimental multimodal (image-text-to-text): the 0731 Flash base + vision encoder & aligner; verified on 4×B200 (TP=4), requires the preview build
DeepSeek-V4-Pro 1.6T 49B high-capacity: B200 / B300 (TP=8) · GB300 (TP=4) · H200 FP4 (TP=8) · GB200 (2-node, TP=8) · H200 FP8 (2-node, TP=16) · H100 (2-node, TP=16)
DeepSeek-V4-Pro-0813 1.65T 49B Pro Official (0813), with a bundled DSpark draft head; verified on 4×GB300 (TP=4) · B200 / B300 / H200 FP4 (TP=8) · GB200 (2-node, TP=8) · H100 (2-node, TP=16) · MI355X
The Instruct checkpoints ship as **FP4 MoE experts + FP8 attention / dense** (one mixed-precision checkpoint covers every FP4-capable GPU). Matching `*-Base` repos ship pure FP8 mixed and are for further pre-training only — not for chat or tool calling. **Highlights:** hybrid CSA + HCA attention (\~27% inference FLOPs / \~10% KV cache vs DSv3.2 at 1M context), manifold-constrained hyper-connections (mHC), Muon optimizer, **1M-token context** (32T+ pre-training tokens), three reasoning modes (*Non-think* / *Think High* / *Think Max* — use ≥ 384K context for Think Max), and a dedicated `encoding_dsv4.encode_messages` Python encoder + DSML tool-call grammar. **Recommended generation:** `temperature=1.0`, `top_p=1.0`. **Resources:** HuggingFace · [Flash Official (0731)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) · [Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) · [Flash Vision (Exp)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) · [Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) · [Pro Official (0813)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813) · ModelScope · [Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro). ## 2. Configuration Tips **Concurrency & DeepEP dispatch buffer** Must hold: `max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`. Violating it blows DeepEP's dispatch buffer at steady-state load (`deep_ep.cpp:1105`). When tuning, move `--cuda-graph-max-bs-decode`, `--max-running-requests`, and the env together. The generator currently picks values on the **conservative** side (mirroring an internal stress-test matrix). They run safely out of the box but likely leave throughput on the table — please tune them up toward your actual workload's peak concurrency and report findings back so the defaults can be revised. **Speculative decoding** The original Flash and Pro recipes use EAGLE. Flash Official (0731) and Pro Official (0813) use the bundled DSpark draft head; see [DSpark](#3-4-dspark-speculative-decoding) for its launch and tuning notes. Do not use EAGLE on the checkpoints that bundle a DSpark head. On 0813, `--speculative-algorithm EAGLE` starts and serves without any error, but the draft head it binds accepts nothing — every decode batch logs `accept len: 1.00, accept rate: 0.00`, so you pay the draft cost for zero speedup. Output stays correct, which is what makes it easy to miss. Switch to `--speculative-algorithm DSPARK`; the startup log then reports `Draft checkpoint bundles a DSpark head`. For the original Flash and Pro checkpoints: * `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1. * `balanced`: steps=1, draft-tokens=2 → gentler MTP, reduces throughput hit at higher batch. * `high-throughput`: MTP disabled — at saturation the verify step costs more than it saves. * MTP runs on the v2 speculative path. **DGX Spark (2x GB10): Flash Official FP4 / NVFP4, Flash Vision FP4** The **DGX Spark** row has three recipes, all **Balanced · Multi-Nodes**: **Flash Official (0731) · FP4**, **Flash Official (0731) · NVFP4**, and **Flash Vision (Exp) · FP4**. None of these checkpoints fits one 128GB GB10, so every recipe runs TP=2 across two DGX Sparks connected over ConnectX-7 (RoCE). Every other DGX Spark combination is greyed out on purpose. * **Docker image** — all three cells use `lmsysorg/sglang:dev-v4f-2dgx-v2`, a preview build made **only for DGX Spark** (branch `b12x-vision` @ `452239a74f`): it bakes in the SM12x `b12x` MoE (W4A8) and compressed-MLA attention kernels ([#34878](https://github.com/sgl-project/sglang/pull/34878), [#35899](https://github.com/sgl-project/sglang/pull/35899), [#34018](https://github.com/sgl-project/sglang/pull/34018)), the Flash Vision model support ([#37253](https://github.com/sgl-project/sglang/pull/37253)), the b12x image-prefill fix that lets Flash Vision serve images on SM12x, the NVFP4 MTP-layer dispatch fix, and the CuTeDSL and NCCL pins the GB10 pair needs. Do not use it on other hardware, and use the panel's Docker mode — the bare Python command needs the `b12x` kernel package this image ships. * **Run the same command on both Sparks** with `--node-rank 0` / `--node-rank 1` and `--dist-init-addr` pointing at node 0 over the ConnectX-7 link. The `docker run` flags the panel emits (`--network host --ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`) are what let NCCL use RDMA; without them NCCL silently falls back to TCP and decode slows by roughly 40%. * **Env knobs** in the cells are part of the recipe: `SGLANG_SM120_FLASHMLA_BACKEND=b12x` selects the b12x attention path, `SGLANG_B12X_MAX_TOKENS` must equal `--chunked-prefill-size`, and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` avoids unified-memory fragmentation OOMs on GB10. * **NVFP4 (`nvidia/DeepSeek-V4-Flash-0731-NVFP4`)** — only the routed experts are NVFP4; attention, shared experts and the DSpark MTP layer stay in the checkpoint's native formats. On SM12x that means three extra flags: `--moe-runner-backend flashinfer_cutlass` (b12x's MoE is MXFP4-only and trtllm-gen kernels are sm100-only), `--speculative-moe-runner-backend b12x` (the DSpark draft's MTP experts are MXFP4 and run on b12x), and `--disable-shared-experts-fusion` (HashTopK rejects fused shared experts under the cutlass runner). Throughput and DSpark acceptance match the FP4 cell within noise. * **Flash Vision** — images are served natively on the b12x recipe with the same flags as Flash Official (send `image_url` content on `/v1/chat/completions`, see [Vision](#3-5-vision-image-inputs)); text-only requests work unchanged. Expect roughly 15–20% lower text throughput than Flash Official on this checkpoint — its bundled DSpark head accepts fewer drafts (\~3.2 vs \~3.9) — with text accuracy intact. **DeepSeek-V4-Flash-Vision-Exp (Experimental)** [`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) is DeepSeek's first experimental multimodal V4 checkpoint: the 0731 Flash base plus a vision encoder and aligner, served through the same `sglang serve` flow with OpenAI-style `image_url` inputs (see [Vision](#3-5-vision-image-inputs) below). Select the **Flash Vision** variant in the Deploy panel for its recipes. * **Preview build required** — support lands via [sgl-project/sglang#37253](https://github.com/sgl-project/sglang/pull/37253) and has not shipped in a release. Docker mode on the Flash Vision cells already emits the preview image `lmsysorg/sglang:dev-dsv4-flash-vision`; for a Python environment, install SGLang from that PR's branch. The DGX Spark Flash Vision cell is the exception: it uses the DGX Spark image `lmsysorg/sglang:dev-v4f-2dgx-v2` (see the [DGX Spark notes](#spark-note)). * **Verified matrix** — MMMU-Pro via sgl-eval at `temperature 1.0`, `top-p 0.95`, `--reasoning-effort max`. * **Engine auto-configuration** — the engine picks the `flashinfer_mxfp4` MoE runner and auto-disables shared-experts fusion for this checkpoint (its HashTopK routing rejects fused shared experts); don't pass `--enforce-shared-experts-fusion`. * **Chunked prefill & radix cache stay enabled** — the scheduler keeps image spans consistent automatically: chunked-prefill truncation points are span-aligned (an image span always prefills within a single extend, overshooting the chunk budget by at most one span), and a radix-cache prefix match ending deep inside an image span is re-issued from the span start. * **Speculative decoding** — the checkpoint bundles a DSpark head, and the low-latency recipes enable it with `--speculative-algorithm DSPARK` (verified with image batches on B200 via the MMMU-Pro round; the other hardware rows are pending verification). The balanced and high-throughput recipes run target-only: they use DP Attention, which DSpark is incompatible with on current releases. As on the 0731/0813 checkpoints, do not pass the EAGLE flags. **Shared experts fusion (Blackwell, flashinfer\_mxfp4)** On the Blackwell fp4 recipes (`--moe-runner-backend flashinfer_mxfp4`), the shared expert runs as a separate FP8 MLP on an alternate stream by default. Adding: ```bash Command theme={null} --enforce-shared-experts-fusion ``` routes it as one extra MXFP4 expert through the same trtllm-gen MoE kernel, so the whole MoE runs on a single stream (\~4 fewer kernel launches and 2 fewer stream syncs per MoE layer). The shared expert is requantized from FP8 to MXFP4 at load time. Measured on GB200 tp4: gsm8k and AIME25 accuracy on par with the unfused baseline; Mean TTFT -13% to -21% and P99 ITL -15% to -53% at QPS 1-8 with neutral throughput. Only for deployments without expert parallelism (e.g. the single-node low-latency recipes): with `moe_ep_size > 1` the flag is rejected at startup, unless the DeepEP/MegaMOE per-rank shared-slot path is in use. Not applicable to [Flash Vision (Exp)](#vision-note) — the engine auto-disables the fusion on that checkpoint. **Compressed attention state dtype** DeepSeek-V4 uses hybrid compressed attention for long-context efficiency. `SGLANG_DSV4_COMPRESS_STATE_DTYPE` controls the dtype of the C4 / C128 compressed attention state pools. Supported values are `float32` / `fp32` (default: `float32`) and `bfloat16` / `bf16`. For BF16 on the offline compression path: ```bash Command theme={null} SGLANG_DSV4_COMPRESS_STATE_DTYPE=bf16 \ sglang serve \ --model-path deepseek-ai/DeepSeek-V4-Flash \ ``` This BF16 setting applies only to the compressed attention state pools and reduces the GPU memory footprint of each compressed-state slot. It does not change model weight precision or the main KV cache dtype. With automatic pool sizing and no explicit capacity cap, the same memory budget holds more slots, and the startup log shows larger `c4_state` and `c128_state` pool sizes. Keep the default `float32` setting for the most conservative behavior. **EPLB + Waterfill (Experimental)** For recorded/static EPLB reproduction, first record an expert-distribution file by following [Capture expert selection distribution in MoE models](../../../docs/basic_usage/native_api.mdx#capture-expert-selection-distribution-in-moe-models). For reproduction runs, use the generated `expert_distribution_recorder_*.pt` as the initial expert location. **Please checkout to latest main branch for this feature.** For non-PD reproduction, use: ```bash Command theme={null} --moe-a2a-backend deepep \ --deepep-mode auto \ --init-expert-location /path/to/expert_distribution_recorder_*.pt \ --enable-waterfill ``` For PD-Disagg reproduction, use `normal` mode on the prefill server and `low_latency` mode on the decode server. Add the same `--init-expert-location` flag to both commands: ```bash Command theme={null} # prefill --moe-a2a-backend deepep \ --deepep-mode normal \ --init-expert-location /path/to/expert_distribution_recorder_*.pt \ --enable-waterfill # decode --moe-a2a-backend deepep \ --deepep-mode low_latency \ --init-expert-location /path/to/expert_distribution_recorder_*.pt \ --enable-waterfill ``` You can also add `--ep-num-redundant-experts` and `--eplb-algorithm` to customize EPLB placement. Waterfill also supports MegaMOE. Use `--moe-a2a-backend megamoe --enable-waterfill` to keep the MegaMOE backend while applying Waterfill to the fused shared expert slot. **FP4 Indexer (Experimental)** DeepSeek-V4 uses the default indexer path unless `--enable-deepseek-v4-fp4-indexer` is set. Enable this flag to use the experimental FP4 C4 indexer. This path is intended for decode-heavy long-context workloads where reducing indexer cache bandwidth is beneficial. On **NVIDIA (SM100 / SM120)**, pair the flag with DeepGEMM FP4 indexer support and the FlashInfer MXFP4 MoE runner: ```bash Command theme={null} # Please use the latest main branch for this feature. sglang serve \ --model-path deepseek-ai/DeepSeek-V4-Flash \ --tp 4 \ --moe-runner-backend flashinfer_mxfp4 \ --enable-deepseek-v4-fp4-indexer ``` On **AMD MI355X (gfx95)**, the AITER FP4 indexer kernels are available on ROCm. Keep the standard MI355 recipe and add only the indexer flag. Do not need to pass `--moe-runner-backend flashinfer_mxfp4`: ```bash Command theme={null} # Please use the latest ROCm image for this feature. sglang serve \ --model-path deepseek-ai/DeepSeek-V4-Pro \ --tp 8 \ --attention-backend dsv4 \ --enable-deepseek-v4-fp4-indexer \ ``` **NVFP4 Hybrid Checkpoints** The [`nvidia/DeepSeek-V4-Pro-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Pro-NVFP4) and [`nvidia/DeepSeek-V4-Flash-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Flash-NVFP4) checkpoints quantize MoE experts to **NVFP4** while keeping attention and dense layers in **FP8**. The official releases have matching NVFP4 checkpoints at [`nvidia/DeepSeek-V4-Flash-0731-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Flash-0731-NVFP4) and [`nvidia/DeepSeek-V4-Pro-0813-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Pro-0813-NVFP4). All of them require `--moe-runner-backend flashinfer_trtllm_routed` which will be automatically selected if not provided. ```bash Command theme={null} sglang serve \ --model-path nvidia/DeepSeek-V4-Pro-NVFP4 \ --tp 8 ``` or ```bash Command theme={null} sglang serve \ --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \ --tp 8 ``` Requires Blackwell (SM100+). The MTP layer in this checkpoint stays MXFP4-packed and is routed through the `Mxfp4FlashinferTrtllmMoEMethod` path automatically. The official (0731 / 0813) NVFP4 checkpoints preserve the bundled DSpark draft head, so their low-latency recipes use `--speculative-algorithm DSPARK` instead of the EAGLE/MTP shape flags — same as the corresponding original-precision official checkpoints. **Hopper (H100 / H200) note** Two options are available for running DeepSeek-V4 on Hopper: * **Original FP4 checkpoints** — run the MoE experts with W4A16 kernels (Marlin or the FlashInfer SM90 CUTLASS runner) as the command generator picks for Hopper cells. With FlashInfer >= 0.6.18 you can instead select the **W4A8** path — MXFP4 weights with FP8 activations via FlashInfer's Humming kernels — by adding `--flashinfer-mxfp4-moe-precision fp8` to `--moe-runner-backend flashinfer_mxfp4`; the low-latency Hopper cells now generate this form. Both work on H100 and H200; FP4 is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16). * **Converted FP8 checkpoints** (H100 and H200 only) — pre-repackaged FP8 weights at [`sgl-project/DeepSeek-V4-Flash-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Flash-FP8) and [`sgl-project/DeepSeek-V4-Pro-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Pro-FP8) unlock DP-attention + DeepEP and richer parallelism (e.g. Pro TP=16 across 2 nodes). On these FP8 checkpoints you can additionally enable the all-FP8 **MegaMoE** path on SM90 for higher long-context / large-decode throughput — see the **SM90 (Hopper) FP8 MegaMoE** note in Configuration Tips below. PD-Disagg recipes on H200 may require `docker run --privileged --ulimit memlock=-1` (or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) so mooncake can discover the IB HCAs; without IB exposure mooncake silently falls back to TCP, which can lead to garbled KV transfer on large checkpoints. **RTX PRO 6000 (SM120 / Blackwell Desktop) note** RTX PRO 6000 (96 GB) runs **Flash only** with the FlashInfer MXFP4 MoE runner. V4-Pro doesn't fit on 8× 96 GB; the Deploy panel greys out unsupported recipes. HiCache and MegaMoE are **not** supported on RTX PRO 6000. **AMD (MI300X / MI355X) note** * **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`. * **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node. * **TP / DP setting (MI355X)** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP** (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set `--attn-dp-size 8 --enable-dp-attention-local-control-broadcast --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10`. * **MTP** — speculative decoding on the original Flash / Pro checkpoints, which bundle an MTP head; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. Pro Official (0813) has no MTP head, so these flags do not apply there and the Playground does not offer them. * **DSpark (MI355X Pro Official 0813)** — the 0813 checkpoint bundles the DSpark draft head, and DSpark is the only speculative algorithm it supports. Enable `--speculative-algorithm DSPARK` (low-latency) and see the [MI355X agentic recipe](#3-6-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark) for the long-context path. DSpark also runs under **PD disaggregation** — see the Pro Official pairs in [§3.7](#3-7-pd-disaggregation-on-mi355x-mori-io). * **Kernels** — uses the Unified KV attention and the flydsl MoE. * **FP4 indexer (MI355X)** — FP4 C4 indexer is supported via `--enable-deepseek-v4-fp4-indexer` on top of the standard ROCm recipe. **MoRI EP (AMD expert parallelism)** On AMD, expert parallelism uses the **MoRI** all-to-all backend (`--moe-a2a-backend mori`), not DeepEP. Add the flags below on top of the verified recipe when sharding experts across GPUs; set `--ep-size` to the EP degree (typically the GPU count on one node). Two optional env vars improve MoRI throughput (both off by default): ```bash Command theme={null} export SGLANG_MORI_DISPATCH_DTYPE=mxfp8 export SGLANG_MORI_RECV_BOUND=1 sglang serve \ --model-path {MODEL_PATH} \ --tp 8 --attn-dp-size 8 \ --ep-size 8 --moe-a2a-backend mori --deepep-mode normal \ ``` * **FP4**: enable both env vars. * **FP8**: use `SGLANG_MORI_RECV_BOUND=1` only; omit `SGLANG_MORI_DISPATCH_DTYPE=mxfp8`. **MegaMoE** MegaMoE fuses expert dispatch + GEMM into a single kernel for higher throughput on MoE layers. To enable it, use the **MegaMoE** chip in the Playground below — the playground will swap `--moe-a2a-backend deepep` for `--moe-a2a-backend megamoe` and add the relevant launch settings automatically. Two variants are exposed: * **W4A8** — default MegaMoE kernel (FP4 weights, FP8 activations). * **W4A4** — adds `--enable-w4a4-mxfp4-megamoe` to run the custom W4A4 kernel (FP4 activations). The flag configures the required DeepGEMM settings. Higher throughput with negligible accuracy drop (\~89.5 GPQA on Pro). Notes: * The W4A8 / W4A4 variants above are **Blackwell-only** (B200 / B300 / GB200 / GB300). On **Hopper (SM90, H100 / H200)** use the all-FP8 MegaMoE path described below instead. * MegaMoE is **only wired into the `high-throughput` recipe** on Blackwell (per [sgl-project/sglang#26451](https://github.com/sgl-project/sglang/pull/26451)). The chip is hidden on `low-latency` and `balanced` — switch to `high-throughput` to expose it. * When running MegaMoE, don't set `--moe-runner-backend` manually. * Adjust `SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` based on your workload and memory usage. Setting higher number of tokens for MegaMoE requires more HBM space (recommended: 8320 for high-throughput). **SM90 (Hopper) FP8 MegaMoE (Experimental)** On SM90 (Hopper, H100 / H200), the all-FP8 MegaMoE path routes MoE through the DeepGEMM `mega_moe` runner for higher long-context / large-decode throughput on the FP8 checkpoints. Unlike the Blackwell W4A8 / W4A4 variants above, experts stay in **FP8** — keep `SGLANG_DSV4_FP4_EXPERTS=0`. It requires a `sgl-deep-gemm` build with SM90 FP8 MegaMoE support. **Please use the latest image for this feature.** Enable the MegaMoE path with `--moe-a2a-backend megamoe` ```bash Command theme={null} SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 \ SGLANG_DSV4_FP4_EXPERTS=0 \ sglang serve \ --model-path sgl-project/DeepSeek-V4-Flash-FP8 \ --tp 8 \ --moe-a2a-backend megamoe \ --chunked-prefill-size 4096 ``` `SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` caps the number of tokens the MegaMoE path processes per rank (i.e. per GPU); the MegaMoE path is only used for batches at or below this cap. The right value depends on your parallelism / token-split scheme, and larger values reserve more HBM. **GB300 PD-Disagg cross-pod MNNVL** On some GB300 clusters with cross-pod KV transfer over NVLink, mooncake may fail with `nvlink_transport.cpp:497 Requested address ... not found!`. If this happens, prepend `MC_FORCE_MNNVL=1 NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1` to both prefill and decode `sglang serve` commands. ## 3. Advanced Usage ### 3.1 Reasoning Enable the `deepseek-v4` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, extra_body={"chat_template_kwargs": {"thinking": True}}, stream=True, ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if not chunk.choices: continue delta = chunk.choices[0].delta if getattr(delta, "reasoning_content", None): if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` ```text Output theme={null} We are asked: "What is 15% of 240?" This is a simple percentage problem. I need to provide a step-by-step solution. The user wants the solution explained step by step. I'll calculate 15% of 240: 0.15 * 240 = 36. I'll break it down into steps: understand what percent means, convert percentage to decimal or fraction, then multiply. I'll present the answer clearly.To find 15% of 240, follow these steps: **Step 1: Understand the meaning of percent** "Percent" means "per hundred," so 15% means 15 out of every100, or \( \frac{15}{100} \). **Step2: Convert the percentage to a decimal or fraction** \( 15\% = \frac{15}{100} = 0.15 \) **Step3: Multiply by the given number** Multiply the decimal form by 240: \( 0.15 \times 240 \) **Step4: Perform the multiplication** \( 0.15 \times 240 = 36 \) **Answer:** 15% of 240 is **36**. ``` ### 3.2 Tool Calling Enable the `deepseekv4` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, extra_body={"chat_template_kwargs": {"thinking": True}}, stream=True, ) thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if not chunk.choices: continue delta = chunk.choices[0].delta if getattr(delta, "reasoning_content", None): if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if getattr(delta, "tool_calls", None): if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = {"name": None, "arguments": ""} if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]["name"] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]["arguments"] += tool_call.function.arguments if delta.content: print(delta.content, end="", flush=True) for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` ```text Output theme={null} The user wants to know the weather in Beijing. I'll use the get_weather function with Beijing as the location. I don't need to specify a unit, so I'll just use the default. <｜DSML｜tool_calls> <｜DSML｜invoke name="get_weather"> <｜DSML｜parameter name="location" string="true">Beijing ``` ### 3.3 HiCache (Hierarchical KV Caching) HiCache enables multi-tier KV cache offloading (GPU → CPU → Storage), significantly expanding effective context capacity for long-context and multi-turn scenarios. Combined with UnifiedRadixTree, it provides intelligent prefix caching across all tiers. To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**: * **L2 (GPU + CPU)** — leave Storage on `auto` (default). Cold KV pages spill to CPU pinned memory only. * **L3 (GPU + CPU + Storage)** — pick a Storage backend (`file` / `mooncake` / `hf3fs` / `nixl`); the Playground emits the canonical `page_first_direct` mem-layout + `direct` IO backend + `wait_complete` prefetch policy, matching the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices). For AMD devices, * **L2 (GPU + CPU)** — leave Storage on `auto` (default). Cold KV pages spill to CPU pinned memory only. Use `direct` IO backend + `page_first_direct` or `layer-first` mem-layout. * **L3 (GPU + CPU + Storage)** — pick a Storage backend (`file`); the Playground emits the canonical `page_first_direct` mem-layout + `direct` IO backend + `wait_complete` prefetch policy, matching the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices). The Write policy knob defaults to `write_through` (the upstream default); switch to `write_back` / `write_through_selective` to trade durability for write speed when the storage tier is slow. For more details, see the [HiCache documentation](../../../docs/advanced_features/hicache). For the alternative that skips the host tier entirely, see [§3.8 UMBP](#3-8-unified-cache-external-linker-umbp) — it is a sibling of HiCache, not a storage backend for it, and the two cannot be enabled together. ### 3.4 DSpark (Speculative Decoding) Flash Official (0731) and Pro Official (0813) bundle a DSpark draft head in `deepseek-ai/DeepSeek-V4-Flash-0731` and `deepseek-ai/DeepSeek-V4-Pro-0813`. The target and draft weights therefore come from the same checkpoint: enable DSpark with `--speculative-algorithm DSPARK` and do not set a separate `--speculative-draft-model-path`. The experimental [Flash Vision checkpoint](#vision-note) also bundles a DSpark head, enabled the same way: the Flash Vision low-latency recipes ship with `--speculative-algorithm DSPARK` (verified with image batches on B200 via the MMMU-Pro round; other hardware rows pending). The balanced and high-throughput Flash Vision recipes stay target-only because they run DP Attention. Unlike the EAGLE recipes for the original Flash and Pro checkpoints, this recipe omits `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. SGLang reads the DSpark shape from the checkpoint. The Pro Official (0813) low-latency speed numbers in the Deploy panel were measured with `SGLANG_SIMULATE_ACC_LEN=4`, which pins the DSpark accept length at exactly 4.00. The recipe as shipped earns **4.678** on the same engine, so those rows read slightly conservative. The GSM8K figure for that cell is from the shipped command. The verified 4×GB300 FP4 low-latency command is: ```bash Command theme={null} sglang serve \ --trust-remote-code \ --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \ --tp 4 \ --moe-runner-backend flashinfer_mxfp4 \ --speculative-algorithm DSPARK \ --mem-fraction-static 0.90 \ --chunked-prefill-size 4096 \ --swa-full-tokens-ratio 0.1 \ --host 0.0.0.0 \ --port 30000 ``` Keep `--mem-fraction-static 0.90` on this topology to leave enough headroom for the batch-256 verify graph. The first cold start can take 10–15 minutes while FlashInfer autotunes and SGLang captures the draft and verify graphs; later starts reuse the cache. This path is verified end-to-end on 4×GB300 with SGLang v0.5.16. **Tune proposed draft tokens.** `--speculative-dspark-block-size N` asks DSpark to propose `N` tokens per step; the target verifies a window of `N + 1`. If the flag is omitted, SGLang reads the value from the checkpoint. Both the 0731 and 0813 checkpoints resolve to five proposed tokens (the startup log reports `gamma=5, verify_num_draft_tokens=6`), which is the verified default. Use the **DSpark Proposed Draft Tokens** slider in the [Playground](#playground) to sweep one through five. Larger blocks can improve decode latency when acceptance stays high, but they also increase verification work and graph memory. Start from the checkpoint default, then sweep downward under the real prompt-length and concurrency distribution. The gain is usually largest for short interactive traffic and narrows as prefill dominates. Track P50/P99 TTFT and TPOT, total throughput, accepted length, GPU memory, and stop rate rather than choosing from acceptance alone. For every candidate, compare with the same recipe without `--speculative-algorithm DSPARK`. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each `bench_serving` leg its own `--flush-cache`. Leave `--speculative-draft-attention-backend` unset unless a separate profiling run justifies an override. DSpark requires `pp_size == 1`. Selecting a prefill or decode role in the Playground keeps the inherited DSpark flags on both roles; for PD-disaggregated configurations that run DSpark, see the [Agentic Long-Context Deployment](#agentic-long-context-deployment) and the [MI355X Pro Official PD pairs](#3-7-pd-disaggregation-on-mi355x-mori-io). The MI355X Flash Official recipes run target-only, and so do the DP-Attention recipes in the Deploy panel — for DP-Attention configurations that do run DSpark, see the [Agentic Long-Context Deployment](#agentic-long-context-deployment) and the [MI355X agentic recipe](#3-6-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark). If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates. ### 3.5 Vision (Image Inputs) The experimental [`DeepSeek-V4-Flash-Vision-Exp`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) checkpoint (the **Flash Vision** variant in the Deploy panel — see the [configuration notes](#vision-note)) takes images via the OpenAI-compatible `image_url` content type, as public URLs or base64 `data:` URIs; text and images mix freely in one message. Vision input works with the same server the Deploy panel produces — no extra model-specific flags needed. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-Vision-Exp", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image in a few sentences."}, ], }], ) message = response.choices[0].message if getattr(message, "reasoning_content", None): print("=============== Thinking =================") print(message.reasoning_content) print("=============== Content =================") print(message.content) ``` ```text Output theme={null} Pending update... ``` ### 3.6 Agentic Long-Context with HiCache DRAM Offload (MI355X FP4, DSpark) Three commands follow, one per concurrency range. They share the model, parser, attention and DSpark flags, and differ in parallelism, MoE path and host offload: | | TP8 | TP8 + HiCache | DP8 | DP8 EP8 MegaMoE | | - | - | - | - | - | | Concurrency | 1–16 | 32–48 | 128 | 128–384 | | Parallelism | TP8 | TP8 | TP8+DP8 | TP8+DP8+EP8 | | MoE | replicated | replicated | replicated | MegaMoEv2 | | HiCache | off | on | on | on | | `--mem-fraction-static` | 0.86 | 0.86 | 0.92 | 0.85 | DeepSeek-V4-Pro-0813 bundles the DSpark head, so `--speculative-draft-model-path` is not needed. `--speculative-dspark-block-size 6` is the AL-optimal draft length on the committed golden curve (verify window 7). Keep `SGLANG_DSV4_UNIFIED_KV_FP8=1` on every command. The first three use `--enforce-shared-experts-fusion`; the MegaMoE command uses `--disable-shared-experts-fusion` instead. Four adjustments are not in the commands below. Add them when they apply: * `--cuda-graph-backend-prefill breakable` at concurrency 1–4. * `--enable-prefill-cp --cp-strategy interleave --enable-cp-decode-attn-tp` on the TP-only commands at concurrency 16 and above. Prefill context parallelism splits each prompt across all eight ranks (CP size = TP size). * `SGLANG_DSV4_GROUPED_ASM=1` on the TP-only commands at concurrency 16 and above — the kernel needs at least 16 requests per draft token. * `--prefill-decode-interval 15` on the MegaMoE command at concurrency 384, replacing the `20` it ships with. **TP8, concurrency 1–16:** ```bash Command theme={null} SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \ SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \ SGLANG_DSV4_UNIFIED_KV_FP8=1 \ SGLANG_USE_ROCM700A=0 \ TORCH_BLAS_PREFER_HIPBLASLT=1 \ SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ AITER_BF16_FP8_MOE_BOUND=0 \ SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \ SGLANG_OPT_FP8_WO_A_FUSED_INVROPE=1 \ python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ --trust-remote-code \ --tp 8 \ --prefill-decode-interval 20 \ --attention-backend dsv4 \ --enable-deepseek-v4-fp4-indexer \ --page-size 256 \ --swa-full-tokens-ratio 0.1 \ --kv-cache-dtype fp8_e4m3 \ --enforce-shared-experts-fusion \ --mem-fraction-static 0.86 \ --chunked-prefill-size 16384 \ --tool-call-parser deepseekv4 \ --reasoning-parser deepseek-v4 \ --speculative-algorithm DSPARK \ --speculative-dspark-block-size 6 \ --watchdog-timeout 3600 ``` **TP8 + HiCache, concurrency 32–48:** the same command with HiCache DRAM offload added. `--hicache-ratio` is the host/device token ratio; `1.5` is what stays under a \~2.7 TB host DRAM budget at TP8. The same five flags appear in the two DP commands below. ```bash Command theme={null} --enable-hierarchical-cache \ --hicache-ratio 1.5 \ --hicache-write-policy write_through \ --hicache-io-backend direct \ --hicache-mem-layout page_first_direct ``` **DP8 (DP Attention), concurrency 128:** ```bash Command theme={null} SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \ SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \ SGLANG_DSV4_UNIFIED_KV_FP8=1 \ SGLANG_USE_ROCM700A=0 \ TORCH_BLAS_PREFER_HIPBLASLT=1 \ SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ AITER_BF16_FP8_MOE_BOUND=0 \ SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \ SGLANG_OPT_FP8_WO_A_FUSED_INVROPE=1 \ SGLANG_SHARED_EXPERT_TP1=1 \ SGLANG_DP_SHARED_EXPERT_LOCAL=1 \ SGLANG_DP_USE_GATHERV=1 \ SGLANG_DP_USE_REDUCE_SCATTER=1 \ python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ --trust-remote-code \ --tp 8 \ --attn-dp-size 8 \ --enable-dp-lm-head \ --enable-prefill-delayer \ --enable-dp-attention-local-control-broadcast \ --tokenizer-worker-num 8 \ --stream-interval 20 \ --prefill-decode-interval 20 \ --prefill-delayer-token-usage-low-watermark 0.7 \ --load-balance-method total_requests \ --attention-backend dsv4 \ --enable-deepseek-v4-fp4-indexer \ --page-size 256 \ --swa-full-tokens-ratio 0.1 \ --kv-cache-dtype fp8_e4m3 \ --enforce-shared-experts-fusion \ --mem-fraction-static 0.92 \ --chunked-prefill-size 65536 \ --tool-call-parser deepseekv4 \ --reasoning-parser deepseek-v4 \ --speculative-algorithm DSPARK \ --speculative-dspark-block-size 6 \ --enable-hierarchical-cache \ --hicache-ratio 1.5 \ --hicache-write-policy write_through \ --hicache-io-backend direct \ --hicache-mem-layout page_first_direct \ --watchdog-timeout 3600 ``` `--chunked-prefill-size` is a global budget divided by `--attn-dp-size`, so 65536 keeps 8192 tokens per rank. `--enable-dp-lm-head` is required for DSpark under DP attention. Front the DP ranks with `sglang_router --policy cache_aware`, so a multi-turn session lands on the rank already holding its prefix; above concurrency 160 add `--balance-abs-threshold 32`. Both apply to the MegaMoE command below as well. **DP8 EP8 MegaMoE, concurrency 128–384:** ```bash Command theme={null} SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \ SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \ SGLANG_DSV4_UNIFIED_KV_FP8=1 \ SGLANG_USE_ROCM700A=0 \ TORCH_BLAS_PREFER_HIPBLASLT=1 \ SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ AITER_BF16_FP8_MOE_BOUND=0 \ SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \ SGLANG_OPT_FP8_WO_A_FUSED_INVROPE=1 \ SGLANG_AMD_USE_FLYDSL_MEGA_MOE=1 \ SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR=8192 \ SGLANG_AMD_FLYDSL_MEGA_QUANT=a8w4 \ MORI_SHMEM_HEAP_SIZE=17179869184 \ SGLANG_AITER_MEGA_RANK_SYNC=1 \ SGLANG_AITER_MEGA_EPLB_PREFILL_ONLY=1 \ SGLANG_AITER_MEGA_EPLB_FUSED_MAP_RECORD=1 \ SGLANG_DP_USE_REDUCE_SCATTER=0 \ GPU_MAX_HW_QUEUES=5 \ python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ --trust-remote-code \ --tp 8 \ --attn-dp-size 8 \ --ep 8 \ --enable-dp-lm-head \ --enable-prefill-delayer \ --enable-dp-attention-local-control-broadcast \ --tokenizer-worker-num 8 \ --stream-interval 20 \ --prefill-decode-interval 20 \ --prefill-delayer-token-usage-low-watermark 0.7 \ --load-balance-method total_requests \ --attention-backend dsv4 \ --enable-deepseek-v4-fp4-indexer \ --page-size 256 \ --swa-full-tokens-ratio 0.1 \ --kv-cache-dtype fp8_e4m3 \ --moe-a2a-backend megamoe \ --moe-dense-tp-size 1 \ --disable-shared-experts-fusion \ --enable-eplb \ --eplb-rebalance-num-iterations 200 \ --expert-distribution-recorder-mode stat \ --mem-fraction-static 0.85 \ --chunked-prefill-size 65536 \ --tool-call-parser deepseekv4 \ --reasoning-parser deepseek-v4 \ --speculative-algorithm DSPARK \ --speculative-dspark-block-size 6 \ --enable-hierarchical-cache \ --hicache-ratio 1.5 \ --hicache-write-policy write_through \ --hicache-io-backend direct \ --hicache-mem-layout page_first_direct \ --watchdog-timeout 3600 ``` This command adds expert parallelism on top of the DP8 one. [§3.7](#3-7-pd-disaggregation-on-mi355x-mori-io) explains `SGLANG_AMD_USE_FLYDSL_MEGA_MOE`, `SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR` and the MORI symmetric heap; the heap is 16 GiB here rather than 8 GiB because `--mem-fraction-static 0.85` leaves more GPU memory free. `SGLANG_DP_USE_REDUCE_SCATTER=0` turns off the reduce-scatter path that MegaMoE's all-to-all replaces. The three `SGLANG_AITER_MEGA_*` variables and `--enable-eplb` enable expert-load balancing, recorded during prefill only. ### 3.7 PD Disaggregation on MI355X (MORI-IO) The prefill and decode roles can use any transport the **PD Disagg** card in the [Playground above](#playground) offers. On MI300X / MI355X that is **MORI-IO**, **Mooncake** and **NiXL**, and all three stay selectable. This section covers MORI-IO only, because it is the transport every AMD recipe in this cookbook was measured with. It is also what the card selects by default on those two platforms, and it is the one transport that is ROCm-only, so the card hides it on the CUDA platforms. The MI355X multi-node recipes start with **IB Device → Auto**. Select the `rdma0-7 (all NICs)` option in the Playground to add the explicit NIC list. ROCm nodes enumerate their NICs as `rdmaN`, so the card hides the ConnectX `mlx5_*` names on AMD platforms. On CUDA platforms it hides the `rdmaN` list instead. Both roles need the RDMA NICs passed into the container, so the **Docker** output adds the fabric flags below on top of the usual ROCm device access from the [Install panel](#install). `/dev/infiniband` covers `rdma_cm` and the per-NIC `uverbs0`…`uverbs7` nodes, and the memlock/`IPC_LOCK` pair is what lets MORI-IO pin the buffers it registers with the NICs: ```bash Command theme={null} docker run \ --device=/dev/kfd --device=/dev/dri --device /dev/infiniband \ --group-add video --cap-add IPC_LOCK \ --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \ --ulimit memlock=-1 --ulimit stack=67108864 \ --ulimit nofile=1048576:1048576 \ --network host --ipc=host --shm-size 32g \ -v ~/.cache/huggingface:/root/.cache/huggingface \ lmsysorg/sglang-rocm:v0.5.21-rocm724-mi35x-20261001 \ python3 -m sglang.launch_server ``` The Deployment and Playground Docker outputs use this `v0.5.21` build for the MI355X Pro and Pro Official FP4 multi-node recipes. It is the build the PD roles ran on, and the first one carrying the Aiter MegaMoEv2 kernels the high-throughput prefill role needs. `--network host` replaces the single-node `-p` mapping because the two roles run on different hosts. For the **MI355X Pro / Pro Official FP4** presets, **High-Throughput** sets `MORI_IO_QP_MAX_SEND_WR=32767` and `MORI_IO_QP_MAX_CQE=32768` on both MORI roles. Low-Latency and Balanced use the transport's default queue sizes. The send-queue backoff timeout uses MORI's default of 5 seconds. #### Role-based configuration Select **MI355X → Pro Official → FP4 → Multi-Nodes**, then a strategy and role. The commands use the [submitted recipe at `df5e433239d7`](https://github.com/SemiAnalysisAI/InferenceX/blob/df5e433239d72106529ece4dd97cc36e4c3adf4c/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/mi355x-fp4/agentx/disagg-umbp-dspark.yaml), with cluster-specific paths replaced by local placeholders and benchmark acceptance simulation removed. | Strategy | Concurrency | Prefill → Decode | `--max-running-requests` | DSpark block size | | - | - | - | - | - | | Low-Latency | 16 | TP4 → TP8 | 32 | 6 | | Balanced | 48 | TP4 → TP8 | 96 | 3 | | High-Throughput | 256 | TP8+DP8+EP8 → TP8+DP8 | 512 | 3 | These are two separate workers: neither receives `--nnodes 2` or a cross-node TP rendezvous address. Prefill listens on `30000`, decode on `30100`, and the router on `8000`. Other hardware's Multi-Nodes recipes can still represent a model sharded across nodes. `--cuda-graph-bs-decode` covers 1 to `--max-running-requests / attn_dp_size`: the full value on the TP-only rows and an eighth on the DP8 rows. Each strategy sets both workers' admission limit and decode's graph sizes together. The TP-only points use `--mem-fraction-static 0.86` and SWA ratio `0.1`; the DP8 points use `0.92` and `0.15`. All prefill roles enable [UMBP](#3-8-unified-cache-external-linker-umbp), `--optimistic-prefill-attempts 2`, and cache reporting. Decode does not use an offload tier. The high-throughput prefill worker uses Aiter MegaMoEv2 with EP8; decode keeps TP MoE. Both high-throughput workers enable the FP8 unified KV layout. The commands below show concurrency 256. Select **Pro** to use the original checkpoint's **EAGLE/MTP 3-1-4** head with the same topology and admission limits. Its commands contain no DSpark parameters. In the Playground, **Off (greedy)** removes speculative decoding. Enabling **HiCache** automatically turns off **Unified Cache External Linker** and removes UMBP's socket configuration and optimistic prefill. Enabling the linker turns HiCache off. Selecting a TP-only role forces **DP-Attention** off and greys the control. `--max-running-requests` is server-wide and floor-divided by `attn_dp_size`, so turning DP on would cut the per-rank batch below the captured `--cuda-graph-bs-decode` values without changing either flag in the command. **Speculative decoding changes how you size `--max-running-requests`.** For a target concurrency of N, set it to `N*2` on **both** roles; at N the served batch stays below the concurrency you are aiming for. This holds for DSpark and EAGLE/MTP alike. The table above follows it: 32 serves a concurrency of 16, 96 serves 48, and 512 serves the high-throughput concurrency of 256. The decode role's `--cuda-graph-bs-decode` then has to cover `N*2 / attn_dp_size`, the per-rank batch. The strategy presets apply both rules: decode graphs cover 1–32, 1–96, and 1–64, respectively. Two things then differ between the two roles regardless of strategy: * **CUDA graphs.** Prefill runs `--disable-cuda-graph`, because the FP4 indexer's prefill path is not captured. Decode captures `--cuda-graph-bs-decode`. * **Speculative decoding** is the same on both. Keep `--speculative-dspark-block-size` equal on the two roles. The original Flash and Pro checkpoints have no DSpark head and use their MTP head instead: `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. **MoE parallelism differs too, on the high-throughput pair.** Each role uses what suits its batch size. Prefill runs **EP8**, with the experts split across the 8 ranks: it processes a 65536-token chunk, so the split pays for the all-to-all it costs. Decode runs **TP8**, with the experts replicated on every rank, because a few tokens per request per step would not pay for it. The two commands below are that pair at concurrency 256, on one prefill node and one decode node. Run each on its own host, then front them with the router. **Prefill, EP8:** ```bash Command theme={null} GPU_MAX_HW_QUEUES=5 \ SGLANG_AMD_USE_FLYDSL_MEGA_MOE=1 \ SGLANG_AMD_FLYDSL_MEGA_QUANT=a8w4 \ MORI_SHMEM_HEAP_SIZE=8G \ SGLANG_DSV4_UNIFIED_KV_FP8=1 \ SGLANG_DP_USE_REDUCE_SCATTER=0 \ UMBP_STANDALONE_ADDRESS=unix:///tmp/umbp/standalone.grpc.sock \ python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ --trust-remote-code \ --host 0.0.0.0 --port 6100 \ --disaggregation-mode prefill \ --disaggregation-bootstrap-port 7200 \ --disaggregation-transfer-backend mori \ --disaggregation-ib-device rdma3,rdma0,rdma2,rdma1,rdma7,rdma4,rdma6,rdma5 \ --tp-size 8 --dp-size 8 --ep-size 8 \ --enable-dp-attention \ --enable-dp-attention-local-control-broadcast \ --enable-dp-lm-head \ --moe-a2a-backend megamoe \ --moe-dense-tp-size 1 \ --enforce-shared-experts-fusion \ --attention-backend dsv4 \ --enable-deepseek-v4-fp4-indexer \ --kv-cache-dtype fp8_e4m3 \ --page-size 256 \ --swa-full-tokens-ratio 0.15 \ --mem-fraction-static 0.92 \ --chunked-prefill-size 65536 \ --max-running-requests 512 \ --context-length 1048576 \ --disable-cuda-graph \ --speculative-algorithm DSPARK \ --speculative-dspark-block-size 3 \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-unified-cache-external-linker \ --unified-cache-external-linker-backend mori \ --optimistic-prefill-attempts 2 \ --enable-cache-report \ --load-balance-method round_robin \ --tokenizer-worker-num 8 \ --stream-interval 20 \ --tool-call-parser deepseekv4 \ --reasoning-parser deepseek-v4 \ --enable-metrics \ --watchdog-timeout 3600 ``` **Decode, TP8:** ```bash Command theme={null} GPU_MAX_HW_QUEUES=5 \ SGLANG_DSV4_UNIFIED_KV_FP8=1 \ SGLANG_SHARED_EXPERT_TP1=1 \ SGLANG_DP_SHARED_EXPERT_LOCAL=1 \ SGLANG_DP_USE_GATHERV=1 \ MORI_MAX_DISPATCH_TOKENS_DECODE=256 \ python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ --trust-remote-code \ --host 0.0.0.0 --port 6100 \ --disaggregation-mode decode \ --disaggregation-transfer-backend mori \ --disaggregation-ib-device rdma3,rdma0,rdma2,rdma1,rdma7,rdma4,rdma6,rdma5 \ --tp-size 8 --dp-size 8 \ --enable-dp-attention \ --enable-dp-attention-local-control-broadcast \ --enable-dp-lm-head \ --enforce-shared-experts-fusion \ --attention-backend dsv4 \ --enable-deepseek-v4-fp4-indexer \ --kv-cache-dtype fp8_e4m3 \ --page-size 256 \ --swa-full-tokens-ratio 0.15 \ --mem-fraction-static 0.92 \ --max-running-requests 512 \ --cuda-graph-bs-decode 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 \ --context-length 1048576 \ --speculative-algorithm DSPARK \ --speculative-dspark-block-size 3 \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --load-balance-method round_robin \ --tokenizer-worker-num 8 \ --stream-interval 20 \ --tool-call-parser deepseekv4 \ --reasoning-parser deepseek-v4 \ --enable-metrics \ --watchdog-timeout 3600 ``` Decode sets neither `--ep-size` nor `--moe-a2a-backend`. `--moe-a2a-backend` names the dispatch path for expert parallelism, so at `ep_size` 1 there is nothing for it to name, and the MoE runs replicated across TP. The shared-expert, `SGLANG_DP_USE_GATHERV`, and `SGLANG_DP_USE_REDUCE_SCATTER` settings control the `all_gatherv` / `reduce_scatterv` path. Shared-expert locality and gatherv default to off; reduce-scatter defaults to on for ROCm. The commands only set the values that differ from those defaults. That is what decode's TP MoE uses and what prefill's all-to-all replaces, so leaving them on under EP8 would run both over the same tokens. The rest of the difference is the prefill-only set already described above: `--chunked-prefill-size`, `--disable-cuda-graph`, and the linker flags with `--optimistic-prefill-attempts` / `--enable-cache-report`. Decode replaces the first two with `--cuda-graph-bs-decode`. On the prefill env: * `SGLANG_AMD_USE_FLYDSL_MEGA_MOE=1` is required. `--moe-a2a-backend megamoe` alone runs the DeepGEMM MegaMoE path, which has no ROCm kernel. Do not set this variable on decode: the scheduler reads it too, so it also changes how decode schedules batches. * `SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR` must be at least `--chunked-prefill-size / attn_dp_size`, here 65536 / 8. Below that SGLang reports no error and runs fused MoE for the extra tokens. 8192 is the default, so the command omits this variable. Set it explicitly if you increase the per-rank chunk size. * `MORI_SHMEM_HEAP_SIZE` sizes the MORI symmetric heap, where MegaMoEv2 allocates its dispatch and combine buffers. It needs about 0.51 GiB per 1024 tokens of MTPR — about 4.2 GiB at 8192 — and the 4 GiB default fails during CUDA graph capture. SGLang allocates the whole heap at the first MegaMoE forward pass, after the KV cache pool, so it consumes the GPU memory `--mem-fraction-static` left free whether MegaMoE fills it or not. 8 GiB fits next to `0.92`; [§3.6](#3-6-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark) runs at `0.85` and can afford more. These recipes leave `MORI_SHMEM_MODE` unset and use MORI's default `STATIC_HEAP`. The pinned Aiter MegaMoEV2 implementation allocates its buffers through `mori_shmem_create_tensor()` / `shmem_malloc()` after SGLang initializes MORI shmem. This requires the symmetric heap; the separate MORI CCO allocator is not used by this implementation. If your launch script exports `MORI_SHMEM_MODE=ISOLATION`, unset it before launching MegaMoE: its peer-pointer lookup requires the heap mapping. Keep `MORI_SHMEM_HEAP_SIZE=8G` for the prefill recipe above. **Both roles set `SGLANG_DSV4_UNIFIED_KV_FP8=1`**, which splits the unified KV cache pool into an FP8 nope pool and a BF16 rope pool. It requires the `unified_kv_triton` kernels, which the base cell already selects through `SGLANG_HACK_FLASHMLA_BACKEND`. The value must match on both roles — the PD handshake compares KV cache layouts and fails if they differ. The DSpark draft worker stays BF16 either way. #### Router configuration Run the two roles on separate nodes with the same transport and IB device, then front them with the router the card prints. On MI355X it is cache-aware: `--policy consistent_hashing` keeps a conversation on the prefill worker that already holds its prefix, and `--balance-abs-threshold 2` / `--balance-rel-threshold 1.1` stop that affinity from starving the peer. Decode holds no reusable prefix, so `--decode-policy round_robin` spreads it evenly. `--dp-aware` takes effect only when the workers run DP attention. The multi-node router presets retain the default request timeout. The standalone high-throughput Playground preset uses `--request-timeout-secs 14400`. At high concurrency the prefill role queues many requests. When the default 1800 s passes the router aborts the request, and an abort that lands while MORI is writing KV cache over RDMA leaves the transfer neither finished nor failed: that prefill worker can no longer send KV cache to decode, and you have to restart it. `--request-timeout-secs` is the timeout of the HTTP request from the router to the worker, not a limit on queue time. At 14400 the router also waits four hours before giving up on a worker that has stopped responding. That is acceptable where prefill queues are long, which is why the lower-concurrency recipes keep the default. Lower it again if you lower the concurrency. ### 3.8 Unified Cache External Linker (UMBP) The **unified cache external linker** is the second way to move KV cache off the GPU. The linker is the SGLang feature, and you enable it with `--enable-unified-cache-external-linker`. UMBP is one store the linker can use; you select it with `--unified-cache-external-linker-backend mori`. The Playground card uses the feature name, not the store name. The linker is an **alternative to [HiCache](#3-3-hicache-hierarchical-kv-caching), not a tier inside it**. HiCache is a hierarchy — GPU, then pinned host memory, then optionally a storage backend. The linker connects the unified radix tree directly to the external store, with no host cache tier in between, so pages load and offload against that store. SGLang rejects the two together, so enabling the **Unified Cache External Linker** card in the [Playground above](#playground) strips the whole HiCache flag family from the command: ```bash Command theme={null} --enable-unified-cache-external-linker \ --unified-cache-external-linker-backend mori ``` For **MI355X Pro / Pro Official FP4**, the Playground keeps the two Enable controls mutually exclusive: turning either on automatically turns the other off. You can also leave both off. The **Backend** knob picks the linker backend. `mori` is the UMBP pool; `mooncake` drives the same direct-linker path against a Mooncake store instead. **The linker does not need PD disaggregation.** It works in aggregated serving as well, with the **PD Disagg** card set to Off. The §3.7 pairs use it on their prefill role because that is where the KV is produced, but that is a property of those recipes, not a requirement of the linker. Turn the card on next to any command whose selection satisfies the condition below. **DP Attention is not required.** The linker runs under pure TP as well, and the Pro Official low-latency and balanced prefill roles in [§3.7](#3-7-pd-disaggregation-on-mi355x-mori-io) are exactly that shape. DP Attention does change how much the tier holds, so it is worth understanding before you size one. The linker keys its objects by rank, and MLA KV is replicated across TP. Under pure TP, a TP8 worker therefore opens eight separate keyspaces, and all eight hold copies of the same tokens: the tier stores an eighth of the distinct tokens that its byte budget suggests. At TP4 it stores a quarter. DP Attention collapses the keys onto one shared keyspace, so the whole budget counts. Size the pool for the copies you will actually get, and do not read the byte budget as a distinct-token figure. Every MI355X Pro / Pro Official multi-node prefill role uses a **standalone** UMBP tier: a separate `umbp_standalone_server` process on the prefill node owns the DRAM pool, and the prefill server attaches to it over a Unix socket. Start the tier server first; the 1500 GB DRAM budget below is what the verified runs used: ```bash Command theme={null} mkdir -p /tmp/umbp UMBP_DRAM_CAPACITY=1500000000000 \ UMBP_DRAM_USE_HUGEPAGES=1 \ UMBP_SSD_ENABLED=0 \ MORI_UMBP_LOG_LEVEL=info \ /umbp_standalone_server unix:///tmp/umbp/standalone.grpc.sock ``` Reserve enough hugepages for the 1.5 TB tier on a host with at least 3 TB DRAM, leaving memory for model loading and the worker. Wait for `host memory registered for GPU access` before starting prefill. For the pinned ROCm image, the tier binary is `/sgl-workspace/mori/python/mori/umbp_standalone_server`; include `/sgl-workspace/mori/python/mori` in `LD_LIBRARY_PATH`. If the tier also runs in Docker, pass the ROCm and RDMA devices and hugepage access through to that container and mount `-v /tmp/umbp:/tmp/umbp` in **both** containers. The generated Prefill Docker command already includes that socket-directory mount. The standalone server defaults to `unix:///run/umbp/standalone/node0.grpc.sock` when neither an address argument nor `UMBP_STANDALONE_ADDRESS` is supplied (`UMBP_NODE_ID` replaces `node0` when set). SGLang's client has no default standalone address: set `UMBP_STANDALONE_ADDRESS` to enable this mode. These commands use an explicit socket under `/tmp/umbp` so the server and worker share the same path. UMBP defaults to a 4 GiB DRAM capacity, hugepages disabled, and a 30-second startup wait. This recipe uses 1.5 TB with hugepages enabled and keeps the default startup wait. It also sets `UMBP_SSD_ENABLED=0` to disable the SSD tier (whose config default is enabled), and `MORI_UMBP_LOG_LEVEL=info` to expose the readiness log (the default log level is error). Then launch the prefill role with the socket in its environment. The Deployment panel and Playground include this address and use the default 30-second startup wait. ```bash Command theme={null} UMBP_STANDALONE_ADDRESS=unix:///tmp/umbp/standalone.grpc.sock \ sglang serve ... ``` Two further limits are worth knowing before you size a run. The pool is a per-node process, so a worker that spans nodes would shard its keyspace by node. And under PD disaggregation the §3.7 pairs run the linker on the prefill role only, so the decode worker offloads nothing; in aggregated serving the single worker offloads its own KV. # DeepSeek-V4.1 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1 Deploy DeepSeek-V4.1 Flash with SGLang — launch recipes, feature compatibility, and tuning notes for GB300, H200, B200, B300 and MI355X. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. DeepSeek-V4.1 Flash support has **not shipped in an SGLang release yet**, so a stock `pip install sglang` cannot serve it. Use the preview Docker image below. For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). **NVIDIA GPUs** — every NVIDIA cell on this page uses the preview build `lmsysorg/sglang:dev-dsv41` (H200 / B200 / B300 / GB300); the command panel picks it automatically. ```bash Command theme={null} docker pull lmsysorg/sglang:dev-dsv41 docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:dev-dsv41 \ sglang serve ``` **AMD GPUs (ROCm)** — AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version. ```bash Command theme={null} docker pull lmsysorg/sglang-rocm:v0.5.21-rocm720-mi35x-20261006 docker run \ --device=/dev/kfd --device=/dev/dri \ --group-add video \ --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \ --shm-size 32g --ipc=host \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ lmsysorg/sglang-rocm:v0.5.21-rocm720-mi35x-20261006 \ sglang serve ``` Pick your hardware + recipe to generate the launch command: * **Low-Latency** — fastest reply for a single user. Pick for chat. * **High-Throughput** — most tokens per second across many users. Best for batch jobs. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## Agentic Long-Context Deployment Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order. * **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks). * **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`. * **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image. ## 1. Model Introduction **DeepSeek-V4.1 Flash** is a sparse-attention Mixture-of-Experts model (`model_type: deepseek_v4.1`) served through SGLang's `dsv4` backend: 40 decoder layers, 384 routed experts at top-6 plus one shared expert, fp8 dense weights at a 32-wide `ue8m0` block scale with fp4 routed experts. It also carries **DSpark**, its own three-stage speculative draft, which the Low-Latency recipe turns on. **Architecture.** Every layer keeps one 512-wide KV latent per position and uses it as both key and value for all 64 query heads, with no separate value projection. Each layer reads two stores that behave oppositely: **compressed latents**, produced only at source layers and shared forward, collapsing 2 positions into 1 in layers 2–19 and 1-to-1 from layer 20 on; and a **128-position sliding window**, recomputed per layer from that layer's own activations, so it can never be shared but also never grows with context. Consumers address a source's cache by plain `loc // ratio` arithmetic with no mapping table, so compressed state lives and dies with the full prefix for free. The residual stream is four parallel copies mixed by a per-token doubly stochastic matrix; each sublayer's mixing coefficients are consumed by the *next* sublayer, which lets that projection overlap the main GEMMs. **Sparse retrieval.** KV source layers and index source layers are different lists — four of the former, eight of the latter. The four extra index layers produce no keys at all; they re-score layer 20's keys with their own query, so retrieval decisions are made twice as often as keys are stored. Each retrieving layer picks a top-512 candidate set. **Engram.** An additive n-gram hash memory at two layers. Token ids are normalized *before* hashing, so `" The"`, `"the"` and `"THE"` cannot fork into separate rows. Its two fp8 tables are the largest single block of weight in the checkpoint, and by default they load row-sharded across the TP group, which costs an all-reduce per engram layer. `SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1` (**opt-in**) moves them to host memory instead: the freed HBM goes to the KV pool and output is bitwise unchanged — at the cost of host RAM, a longer load, and needing huge-page backing to keep the gather cheap. `SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT` trades the two all-reduces for page size: `shared` (the default) maps a single copy all ranks read and needs no reduce; `per_rank` splits that same table into one shard per rank and keeps the reduce, but is the only one the kernel reliably backs with 2 MiB pages. **SWA bounded replay.** Because the window store is per-layer and cheap to rebuild, it need not all be recomputed. `--enable-decoder-swa-bounded-replay` (**opt-in**) runs the early layers over the whole extend and the late layers over only each request's last 128 tokens, sharing late-layer KV from source layer 20. Prefill gets materially faster. It is validated on the decode path only, refuses prompt logprobs by design, is not numerically equivalent to full prefill, and is excluded at launch with the prefill CUDA graph and with DP attention. **Kernels.** At decode token counts these kernels are launch-bound rather than bandwidth-bound, so the work is fusion and overlap: the compressor projection, ratio-2 pooling, RoPE with fp4 quantization and indexer packing, paged fp4 indexer scoring with top-k resolved straight to cache addresses, and the mHC statistics and Sinkhorn — with compression, indexing and mHC overlapped against attention and FFN on dedicated streams. All of it is selected automatically; the environment switches it once sat behind were removed, so there is nothing to turn on. One caveat outranks the fusions: **output is not bitwise stable across batch composition today** — two default kernels are shape-guarded to a single token — and `--enable-deterministic-inference` is refused on this backend. **Recommended generation:** reasoning evaluations were run at `temperature=1.0`, `top_p=0.95` with reasoning effort `max` (informational — do not hardcode these in application code). **Resources:** [HuggingFace](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash). ## 2. Configuration Tips ### Do not override the backends `--attention-backend`, `--moe-runner-backend` and `--fp8-gemm-backend` are selected automatically from the model, hardware, forward mode and shape. On GB300 they resolve to `dsv4` / `flashinfer_mxfp4` / `flashinfer_cutedsl`. Confirm them in the startup log rather than passing them. On gfx950 the MI355X recipes pin `--fp8-gemm-backend aiter` and leave the other two to resolve. Overriding them is the most common cause of a disappointing measurement: it leaves the 32-wide ue8m0 blocks on the Triton `_w8a8_block_fp8_matmul` fallback, which dominates the decode step and costs most of the model's bs=1 throughput. If your decode rate looks like a small fraction of what you expected, check the resolved backends first. ## 3. Advanced Usage ### 3.1 Reasoning Enable the reasoning parser — `--reasoning-parser auto` resolves to `deepseek-v41` (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer. The parser puts the thinking block in `reasoning_content` and the answer in `content`; without it both arrive concatenated in `content`. The parser can only split a thinking block the model actually produced, and **thinking is off by default** (`SGLANG_DEFAULT_THINKING=false`). Sending `reasoning_effort` turns it on: any value other than `none` also switches `chat_template_kwargs.thinking` on for you. A request that carries neither comes back with an empty `reasoning_content` however the parser is configured. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4.1-Flash", messages=[{"role": "user", "content": "What is 15% of 240?"}], reasoning_effort="high", ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` On the request, `reasoning_effort` accepts the tiers `low` (50), `high` (75), `xhigh` (75), and `max` (100), or a float in `[0.0, 0.99]` that maps onto the model's 1–100 budget. An integer budget is reachable only through `chat_template_kwargs.reasoning_effort`, and that route does not switch thinking on by itself — pair it with `chat_template_kwargs.thinking`. `none` turns thinking off. The remaining OpenAI tiers, which have no V4.1 counterpart (`minimal`, `medium`), log a warning and fall back to the server-side default rather than erroring; that default is `high` (75), and `SGLANG_DSV41_REASONING_EFFORT` overrides it. ### 3.2 Tool Calling Enable the tool-call parser — `--tool-call-parser auto` resolves to `deepseekv41` (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. DeepSeek-V4.1 uses spaced DSML tool tags, which the stock DeepSeek-V4 detector does not parse — the `deepseekv41` detector is required. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather in a city", "parameters": { "type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"], }, }, }] resp = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4.1-Flash", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, reasoning_effort="high", ) choice = resp.choices[0] print("finish_reason:", choice.finish_reason) print("tool_calls:", choice.message.tool_calls) print("reasoning:", getattr(choice.message, "reasoning_content", None)) ``` A successful call returns `finish_reason: "tool_calls"` with a single `get_weather` entry whose arguments carry the requested location. If you see the raw DSML markup in `content` instead, the parser flag is missing. ### 3.3 Speculative Decoding (DSpark) DSpark is DeepSeek-V4.1 Flash's own bundled draft — there is no EAGLE or MTP path for this model, and no `--speculative-num-steps` knob (the draft-token count is resolved from the checkpoint). Turn it on with `--speculative-algorithm DSPARK --speculative-dspark-block-size 5`, which is what the Low-Latency cells do. What to expect: * A DSpark step costs meaningfully more than a plain decode step, so the win at bs=1 is roughly the accept length divided by that fixed step cost. It is a real speed-up on interactive workloads, but not proportional to the number of draft tokens. * Acceptance is strongly workload-dependent — highest on maths and code, lower on open-ended chat, lower still on agentic traces. Size your expectations from the workload you actually run, and measure it rather than assuming. * The step cost does not move with accept length or with the draft block size, so tuning `--speculative-dspark-block-size` will not buy throughput. This is also why the High-Throughput recipe turns speculation off: at large batch the fixed step cost stops paying for itself. ### 3.4 PD Disaggregation Prefill/decode disaggregation is validated token-identical against a single server across 32 greedy prompts, with GSM8K matching through the router. Use the **PD Disaggregation** card in the Playground to generate the prefill role, the decode role and the router command. One deployment note: Mooncake needs the RDMA fabric visible inside the container, so launch with `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK --ulimit memlock=-1`. Without it Mooncake selects its NVLink transport, which only serves buffers from its own allocator and fails with `Requested address ... not found`. If you hit that, force TCP with `MOONCAKE_PROTOCOL=tcp` and `MC_FORCE_TCP=1` — the Playground's Mooncake option sets both. PD and speculative decoding cannot be combined. ### 3.5 HiCache (Hierarchical KV Caching) HiCache extends RadixAttention with a hierarchy of KV cache tiers, significantly expanding effective context capacity for long-context and multi-turn scenarios. To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**: the Playground emits `--enable-hierarchical-cache` on top of the recipe's flags, and cold KV pages spill to CPU pinned memory (L2, GPU + CPU) instead of being dropped from the GPU pool. The host pool is sized by ratio (`--hicache-ratio 2`), not by a fixed `--hicache-size`. The Write policy knob controls the GPU → CPU write and defaults to `write_through` (the upstream default): every page is mirrored to the CPU tier as it is written. `write_through_selective` backs up only hot data and `write_back` defers the copy to eviction, trading cache freshness for host-side I/O. The card is not offered on MI355X: hierarchical cache has not been validated on ROCm for this model yet. Only the L2 tier is exposed here. For the storage (L3) tier and the canonical flag set, see the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices) and the [HiCache documentation](../../../docs/advanced_features/hicache). ### 3.6 Agentic Long-Context on MI355X (FP4, DSpark) Agentic replay is a different workload: long-context prompts, multi-turn sessions that reuse their own prefix across turns, and subagent branches that fan out within one session. Concurrency below counts live sessions, not requests. Both topologies keep the engram tables on the host, and neither adds a HiCache tier. The sessions fit the device pool. TP2 sweeps concurrency 1 to 64 and raises `--mem-fraction-static` as it climbs; TP4 sweeps 1 to 16 and holds 0.70, with a quarter of the checkpoint per rank leaving room to spare. The braced values in the command follow concurrency: | Concurrency | 1 | 2 | 4 | 8 | 16 | 32 | 64 | | - | - | - | - | - | - | - | - | | `--max-running-requests` | 2 | 4 | 8 | 16 | 32 | 64 | 128 | | `--cuda-graph-max-bs-decode` | 64 | 64 | 64 | 64 | 64 | 64 | 128 | | `--swa-prefix-tails` | 128 | 128 | 256 | 512 | 1024 | 2048 | 4096 | | `--chunked-prefill-size` | 16384 | 16384 | 16384 | 16384 | 16384 | 16384 | 4096 | | `--prefill-decode-interval` | 16 | 16 | 16 | 16 | 16 | 4 | 4 | | `--mem-fraction-static`, TP2 | 0.70 | 0.70 | 0.70 | 0.70 | 0.80 | 0.80 | 0.85 | | `--mem-fraction-static`, TP4 | 0.70 | 0.70 | 0.70 | 0.70 | 0.70 | — | — | ```bash Command theme={null} SGLANG_DEFAULT_THINKING=1 \ SGLANG_DSV41_REASONING_EFFORT=high \ SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 \ SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \ SGLANG_USE_AITER=1 \ SGLANG_USE_ROCM700A=0 \ TORCH_BLAS_PREFER_HIPBLASLT=1 \ AITER_BF16_FP8_MOE_BOUND=0 \ TRITON_HIP_USE_ASYNC_COPY=0 \ ROCM_QUICK_REDUCE_QUANTIZATION=NONE \ HSA_NO_SCRATCH_RECLAIM=0 \ GPU_MAX_HW_QUEUES=2 \ sglang serve \ --model-path deepseek-ai/DeepSeek-V4.1-Flash \ --trust-remote-code \ --tp {2 or 4} \ --ep-size 1 \ --speculative-algorithm DSPARK \ --speculative-dspark-block-size 5 \ --enforce-shared-experts-fusion \ --fp8-gemm-backend aiter \ --enable-decoder-swa-bounded-replay \ --cuda-graph-backend-prefill disabled \ --mem-fraction-static {mem-fraction-static} \ --chunked-prefill-size {chunked-prefill-size} \ --prefill-decode-interval {prefill-decode-interval} \ --swa-prefix-tails {swa-prefix-tails} \ --max-running-requests {max-running-requests} \ --cuda-graph-max-bs-decode {cuda-graph-max-bs-decode} \ --reasoning-parser auto \ --tool-call-parser auto \ --watchdog-timeout 3600 ``` # Ernie4.5 Source: https://docs.sglang.io/cookbook/autoregressive/Ernie/Ernie4.5 ## 1. Model Introduction The **ERNIE-4.5** series is a family of large language models developed by Baidu. ERNIE (Enhanced Representation through Knowledge Integration) 4.5 represents an advanced version of the ERNIE series, optimized for general-purpose tasks and conversational scenarios. ERNIE-4.5 delivers advanced features as below: * **Heterogeneous Modality Structure**: MoE architecture that supports parameter sharing across modalities while allowing dedicated parameters for each individual modality, enhancing multimodal understanding without compromising, and even improving, performance on text-related tasks. * **Vision Encoder**: Dedicated adaptive-resolution ViT with 2D RoPE and image packing; for video, adaptive frame sampling and timestamp rendering, supporting both shared and modality-specific visual processing. * **Adapter**: Shared modality-bridging module with spatial and temporal compression to align vision to text embedding space, enabling cross-modal understanding without compromising text representations. * **Multimodal Position Embedding**: Unified 3D RoPE (temporal, height, width) for vision and 1D RoPE for text in a single embedding space, supporting parameter sharing while encoding modality-specific positions. * **Hardware Optimization**: Specifically tuned for AMD MI300X, MI325X, and MI355X GPUs. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. ## 4. API Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) The following example demonstrates deployment using ERNIE-4.5-21B-A3B-PT. ```shell Command theme={null} python -m sglang.launch_server \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --tp 1 ``` **Basic Python Client Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="baidu/ERNIE-4.5-21B-A3B-PT", messages=[ {"role": "user", "content": "What is artificial intelligence?"} ], temperature=1.0, top_p=0.95, max_tokens=1024 ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} **Artificial Intelligence (AI)** is the simulation of human intelligence processes by machines, particularly computer systems. These processes include **learning** (acquiring information and rules for using the information), **reasoning** (using rules to reach approximate or definite conclusions), and **self-correction**. AI encompasses a wide range of techniques, algorithms, and methodologies designed to enable machines to perform tasks that typically require human intelligence. ### Key Characteristics of AI: ... ### In Summary: AI represents a transformative force with the potential to revolutionize industries and enhance human capabilities. However, its development requires careful consideration of ethical, legal, and social implications to ensure that it benefits society as a whole. As AI continues to evolve, ongoing dialogue among stakeholders will be crucial to balancing innovation with responsibility. ``` **Streaming Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="baidu/ERNIE-4.5-21B-A3B-PT", messages=[ {"role": "user", "content": "Explain quantum computing in simple terms."} ], temperature=1.0, top_p=0.95, max_tokens=2048, stream=True ) for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} Sure! Here’s a simple explanation of quantum computing: ### **Quantum Computing: Making Computers Super Fast (But Weird) Using Quantum Rules** 1. **Classic vs. Quantum Computers** - **Normal computers** use **bits** (0s and 1s) to store and process information. - **Quantum computers** use **qubits** (short for quantum bits). Unlike bits, qubits can be **0, 1, or both at the same time** (this is called **superposition**). 2. **Superposition: The Magic Behind Speed** - A single qubit can represent **0 and 1 simultaneously**, like a coin spinning in the air. - Many qubits working together (in something called **quantum parallelism**) can **check multiple possibilities at once**, making quantum computers much faster for certain problems. 3. **Entanglement: Making Qubits Link** - When qubits are **entangled**, their states are linked—changing one instantly affects the other, no matter how far apart they are (this is called **spooky action at a distance** by Einstein). - Entanglement allows quantum computers to process information in **very efficient ways**. 4. **What Quantum Computers Are Good At** - **Cracking encryption** (like RSA). - **Factoring large numbers** (used in encryption and cryptography). - **Searching unsorted databases** (way faster than classical computers). - **Simulating quantum systems** (like molecules for drug discovery). - **Optimizing problems** (like logistics or finance). 5. **Challenges & Current State** - Qubits are **fragile** and easily disturbed (called **decoherence**). - Engineers are working to keep qubits stable long enough to do useful calculations. - Today’s quantum computers are **small and experimental**, but the goal is to build powerful ones that outperform classical supercomputers. ### **Final Thought** Quantum computing isn’t just a faster calculator—it’s a **new way of thinking about problems** using the weird laws of physics. While still new, it has the potential to revolutionize fields like medicine, AI, and cybersecurity. Would you like an example of how a quantum computer might solve a problem? 😊 ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X GPU (1x) * Model: ERNIE-4.5-21B-A3B-PT * Tensor Parallelism: 1 * SGLang Version: 0.5.7 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Scenario Benchmark * Model Deployment Command: ```bash Command theme={null} python -m sglang.launch_server \ --model-path baidu/ERNIE-4.5-21B-A3B-PT \ --tp 1 ``` ##### 5.1.1.1 Low Concurrency (Latency-Optimized) * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 58.72 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4219 Request throughput (req/s): 0.17 Input token throughput (tok/s): 103.90 Output token throughput (tok/s): 71.87 Peak output token throughput (tok/s): 245.00 Peak concurrent requests: 2 Total token throughput (tok/s): 175.77 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5869.86 Median E2E Latency (ms): 1870.80 ---------------Time to First Token---------------- Mean TTFT (ms): 4152.58 Median TTFT (ms): 36.81 P99 TTFT (ms): 37498.23 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.07 Median TPOT (ms): 4.09 P99 TPOT (ms): 4.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.08 Median ITL (ms): 4.08 P95 ITL (ms): 4.14 P99 ITL (ms): 4.20 Max ITL (ms): 4.67 ================================================== ``` ##### 5.1.1.2 Medium Concurrency (Balanced) * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 34.30 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40773 Request throughput (req/s): 2.33 Input token throughput (tok/s): 1156.62 Output token throughput (tok/s): 1189.77 Peak output token throughput (tok/s): 1392.00 Peak concurrent requests: 21 Total token throughput (tok/s): 2346.39 Concurrency: 14.14 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6060.62 Median E2E Latency (ms): 6496.70 ---------------Time to First Token---------------- Mean TTFT (ms): 78.90 Median TTFT (ms): 45.90 P99 TTFT (ms): 234.33 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 11.99 Median TPOT (ms): 12.16 P99 TPOT (ms): 14.81 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.75 Median ITL (ms): 11.48 P95 ITL (ms): 12.24 P99 ITL (ms): 34.85 Max ITL (ms): 105.01 ================================================== ``` ##### 5.1.1.3 High Concurrency (Throughput-Optimized) * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 66.63 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 252449 Request throughput (req/s): 7.50 Input token throughput (tok/s): 3749.79 Output token throughput (tok/s): 3792.28 Peak output token throughput (tok/s): 4902.00 Peak concurrent requests: 113 Total token throughput (tok/s): 7542.06 Concurrency: 90.33 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 12036.90 Median E2E Latency (ms): 11782.16 ---------------Time to First Token---------------- Mean TTFT (ms): 104.86 Median TTFT (ms): 84.62 P99 TTFT (ms): 297.85 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 23.89 Median TPOT (ms): 24.62 P99 TPOT (ms): 26.91 ---------------Inter-Token Latency---------------- Mean ITL (ms): 23.66 Median ITL (ms): 20.48 P95 ITL (ms): 45.57 P99 ITL (ms): 54.31 Max ITL (ms): 185.12 ================================================== ``` #### 5.1.2 Reasoning Scenario Benchmark ##### 5.1.2.1 Low Concurrency * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 185.11 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 44462 Total generated tokens (retokenized): 44423 Request throughput (req/s): 0.05 Input token throughput (tok/s): 32.96 Output token throughput (tok/s): 240.19 Peak output token throughput (tok/s): 245.00 Peak concurrent requests: 2 Total token throughput (tok/s): 273.15 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 18508.84 Median E2E Latency (ms): 19866.81 ---------------Time to First Token---------------- Mean TTFT (ms): 32.59 Median TTFT (ms): 32.14 P99 TTFT (ms): 38.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.13 Median TPOT (ms): 4.13 P99 TPOT (ms): 4.20 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.16 Median ITL (ms): 4.12 P95 ITL (ms): 4.31 P99 ITL (ms): 4.36 Max ITL (ms): 7.28 ================================================== ``` ##### 5.1.2.2 Medium Concurrency * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 263.48 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 318306 Total generated tokens (retokenized): 317984 Request throughput (req/s): 0.30 Input token throughput (tok/s): 150.55 Output token throughput (tok/s): 1208.09 Peak output token throughput (tok/s): 1408.00 Peak concurrent requests: 19 Total token throughput (tok/s): 1358.64 Concurrency: 14.35 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 47249.55 Median E2E Latency (ms): 47828.67 ---------------Time to First Token---------------- Mean TTFT (ms): 62.77 Median TTFT (ms): 57.10 P99 TTFT (ms): 93.70 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 11.92 Median TPOT (ms): 12.09 P99 TPOT (ms): 12.50 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.86 Median ITL (ms): 12.04 P95 ITL (ms): 12.68 P99 ITL (ms): 13.61 Max ITL (ms): 39.94 ================================================== ``` ##### 5.1.2.3 High Concurrency * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 428.30 Total input tokens: 158939 Total input text tokens: 158939 Total input vision tokens: 0 Total generated tokens: 1301025 Total generated tokens (retokenized): 1299877 Request throughput (req/s): 0.75 Input token throughput (tok/s): 371.09 Output token throughput (tok/s): 3037.63 Peak output token throughput (tok/s): 3880.00 Peak concurrent requests: 69 Total token throughput (tok/s): 3408.73 Concurrency: 57.08 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 76392.58 Median E2E Latency (ms): 79698.73 ---------------Time to First Token---------------- Mean TTFT (ms): 92.79 Median TTFT (ms): 78.71 P99 TTFT (ms): 168.89 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 18.81 Median TPOT (ms): 19.15 P99 TPOT (ms): 19.81 ---------------Inter-Token Latency---------------- Mean ITL (ms): 18.77 Median ITL (ms): 18.77 P95 ITL (ms): 19.86 P99 ITL (ms): 42.08 Max ITL (ms): 74.36 ================================================== ``` #### 5.1.3 Summarization Scenario Benchmark ##### 5.1.3.1 Low Concurrency * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 18.59 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4216 Request throughput (req/s): 0.54 Input token throughput (tok/s): 2256.43 Output token throughput (tok/s): 227.04 Peak output token throughput (tok/s): 245.00 Peak concurrent requests: 2 Total token throughput (tok/s): 2483.46 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1856.72 Median E2E Latency (ms): 1513.87 ---------------Time to First Token---------------- Mean TTFT (ms): 86.66 Median TTFT (ms): 72.30 P99 TTFT (ms): 167.13 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.19 Median TPOT (ms): 4.22 P99 TPOT (ms): 4.30 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.20 Median ITL (ms): 4.23 P95 ITL (ms): 4.34 P99 ITL (ms): 4.42 Max ITL (ms): 5.68 ================================================== ``` ##### 5.1.3.2 Medium Concurrency * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 40.25 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41669 Total generated tokens (retokenized): 41646 Request throughput (req/s): 1.99 Input token throughput (tok/s): 7454.72 Output token throughput (tok/s): 1035.37 Peak output token throughput (tok/s): 1310.00 Peak concurrent requests: 20 Total token throughput (tok/s): 8490.09 Concurrency: 14.37 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7229.56 Median E2E Latency (ms): 7578.95 ---------------Time to First Token---------------- Mean TTFT (ms): 137.38 Median TTFT (ms): 122.59 P99 TTFT (ms): 485.34 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.04 Median TPOT (ms): 14.24 P99 TPOT (ms): 20.77 ---------------Inter-Token Latency---------------- Mean ITL (ms): 13.64 Median ITL (ms): 12.36 P95 ITL (ms): 14.72 P99 ITL (ms): 57.39 Max ITL (ms): 411.31 ================================================== ``` ##### 5.1.3.3 High Concurrency * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 78.33 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 170000 Total generated tokens (retokenized): 169888 Request throughput (req/s): 4.09 Input token throughput (tok/s): 16262.33 Output token throughput (tok/s): 2170.20 Peak output token throughput (tok/s): 3005.00 Peak concurrent requests: 73 Total token throughput (tok/s): 18432.53 Concurrency: 58.79 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 14392.52 Median E2E Latency (ms): 14460.70 ---------------Time to First Token---------------- Mean TTFT (ms): 184.82 Median TTFT (ms): 155.24 P99 TTFT (ms): 379.82 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 26.97 Median TPOT (ms): 28.31 P99 TPOT (ms): 33.61 ---------------Inter-Token Latency---------------- Mean ITL (ms): 26.79 Median ITL (ms): 20.55 P95 ITL (ms): 47.55 P99 ITL (ms): 145.64 Max ITL (ms): 287.62 ================================================== ``` ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command: ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 1319 \ --num-threads 1316 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. * Test Results: * ERNIE-4.5-21B-A3B-PT ``` Accuracy: 0.865 Invalid: 0.000 Latency: 21.669 s Output throughput: 10359.790 token/s ``` # Ernie4.5-VL Source: https://docs.sglang.io/cookbook/autoregressive/Ernie/Ernie4.5-VL ## 📝 Community Contribution Welcome This guide is currently under development. We welcome community contributions! If you have experience deploying **Ernie4.5-VL** with SGLang, please help us complete this documentation. ## 🚀 How to Contribute ```shell Command theme={null} git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git cd sglang-cookbook git checkout -b add-ernie4-5-vl-guide # Edit this file and submit a PR ``` ## 📚 Reference * [GLM-4.6V](../GLM/GLM-4.6V) *** **Let's build this together!** 🌟 # Chroma-1.0 Source: https://docs.sglang.io/cookbook/autoregressive/FlashLabs/Chroma1.0 ## 1. Model Introduction [Chroma-1.0](https://github.com/FlashLabs-AI-Corp/FlashLabs-Chroma) is an open-source end-to-end speech conversation model developed by FlashLabs, focusing on the following core capabilities: * **Real-time Speech Generation**: Supports low-latency speech synthesis, suitable for real-time conversational scenarios. * **Customized Voice Cloning**: Capable of cloning and replicating specific speaker voice characteristics. * **End-to-End Architecture**: Provides a complete processing workflow from speech to speech. * **Speech Reasoning**: Equipped with reasoning capabilities to understand and process speech content. ## 2. Architecture Overview **Chroma-1.0** utilizes a hybrid serving architecture rather than a direct SGLang deployment. This design choice is driven by: 1. **Complex Model Architecture**: The end-to-end speech processing pipeline involves specialized components that go beyond standard text generation loops. 2. **KV Cache & State Management**: The model requires custom handling of KV caches that differs from standard implementations. 3. **Batching Limitations**: The current implementation supports a batch size of 1, meaning SGLang's advanced continuous batching capabilities are not yet fully applicable. Therefore, you will start the **FlashLabs Server**, which manages the overall workflow and selectively leverages SGLang for specific inference components where supported. * **Outer Layer**: FlashLabs Server (Handles Audio I/O, State, and Model Logic) * **Inner Engine**: SGLang Instance (Utilized for specific acceleration where applicable) ## 3. Installation & Setup We recommend following these steps to set up the environment and prepare the model. ### Step 1: Get the Docker Image Pull the official pre-built image from Docker Hub to ensure all dependencies are correctly configured. ```bash Command theme={null} docker pull flashlabs/chroma:latest ``` ### Step 2: Download Model Weights Download the **Chroma-4B** weights from Hugging Face. You can choose one of the following methods: **Method 1: Using Python (Recommended)** ```bash Command theme={null} huggingface-cli download FlashLabs/Chroma-4B --local-dir Chroma-4B ``` **Method 2: Using Git Clone** Make sure you have Git LFS installed before cloning. ```bash Command theme={null} # Install Git LFS first git lfs install # Clone the repository git clone https://huggingface.co/FlashLabs/Chroma-4B Chroma-4B ``` ### Step 3: Download Chroma Codes (SGLang version) ```bash Command theme={null} git clone https://github.com/FlashLabs-AI-Corp/Chroma-SGLang.git cd Chroma-SGLang ``` ### Step 4: Run the Server ```bash Command theme={null} docker run -d \ --gpus all \ -p 8000:8000 \ -w /app/Chroma-SGLang \ -v "your_Chroma-SGLang_path":/app/Chroma-SGLang \ -v "your_chroma_path":/model \ -e CHROMA_MODEL_PATH=/model \ -e DP_SIZE="1" \ flashlabs/chroma:latest \ /opt/conda/bin/python -m uvicorn api_server:app \ --host 0.0.0.0 \ --port 8000 \ --workers 1 ``` or run simply the following one line command ```bash Command theme={null} docker-compose up -d ``` ## 5. Client Usage Example Once the server is running, you can interact with it using HTTP requests. ### Python Client ```python Example theme={null} import requests import base64 url = "http://localhost:8000/v1/chat/completions" headers = {"Content-Type": "application/json"} payload = { "model": "chroma", "messages": [ { "role": "system", "content": "You are Chroma, a voice agent developed by FlashLabs." }, { "role": "user", "content": [ {"type": "audio", "audio": "assets/question_audio.wav"} ] } ], "max_tokens": 1000, "return_audio": True } response = requests.post(url, json=payload, headers=headers) result = response.json() if result.get("audio"): audio_data = base64.b64decode(result["audio"]) with open("output.wav", "wb") as f: f.write(audio_data) print("Audio saved to output.wav") ``` ### OpenAI SDK Compatible Example ```python Example theme={null} from openai import OpenAI client = OpenAI( api_key="dummy", base_url="http://localhost:8000/v1" ) response = client.chat.completions.create( model="chroma", messages=[ {"role": "system", "content": "You are a helpful assistant."}, { "role": "user", "content": [ {"type": "audio", "audio": "assets/question_audio.wav"} ] } ], extra_body={ "prompt_text": "I have not... I'm so exhausted, I haven't slept in a very long time. It could be because... Well, I used our... Uh, I'm, I just use... This is what I use every day. I use our cleanser every day, I use serum in the morning and then the moistu- daily moisturizer. That's what I use every morning.", "prompt_audio": "assets/ref_audio.wav", "return_audio": True } ) print(response) ``` ### CLI (cURL) ```bash Command theme={null} curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "chroma", "messages": [ { "role": "system", "content": "You are Chroma, a voice agent developed by FlashLabs." }, { "role": "user", "content": [ { "type": "audio", "audio": "assets/question_audio.wav" } ] } ], "max_tokens": 1000, "return_audio": true }' | jq -r '.audio' | base64 -d > output.wav ``` # GLM-4.5 Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.5 ## 1. Model Introduction [GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding. **Key Features:** * **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving * **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs * **Hardware Optimization**: Specifically tuned for AMD MI300X/MI325X/MI355X GPUs * **High Performance**: Optimized for both throughput and latency scenarios **Available Models:** * **BF16 (Full precision)**: [zai-org/GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) - Recommended for MI300X/MI325X/MI355X * **FP8 (8-bit quantized)**: [zai-org/GLM-4.5-FP8](https://huggingface.co/zai-org/GLM-4.5-FP8) - Recommended for MI300X/MI325X/MI355X **License:** Please refer to the [official GLM-4.5 model card](https://huggingface.co/zai-org/GLM-4.5) for license details. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips * **EAGLE Speculative Decoding:** Supported for GLM-4.5/4.6. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. * **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3). ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser GLM-4.5 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.5 \ --reasoning-parser glm45 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/GLM-4.5", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling **Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation. GLM-4.5 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.5 \ --reasoning-parser glm45 \ --tool-call-parser glm45 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-4.5", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` #### 4.2.3 Thinking Budget Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`: ```python Example theme={null} import openai from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*") response = client.chat.completions.create( model="zai-org/GLM-4.5", messages=[{"role": "user", "content": "Is Paris the Capital of France?"}], max_tokens=1024, extra_body={ "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(), "custom_params": {"thinking_budget": 512}, }, ) print(response) ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x) * Model: GLM-4.5 * Tensor Parallelism: 8 * SGLang Version: 0.5.6.post1 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Test Scenarios Three core scenarios reflect real-world usage patterns:
Scenario Input Length Output Length Use Case
**Chat** 1K 1K Most common conversational AI workload
**Reasoning** 1K 8K Long-form generation, complex reasoning tasks
**Summarization** 8K 1K Document summarization, RAG retrieval
#### 5.1.2 Concurrency Levels Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier): * **Low Concurrency**: `--max-concurrency 1` (Latency-optimized) * **Medium Concurrency**: `--max-concurrency 16` (Balanced) * **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) #### 5.1.3 Number of Prompts For each concurrency level, configure `num_prompts` to simulate realistic user loads: * **Quick Test**: `num_prompts = concurrency × 1` (minimal test) * **Recommended**: `num_prompts = concurrency × 5` (standard benchmark) * **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade) *** #### 5.1.4 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.5 \ --tp 8 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` #### 5.1.5 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. # GLM-4.5V Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.5V ## 1. Model Introduction [GLM-4.5V](https://huggingface.co/zai-org/GLM-4.5V) is a state-of-the-art multimodal vision-language model from ZhipuAI, built on the next-generation flagship text foundation model GLM-4.5-Air (106B parameters, 12B active). It achieves SOTA performance among models of the same scale across 42 public vision-language benchmarks. Through efficient hybrid training, GLM-4.5V focuses on real-world usability and enables full-spectrum vision reasoning across diverse visual content types. **Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X GLM-4.5V introduces several key features: * **Image Reasoning & Grounding** Scene understanding, complex multi-image analysis, and spatial recognition with precise visual element localization. Supports bounding box predictions with normalized coordinates (0-1000) for accurate object detection. * **Video Understanding** Long video segmentation and event recognition, supporting comprehensive temporal analysis across extended video sequences. * **GUI Agent Tasks** Screen reading, icon recognition, and desktop operation assistance for agent-based applications. Enables natural interaction with graphical user interfaces. * **Complex Chart & Long Document Parsing** Research report analysis and information extraction from documents with text, charts, tables, and figures. Processes up to 64K tokens of multimodal context. * **Thinking Mode Switch** Allows users to balance between quick responses and deep reasoning. Users can enable/disable Chain-of-Thought reasoning based on task requirements for improved accuracy and interpretability. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The GLM-4.5V offers models in various sizes and architectures, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size. **Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command. ### 3.2 Configuration Tips * **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size \* number of images in current running requests.) * **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder`. * **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`. * **Hardware Notes:** * **H100 (FP8):** Use the FP8 checkpoint for best memory efficiency. * **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage. * **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing. * **Additional Multimodal Parameters:** * `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3). * `--mm-feature-transport=cuda_ipc`: Use CUDA IPC shared memory for multimodal data transport to avoid D2H memory copies and significantly improve E2E latency. **Example with full multimodal optimizations:** ```bash Command theme={null} SGLANG_USE_CUDA_IPC_TRANSPORT=1 \ SGLANG_VLM_CACHE_SIZE_MB=0 \ python -m sglang.launch_server \ --model-path zai-org/GLM-4.5V \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code \ --tp-size 8 \ --enable-cache-report \ --log-level info \ --max-running-requests 64 \ --mem-fraction-static 0.65 \ --chunked-prefill-size 8192 \ --attention-backend fa3 \ --mm-attention-backend fa3 \ --mm-enable-dp-encoder \ --enable-metrics ``` ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Multi-Modal Inputs GLM-4.5V supports both image and video inputs. Here's a basic example with image input: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Describe this image in detail." } ] } ] start = time.time() response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 3.37s Generated text: Auntie Anne's CINNAMON SUGAR 1 x 17,000 17,000 SUB TOTAL 17,000 GRAND TOTAL 17,000 CASH IDR 20,000 CHANGE DUE 3,000 ``` **Multi-Image Input Example:** GLM-4.5V can process multiple images in a single request for comparison or analysis: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg" } }, { "type": "image_url", "image_url": { "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg" } }, { "type": "text", "text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image." } ] } ] start = time.time() response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 3.86s Generated text: The first image shows a close - up of a few red taxis on a street with storefronts in the background. The taxis are in a line, and the scene has an urban, busy feel with visible shop displays. The second image is an aerial view of a large taxi parking area with numerous red and green taxis, some with hoods open. The scene is more open, with a parking lot layout, and includes elements like a bridge and grassy areas. Key differences: number of taxis (few vs many), perspective (close - up vs aerial), color variety (mostly red vs red and green), and setting (street with shops vs parking lot). ``` **Video Input Example:** GLM-4.5V supports video understanding by processing video URLs: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "video_url", "video_url": { "url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4" } }, { "type": "text", "text": "Describe what happens in this video." } ] } ] start = time.time() response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Note:** * For video processing, ensure you have sufficient context length configured (up to 64K tokens) * Video processing may require more memory; adjust `--mem-fraction-static` accordingly * You can also provide local file paths using `file://` protocol **Example Output:** ```text Output theme={null} Response costs: 3.89s Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen. ``` #### 4.2.2 Thinking Mode GLM-4.5V supports thinking mode for enhanced reasoning. Enable thinking mode during deployment: ```shell Command theme={null} python -m sglang.launch_server \ --model-path zai-org/GLM-4.5V \ --reasoning-parser glm45 \ --tp 4 \ --host 0.0.0.0 \ --port 30000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. **Disable Thinking Mode:** To disable thinking mode for a specific request: ```python Example theme={null} response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=[{"role": "user", "content": "What is the capital of France?"}], extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) ``` #### 4.2.3 Tool Calling GLM-4.5V supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model-path zai-org/GLM-4.5V \ --reasoning-parser glm45 \ --tool-call-parser glm45 \ --tp 4 \ --host 0.0.0.0 \ --port 30000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.4 Thinking Budget Beyond enabling/disabling the full reasoning mode (section 4.2.2), you can cap the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor` and pass `Glm4MoeThinkingBudgetLogitProcessor` in the request: ```python Example theme={null} import openai from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*") response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=[{"role": "user", "content": "Describe this image briefly."}], max_tokens=1024, extra_body={ "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(), "custom_params": {"thinking_budget": 512}, }, ) print(response) ``` ## 5. Benchmark ### 5.1 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.1.1 MMMU Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64 ``` * Test Result ```text Output theme={null} Benchmark time: 616.6163094160147 answers saved to: ./answer_sglang.json Evaluating... answers saved to: ./answer_sglang.json {'Accounting': {'acc': 0.867, 'num': 30}, 'Agriculture': {'acc': 0.567, 'num': 30}, 'Architecture_and_Engineering': {'acc': 0.667, 'num': 30}, 'Art': {'acc': 0.667, 'num': 30}, 'Art_Theory': {'acc': 0.9, 'num': 30}, 'Basic_Medical_Science': {'acc': 0.8, 'num': 30}, 'Biology': {'acc': 0.6, 'num': 30}, 'Chemistry': {'acc': 0.533, 'num': 30}, 'Clinical_Medicine': {'acc': 0.667, 'num': 30}, 'Computer_Science': {'acc': 0.8, 'num': 30}, 'Design': {'acc': 0.867, 'num': 30}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.667, 'num': 30}, 'Economics': {'acc': 0.833, 'num': 30}, 'Electronics': {'acc': 0.433, 'num': 30}, 'Energy_and_Power': {'acc': 0.733, 'num': 30}, 'Finance': {'acc': 0.767, 'num': 30}, 'Geography': {'acc': 0.667, 'num': 30}, 'History': {'acc': 0.8, 'num': 30}, 'Literature': {'acc': 0.9, 'num': 30}, 'Manage': {'acc': 0.733, 'num': 30}, 'Marketing': {'acc': 0.9, 'num': 30}, 'Materials': {'acc': 0.567, 'num': 30}, 'Math': {'acc': 0.8, 'num': 30}, 'Mechanical_Engineering': {'acc': 0.767, 'num': 30}, 'Music': {'acc': 0.3, 'num': 30}, 'Overall': {'acc': 0.732, 'num': 900}, 'Overall-Art and Design': {'acc': 0.683, 'num': 120}, 'Overall-Business': {'acc': 0.82, 'num': 150}, 'Overall-Health and Medicine': {'acc': 0.787, 'num': 150}, 'Overall-Humanities and Social Science': {'acc': 0.783, 'num': 120}, 'Overall-Science': {'acc': 0.707, 'num': 150}, 'Overall-Tech and Engineering': {'acc': 0.648, 'num': 210}, 'Pharmacy': {'acc': 0.9, 'num': 30}, 'Physics': {'acc': 0.933, 'num': 30}, 'Psychology': {'acc': 0.767, 'num': 30}, 'Public_Health': {'acc': 0.9, 'num': 30}, 'Sociology': {'acc': 0.667, 'num': 30}} eval out saved to ./val_sglang.json Overall accuracy: 0.732 ``` # GLM-4.6 Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.6 ## 1. Model Introduction [GLM-4.6](https://huggingface.co/zai-org/GLM-4.6) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding. As the latest iteration in the GLM series, GLM-4.6 achieves comprehensive enhancements across multiple domains, including real-world coding, long-context processing, reasoning, searching, writing, and agentic applications. Details are as follows: * **Longer context window**: The context window has been expanded from 128K to 200K tokens, enabling the model to handle more complex agentic tasks. * **Superior coding performance**: The model achieves higher scores on code benchmarks and demonstrates better real-world performance in applications such as Claude Code, Cline, Roo Code and Kilo Code, including improvements in generating visually polished front-end pages. * **Advanced reasoning**: GLM-4.6 shows a clear improvement in reasoning performance and supports tool use during inference, leading to stronger overall capability. * **More capable agents**: GLM-4.6 exhibits stronger performance in tool use and search-based agents, and integrates more effectively within agent frameworks. * **Refined writing**: Better aligns with human preferences in style and readability, and performs more naturally in role-playing scenarios. For more details, please refer to the [official GLM-4.6 documentation](https://docs.z.ai/guides/llm/glm-4.6). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips * **EAGLE Speculative Decoding:** Supported for GLM-4.5/4.6. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. * **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3). ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser GLM-4.6 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.6 \ --reasoning-parser glm45 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/GLM-4.6", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling **Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation. GLM-4.6 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.6 \ --reasoning-parser glm45 \ --tool-call-parser glm45 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-4.6", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"🔧 Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="zai-org/GLM-4.6", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.3 Thinking Budget Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`: ```python Example theme={null} import openai from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*") response = client.chat.completions.create( model="zai-org/GLM-4.6", messages=[{"role": "user", "content": "Is Paris the Capital of France?"}], max_tokens=1024, extra_body={ "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(), "custom_params": {"thinking_budget": 512}, }, ) print(response) ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (8x), AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x) * Model: GLM-4.6 * Tensor Parallelism: 8 * SGLang Version: 0.5.6.post1 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Test Scenarios Three core scenarios reflect real-world usage patterns:
Scenario Input Length Output Length Use Case
**Chat** 1K 1K Most common conversational AI workload
**Reasoning** 1K 8K Long-form generation, complex reasoning tasks
**Summarization** 8K 1K Document summarization, RAG retrieval
#### 5.1.2 Concurrency Levels Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier): * **Low Concurrency**: `--max-concurrency 1` (Latency-optimized) * **Medium Concurrency**: `--max-concurrency 16` (Balanced) * **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) #### 5.1.3 Number of Prompts For each concurrency level, configure `num_prompts` to simulate realistic user loads: * **Quick Test**: `num_prompts = concurrency × 1` (minimal test) * **Recommended**: `num_prompts = concurrency × 5` (standard benchmark) * **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade) *** #### 5.1.4 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.6 \ --tp 8 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 63.82 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4210 Total generated tokens (retokenized): 4209 Request throughput (req/s): 0.16 Input token throughput (tok/s): 95.60 Output token throughput (tok/s): 65.97 Peak output token throughput (tok/s): 68.00 Peak concurrent requests: 2 Total token throughput (tok/s): 161.57 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6379.24 Median E2E Latency (ms): 5085.00 ---------------Time to First Token---------------- Mean TTFT (ms): 155.57 Median TTFT (ms): 149.79 P99 TTFT (ms): 207.69 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.81 Median TPOT (ms): 14.80 P99 TPOT (ms): 14.84 ---------------Inter-Token Latency---------------- Mean ITL (ms): 14.82 Median ITL (ms): 14.82 P95 ITL (ms): 15.17 P99 ITL (ms): 15.36 Max ITL (ms): 25.05 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 72.06 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40725 Total generated tokens (retokenized): 40672 Request throughput (req/s): 1.11 Input token throughput (tok/s): 550.47 Output token throughput (tok/s): 565.14 Peak output token throughput (tok/s): 752.00 Peak concurrent requests: 20 Total token throughput (tok/s): 1115.61 Concurrency: 13.71 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 12348.93 Median E2E Latency (ms): 13164.81 ---------------Time to First Token---------------- Mean TTFT (ms): 196.08 Median TTFT (ms): 155.22 P99 TTFT (ms): 377.98 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 24.24 Median TPOT (ms): 24.55 P99 TPOT (ms): 30.42 ---------------Inter-Token Latency---------------- Mean ITL (ms): 23.92 Median ITL (ms): 21.40 P95 ITL (ms): 22.49 P99 ITL (ms): 123.83 Max ITL (ms): 486.54 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 138.50 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252162 Total generated tokens (retokenized): 251841 Request throughput (req/s): 3.61 Input token throughput (tok/s): 1803.78 Output token throughput (tok/s): 1820.61 Peak output token throughput (tok/s): 2900.00 Peak concurrent requests: 107 Total token throughput (tok/s): 3624.40 Concurrency: 90.91 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 25183.97 Median E2E Latency (ms): 23968.49 ---------------Time to First Token---------------- Mean TTFT (ms): 337.77 Median TTFT (ms): 180.65 P99 TTFT (ms): 906.14 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 49.97 Median TPOT (ms): 52.20 P99 TPOT (ms): 61.81 ---------------Inter-Token Latency---------------- Mean ITL (ms): 49.36 Median ITL (ms): 35.05 P95 ITL (ms): 124.91 P99 ITL (ms): 187.69 Max ITL (ms): 440.34 ================================================== ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 666.64 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 44452 Total generated tokens (retokenized): 44387 Request throughput (req/s): 0.02 Input token throughput (tok/s): 9.15 Output token throughput (tok/s): 66.68 Peak output token throughput (tok/s): 68.00 Peak concurrent requests: 2 Total token throughput (tok/s): 75.83 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 66661.35 Median E2E Latency (ms): 71902.36 ---------------Time to First Token---------------- Mean TTFT (ms): 160.21 Median TTFT (ms): 140.32 P99 TTFT (ms): 295.56 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.92 Median TPOT (ms): 14.94 P99 TPOT (ms): 15.02 ---------------Inter-Token Latency---------------- Mean ITL (ms): 14.96 Median ITL (ms): 14.96 P95 ITL (ms): 15.36 P99 ITL (ms): 15.57 Max ITL (ms): 19.06 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 503.30 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 318226 Total generated tokens (retokenized): 318025 Request throughput (req/s): 0.16 Input token throughput (tok/s): 78.82 Output token throughput (tok/s): 632.28 Peak output token throughput (tok/s): 752.00 Peak concurrent requests: 19 Total token throughput (tok/s): 711.09 Concurrency: 13.88 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 87349.22 Median E2E Latency (ms): 88248.04 ---------------Time to First Token---------------- Mean TTFT (ms): 228.54 Median TTFT (ms): 142.78 P99 TTFT (ms): 569.84 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 21.97 Median TPOT (ms): 22.14 P99 TPOT (ms): 22.47 ---------------Inter-Token Latency---------------- Mean ITL (ms): 21.91 Median ITL (ms): 21.80 P95 ITL (ms): 22.30 P99 ITL (ms): 22.78 Max ITL (ms): 137.19 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 772.28 Total input tokens: 158939 Total input text tokens: 158939 Total input vision tokens: 0 Total generated tokens: 1300705 Total generated tokens (retokenized): 1299924 Request throughput (req/s): 0.41 Input token throughput (tok/s): 205.80 Output token throughput (tok/s): 1684.24 Peak output token throughput (tok/s): 2112.00 Peak concurrent requests: 68 Total token throughput (tok/s): 1890.05 Concurrency: 56.17 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 135563.36 Median E2E Latency (ms): 140888.88 ---------------Time to First Token---------------- Mean TTFT (ms): 232.45 Median TTFT (ms): 145.59 P99 TTFT (ms): 576.49 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 33.47 Median TPOT (ms): 34.02 P99 TPOT (ms): 35.10 ---------------Inter-Token Latency---------------- Mean ITL (ms): 33.30 Median ITL (ms): 32.63 P95 ITL (ms): 34.27 P99 ITL (ms): 104.39 Max ITL (ms): 155.65 ================================================== ``` **Scenario 3: Summarization (8K/1K)** * Low ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 65.11 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4210 Total generated tokens (retokenized): 4210 Request throughput (req/s): 0.15 Input token throughput (tok/s): 644.17 Output token throughput (tok/s): 64.66 Peak output token throughput (tok/s): 68.00 Peak concurrent requests: 2 Total token throughput (tok/s): 708.83 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6508.31 Median E2E Latency (ms): 5263.36 ---------------Time to First Token---------------- Mean TTFT (ms): 189.48 Median TTFT (ms): 159.23 P99 TTFT (ms): 304.09 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 15.02 Median TPOT (ms): 15.03 P99 TPOT (ms): 15.27 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.04 Median ITL (ms): 15.03 P95 ITL (ms): 15.46 P99 ITL (ms): 15.65 Max ITL (ms): 24.20 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 76.43 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41589 Total generated tokens (retokenized): 41577 Request throughput (req/s): 1.05 Input token throughput (tok/s): 3925.47 Output token throughput (tok/s): 544.15 Peak output token throughput (tok/s): 752.00 Peak concurrent requests: 19 Total token throughput (tok/s): 4469.62 Concurrency: 13.95 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 13329.63 Median E2E Latency (ms): 14141.09 ---------------Time to First Token---------------- Mean TTFT (ms): 339.88 Median TTFT (ms): 252.75 P99 TTFT (ms): 906.54 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 25.37 Median TPOT (ms): 25.73 P99 TPOT (ms): 30.94 ---------------Inter-Token Latency---------------- Mean ITL (ms): 25.04 Median ITL (ms): 21.68 P95 ITL (ms): 22.69 P99 ITL (ms): 146.98 Max ITL (ms): 483.14 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 136.24 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 169680 Total generated tokens (retokenized): 169452 Request throughput (req/s): 2.35 Input token throughput (tok/s): 9350.32 Output token throughput (tok/s): 1245.44 Peak output token throughput (tok/s): 1984.00 Peak concurrent requests: 69 Total token throughput (tok/s): 10595.77 Concurrency: 58.46 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 24889.40 Median E2E Latency (ms): 25123.37 ---------------Time to First Token---------------- Mean TTFT (ms): 355.82 Median TTFT (ms): 268.84 P99 TTFT (ms): 858.64 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 46.62 Median TPOT (ms): 49.04 P99 TPOT (ms): 58.88 ---------------Inter-Token Latency---------------- Mean ITL (ms): 46.36 Median ITL (ms): 32.46 P95 ITL (ms): 135.23 P99 ITL (ms): 204.27 Max ITL (ms): 508.14 ================================================== ``` #### 5.1.5 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * Test Result ```text Output theme={null} Accuracy: 0.975 Invalid: 0.000 Latency: 16.574 s Output throughput: 1194.637 token/s ``` # GLM-4.6V Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.6V ## 1. Model Introduction GLM-4.6V series model includes two versions: GLM-4.6V (106B), a foundation model designed for cloud and high-performance cluster scenarios, and GLM-4.6V-Flash (9B), a lightweight model optimized for local deployment and low-latency applications. GLM-4.6V scales its context window to 128k tokens in training, and achieves SoTA performance in visual understanding among models of similar parameter scales. Crucially, GLM team integrated native Function Calling capabilities for the first time. This effectively bridges the gap between "visual perception" and "executable action" providing a unified technical foundation for multimodal agents in real-world business scenarios. Beyond achieves SoTA performance across major multimodal benchmarks at comparable model scales. GLM-4.6V introduces several key features: * **Native Multimodal Function Calling** Enables native vision-driven tool use. Images, screenshots, and document pages can be passed directly as tool inputs without text conversion, while visual outputs (charts, search images, rendered pages) are interpreted and integrated into the reasoning chain. This closes the loop from perception to understanding to execution. Please refer to this [example](#4-2-3-tool-calling). * **Interleaved Image-Text Content Generation** Supports high-quality mixed media creation from complex multimodal inputs. GLM-4.6V takes a multimodal context—spanning documents, user inputs, and tool-retrieved images—and synthesizes coherent, interleaved image-text content tailored to the task. During generation it can actively call search and retrieval tools to gather and curate additional text and visuals, producing rich, visually grounded content. * **Multimodal Document Understanding** GLM-4.6V can process up to 128K tokens of multi-document or long-document input, directly interpreting richly formatted pages as images. It understands text, layout, charts, tables, and figures jointly, enabling accurate comprehension of complex, image-heavy documents without requiring prior conversion to plain text. * **Frontend Replication & Visual Editing** Reconstructs pixel-accurate HTML/CSS from UI screenshots and supports natural-language-driven edits. It detects layout, components, and styles visually, generates clean code, and applies iterative visual modifications through simple user instructions. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. ### 2.1 Docker Installation (Recommended) ```shell Command theme={null} docker pull lmsysorg/sglang:latest ``` **Advantages:** * Ready to use out of the box, no manual environment configuration needed * Avoids dependency conflict issues * Easy to migrate between different environments ### 2.2 Build from Source If you need to use the latest development version or require custom modifications, you can build from source: ```bash Command theme={null} # Install SGLang using UV (recommended) git clone https://github.com/sgl-project/sglang.git cd sglang uv venv source .venv/bin/activate uv pip install --prerelease=allow -e "python[all]" --index-url=https://pypi.org/simple pip install nvidia-cudnn-cu12==9.16.0.29 # Install ffmpeg to support video input sudo apt update sudo apt install ffmpeg ``` **Use Cases:** * Need to customize and modify SGLang source code * Want to use the latest development features * Participate in SGLang project development For general installation instructions, you can also refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command. ### 3.2 Configuration Tips * **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size \* number of images in current running requests.) * **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder` (which the generator above handles automatically). * **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`. * **Hardware Notes:** * **H100 (FP8):** Use the FP8 checkpoint for best memory efficiency. * **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage. * **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing. * **Additional Multimodal Parameters:** * `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3). * `--mm-feature-transport=cuda_ipc`: Use CUDA IPC shared memory for multimodal data transport to avoid D2H memory copies and significantly improve E2E latency. **Example with full multimodal optimizations:** ```bash Command theme={null} SGLANG_USE_CUDA_IPC_TRANSPORT=1 \ SGLANG_VLM_CACHE_SIZE_MB=0 \ python -m sglang.launch_server \ --model-path zai-org/GLM-4.6V \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code \ --tp-size 8 \ --enable-cache-report \ --log-level info \ --max-running-requests 64 \ --mem-fraction-static 0.65 \ --chunked-prefill-size 8192 \ --attention-backend fa3 \ --mm-attention-backend fa3 \ --mm-enable-dp-encoder \ --enable-metrics ``` ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Multi-Modal Inputs GLM-4.6V supports image and video inputs via the OpenAI-compatible API. **Image Input:** ```python Example theme={null} import subprocess curl_command = f""" curl -s http://localhost:{30000}/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{{ "model": "default", "messages": [ {{ "role": "user", "content": [ {{ "type": "image_url", "image_url": {{ "url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true" }} }}, {{ "type": "text", "text": "What is the image" }} ] }} ], "temperature": "0", "max_completion_tokens": "1000", "max_tokens": "1000" }}' """ response = subprocess.check_output(curl_command, shell=True).decode() print(response) ``` ```text Output theme={null} {"id":"b61596ca71394dd699fd8abd4f650c44","object":"chat.completion","created":1765259019,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a logo featuring the text \"SGL\" (in a bold, orange-brown font) alongside a stylized icon. The icon includes a network-like structure with circular nodes (suggesting connectivity or a tree/graph structure) and a tag with \"\" (a common symbol for coding, web development, or software). The color scheme uses warm orange-brown tones with a black background, giving it a tech-focused, modern aesthetic (likely representing a company, project, or tool related to software, web development, or digital technology).<|begin_of_box|>SGL logo (stylized text + network/coding icon)<|end_of_box|>","reasoning_content":"Okay, let's see. The image has a logo with the text \"SGL\" and a little icon on the left. The icon looks like a network or a tree structure with circles, and there's a tag with \"\" which is a common symbol for coding or web development. The colors are orange and brown tones, with a black background. So probably a logo for a company or project named SGL, maybe related to software, web development, or a tech company.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":2222,"total_tokens":2448,"completion_tokens":226,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}} ``` **Video Input:** ```python Example theme={null} import subprocess curl_command = f""" curl -s http://localhost:{30000}/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{{ "model": "default", "messages": [ {{ "role": "user", "content": [ {{ "type": "video_url", "video_url": {{ "url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4" }} }}, {{ "type": "text", "text": "What is in the video" }} ] }} ], "temperature": "0", "max_completion_tokens": "1000", "max_tokens": "1000" }}' """ response = subprocess.check_output(curl_command, shell=True).decode() print(response) ``` ```text Output theme={null} {"id":"520e0a079e5d4b17b82a6af619315a97","object":"chat.completion","created":1765259029,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a still from a presentation by a man on a stage. He is pointing to a small pocket on his jeans and asking the audience what the pocket is for. The video is being shared by Evan Carmichael. The man then reveals that the pocket is for an iPod Nano.","reasoning_content":"Based on the visual evidence in the video, here is a breakdown of what is being shown:\n\n* **Subject:** The video features a man on a stage, giving a presentation. He is wearing a black t-shirt and dark jeans.\n* **Action:** The man is pointing to a pocket on his jeans. He is asking the audience a question about the purpose of this pocket.\n* **Context:** The presentation is being filmed, and the video is being shared by \"Evan Carmichael,\" a well-known motivational speaker and content creator. The source of the clip is credited to \"JoshuaG.\"\n* **Reveal:** The man then reveals the answer to his question. He pulls a small, white, rectangular device out of the pocket. He identifies this device as an \"iPod Nano.\"\n\nIn summary, the image is a still from a presentation where a speaker is explaining the purpose of the small pocket found on many pairs of jeans.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":30276,"total_tokens":30532,"completion_tokens":256,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}} ``` #### 4.2.2 Thinking Mode GLM-4.6V supports Thinking mode. Enable the reasoning parser during deployment: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.6V \ --reasoning-parser glm45 \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/GLM-4.6V", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.3 Tool Calling GLM-4.6V supports tool calling with vision capabilities. Pass tools in your API request: ```python Example theme={null} from openai import OpenAI openai_api_key = "EMPTY" openai_api_base = "http://127.0.0.1:30000/v1" client = OpenAI(api_key=openai_api_key, base_url=openai_api_base) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get current temperature for a given location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "City and country e.g. Beijing, China", } }, "required": ["location"], "additionalProperties": False, }, }, } ] messages = [ { "role": "user", "content": "Please help me check today's weather in Beijing, and tell me whether the tool returned an image." }, { "role": "assistant", "tool_calls": [ { "id": "call_bk32t88BGpSdbtDgzT044Rh4", "type": "function", "function": { "name": 'get_weather', "arguments": '{"location":"Beijing, China"}' } } ] }, { "role": "tool", "tool_call_id": "call_bk32t88BGpSdbtDgzT044Rh4", "content": [ { "type": "text", "text": "Weather report generated: Beijing, November 7, 2025, sunny, temperature 2°C." }, { "type": "image_url", "image_url": { "url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true" } } ] }, ] response = client.chat.completions.create( model="zai-org/GLM-4.6V", messages=messages, timeout=900, tools=tools ) print(response.choices[0].message.content.strip()) ``` **Output Example:** ```text Output theme={null} The weather in Beijing today (November 7, 2025) is sunny with a temperature of 2°C. Yes, the tool returned an image (the SGL logo). ``` #### 4.2.4 Thinking Budget Beyond the reasoning parser, you can cap the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor` and pass `Glm4MoeThinkingBudgetLogitProcessor` in the request — same as the [GLM-4.6 text model approach](./GLM-4.6#4-2-3-thinking-budget): ```python Example theme={null} import openai from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*") response = client.chat.completions.create( model="zai-org/GLM-4.6V", messages=[{"role": "user", "content": "Describe this image briefly."}], max_tokens=1024, extra_body={ "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(), "custom_params": {"thinking_budget": 512}, }, ) print(response) ``` ## 5. Benchmark ### 5.1. Text Benchmark: Latency, Throughput and Accuracy #### Command ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. #### Result Output ```text Output theme={null} Accuracy: 0.925 Invalid: 0.000 Latency: 15.327 s Output throughput: 1788.375 token/s ``` ### 5.2. Multimodal Benchmark - Latency and Throughput #### Command ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --port 30000 \ --model zai-org/GLM-4.6V \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 128 \ --max-concurrency 8 ``` #### Result Output ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 8 Successful requests: 128 Benchmark duration (s): 89.27 Total input tokens: 315390 Total input text tokens: 8702 Total input vision tokens: 306688 Total generated tokens: 66020 Total generated tokens (retokenized): 31037 Request throughput (req/s): 1.43 Input token throughput (tok/s): 3533.17 Output token throughput (tok/s): 739.59 Peak output token throughput (tok/s): 823.00 Peak concurrent requests: 12 Total token throughput (tok/s): 4272.76 Concurrency: 7.67 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5349.20 Median E2E Latency (ms): 5380.98 ---------------Time to First Token---------------- Mean TTFT (ms): 1724.04 Median TTFT (ms): 1688.16 P99 TTFT (ms): 6152.34 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.15 Median TPOT (ms): 7.77 P99 TPOT (ms): 23.97 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.00 Median ITL (ms): 8.44 P95 ITL (ms): 9.23 P99 ITL (ms): 116.02 Max ITL (ms): 173.48 ================================================== ``` ### 5.3. Multimodal Accuracy Benchmark - MMMU #### Command ```shell Command theme={null} python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64 --extra-request-body '{"max_tokens": 4096}' ``` #### Result Output ```text Output theme={null} Benchmark time: 487.2229107860476 answers saved to: ./answer_sglang.json Evaluating... answers saved to: ./answer_sglang.json {'Accounting': {'acc': 0.962, 'num': 26}, 'Agriculture': {'acc': 0.5, 'num': 30}, 'Architecture_and_Engineering': {'acc': 0.733, 'num': 15}, 'Art': {'acc': 0.833, 'num': 30}, 'Art_Theory': {'acc': 0.9, 'num': 30}, 'Basic_Medical_Science': {'acc': 0.733, 'num': 30}, 'Biology': {'acc': 0.586, 'num': 29}, 'Chemistry': {'acc': 0.654, 'num': 26}, 'Clinical_Medicine': {'acc': 0.633, 'num': 30}, 'Computer_Science': {'acc': 0.76, 'num': 25}, 'Design': {'acc': 0.867, 'num': 30}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.633, 'num': 30}, 'Economics': {'acc': 0.862, 'num': 29}, 'Electronics': {'acc': 0.5, 'num': 18}, 'Energy_and_Power': {'acc': 0.875, 'num': 16}, 'Finance': {'acc': 0.857, 'num': 28}, 'Geography': {'acc': 0.714, 'num': 28}, 'History': {'acc': 0.767, 'num': 30}, 'Literature': {'acc': 0.897, 'num': 29}, 'Manage': {'acc': 0.759, 'num': 29}, 'Marketing': {'acc': 1.0, 'num': 26}, 'Materials': {'acc': 0.833, 'num': 18}, 'Math': {'acc': 0.76, 'num': 25}, 'Mechanical_Engineering': {'acc': 0.619, 'num': 21}, 'Music': {'acc': 0.286, 'num': 28}, 'Overall': {'acc': 0.761, 'num': 803}, 'Overall-Art and Design': {'acc': 0.729, 'num': 118}, 'Overall-Business': {'acc': 0.884, 'num': 138}, 'Overall-Health and Medicine': {'acc': 0.773, 'num': 150}, 'Overall-Humanities and Social Science': {'acc': 0.78, 'num': 118}, 'Overall-Science': {'acc': 0.728, 'num': 136}, 'Overall-Tech and Engineering': {'acc': 0.671, 'num': 143}, 'Pharmacy': {'acc': 0.933, 'num': 30}, 'Physics': {'acc': 0.929, 'num': 28}, 'Psychology': {'acc': 0.733, 'num': 30}, 'Public_Health': {'acc': 0.933, 'num': 30}, 'Sociology': {'acc': 0.724, 'num': 29}} eval out saved to ./val_sglang.json Overall accuracy: 0.761 ``` # GLM-4.7 Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.7 ## 1. Model Introduction [GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and agent workflows. GLM-4.7 brings improvements across all major domains: * **Extended Context Window**: Expanded context window supporting even longer documents and complex multi-turn conversations * **Enhanced Reasoning**: Improved reasoning capabilities with better chain-of-thought processing * **Superior Coding**: Significantly improved code generation and understanding, with better real-world application performance * **Advanced Tool Use**: More robust tool calling and agent capabilities for complex workflows * **Optimized Performance**: Better throughput and latency characteristics across all hardware platforms For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7). **Key Features:** * **State-of-the-Art Reasoning**: Enhanced reasoning capabilities for the most complex problem-solving tasks * **Multiple Quantizations**: BF16, FP8, and NVFP4 variants for different performance/memory trade-offs * **Hardware Optimization**: Tuned for NVIDIA Blackwell (B200, GB200) and AMD MI300X/MI325X/MI355X GPUs * **High Performance**: Optimized for both throughput and latency scenarios **Available Models:** * **BF16 (Full precision)**: [zai-org/GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) * **FP8 (8-bit quantized)**: [zai-org/GLM-4.7-FP8](https://huggingface.co/zai-org/GLM-4.7-FP8) * **NVFP4 (4-bit, NVIDIA Blackwell)**: [nvidia/GLM-4.7-NVFP4](https://huggingface.co/nvidia/GLM-4.7-NVFP4) **License:** Please refer to the [official GLM-4.7 model card](https://huggingface.co/zai-org/GLM-4.7) for license details. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. **Docker Images by Hardware Platform:**
Hardware Platform Docker Image
NVIDIA H100 / H200 / B200 `lmsysorg/sglang:v0.5.12`
NVIDIA GB200 / B300 / GB300 (aarch64) `lmsysorg/sglang:v0.5.12-cu130`
AMD MI300X / MI325X `lmsysorg/sglang:v0.5.12-rocm720-mi30x`
AMD MI355X `lmsysorg/sglang:v0.5.12-rocm720-mi35x`
## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips Pick a weight format by hardware: **NVFP4** on NVIDIA Blackwell (B200, GB200), **FP8** on H100/H200/AMD, **BF16** as the full-precision fallback. The recommended tensor-parallel size per platform:
Hardware NVFP4 FP8 BF16
B200 (8×, single node) tp=2 / 4 / 8 tp=4 / 8 tp=8
GB200 (NVL72, 4× per tray) tp=2 / 4 tp=4 —
H200 (8×) — tp=8 tp=8
AMD MI300X / MI325X / MI355X — tp=2 / 4 / 8 tp=4 / 8
* **EAGLE Speculative Decoding:** Supported for GLM-4.7. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. Enable via the interactive command generator above. * **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3). For general GLM-4.x family launch guidance (AMD ROCm notes and more), see [Launch GLM-4.5 / GLM-4.6 / GLM-4.7 with SGLang](/cookbook/autoregressive/GLM/GLM-4.5). Per-hardware bench commands and flags are inline in §5.1 below. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser GLM-4.7 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.7 \ --reasoning-parser glm45 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/GLM-4.7", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling **Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation. GLM-4.7 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.7 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-4.7", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="zai-org/GLM-4.7", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.3 Thinking Budget Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`: ```python Example theme={null} import openai from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*") response = client.chat.completions.create( model="zai-org/GLM-4.7", messages=[{"role": "user", "content": "Is Paris the Capital of France?"}], max_tokens=1024, extra_body={ "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(), "custom_params": {"thinking_budget": 512}, }, ) print(response) ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200, NVIDIA GB200, AMD MI300X/MI325X/MI355X (8x) * Model: GLM-4.7-NVFP4 on NVIDIA Blackwell; GLM-4.7-FP8 or GLM-4.7 (BF16) on AMD * SGLang Version: 0.5.12 (NVIDIA Blackwell), 0.5.6.post1 (AMD) * Best per-GPU throughput config on B200: **TP=2 NVFP4 bf16-KV** (NVFP4 weights, no EP). Numbers below come from this config. **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Test Scenarios Four core scenarios reflect real-world usage patterns:
Scenario Input Length Output Length Use Case
**Chat** 1K 1K Most common conversational AI workload
**Reasoning** 1K 8K Long-form generation, complex reasoning tasks
**Summarization** 8K 1K Document summarization, RAG retrieval
**Throughput** 4K 1K Mixed RAG / agent / multi-turn conversation (used for the inline B200 / GB200 results below)
#### 5.1.2 Concurrency Levels Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier): * **Low Concurrency**: `--max-concurrency 1` (Latency-optimized) * **Medium Concurrency**: `--max-concurrency 16` (Balanced) * **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) — the Throughput (4K/1K) scenario uses `--max-concurrency 128` to match the inline B200/GB200 results below. #### 5.1.3 Number of Prompts For each concurrency level, configure `num_prompts` to simulate realistic user loads: * **Quick Test**: `num_prompts = concurrency × 1` (minimal test) * **Recommended**: `num_prompts = concurrency × 5` (standard benchmark) * **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade) *** #### 5.1.4 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.7 \ --tp 8 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` **Scenario 4: Throughput (4K/1K) — NVIDIA Blackwell with NVFP4** The remaining sub-sections (§5.1.4.1 NVIDIA B200, §5.1.4.2 NVIDIA GB200) measure this scenario with `nvidia/GLM-4.7-NVFP4` weights and report the full `bench_serving` output verbatim. The same commands apply to other NVIDIA hardware after substituting the deployment line from §3.1. > **Note**: These runs use EOS-enabled generation (no `--disable-ignore-eos`), so generated-token counts reflect natural model behavior rather than a strict fixed-OSL pin. Compare against other EOS-enabled runs at the same workload, not against fixed-output-length benchmarks. #### 5.1.4.1 NVIDIA B200 **Model Deployment (NVIDIA B200, TP=2 NVFP4 — max tok/s/gpu config):** ```bash Command theme={null} python -m sglang.launch_server \ --model nvidia/GLM-4.7-NVFP4 \ --tp-size 2 \ --mem-fraction-static 0.85 \ --reasoning-parser glm45 \ --tool-call-parser glm47 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model nvidia/GLM-4.7-NVFP4 \ --dataset-name random \ --random-input-len 4096 \ --random-output-len 1024 \ --num-prompts 5 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 1 Successful requests: 5 Benchmark duration (s): 25.07 Total input tokens: 8105 Total generated tokens: 2674 Request throughput (req/s): 0.20 Input token throughput (tok/s): 323.25 Output token throughput (tok/s): 106.65 Total token throughput (tok/s): 429.90 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5011.93 Median E2E Latency (ms): 6441.44 ---------------Time to First Token---------------- Mean TTFT (ms): 179.61 Median TTFT (ms): 169.05 P99 TTFT (ms): 238.01 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.05 Median TPOT (ms): 9.03 P99 TPOT (ms): 9.16 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.05 Median ITL (ms): 9.05 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model nvidia/GLM-4.7-NVFP4 \ --dataset-name random \ --random-input-len 4096 \ --random-output-len 1024 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 60.60 Total input tokens: 179772 Total generated tokens: 39657 Request throughput (req/s): 1.32 Input token throughput (tok/s): 2966.39 Output token throughput (tok/s): 654.37 Total token throughput (tok/s): 3620.76 Concurrency: 14.01 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 10615.87 Median E2E Latency (ms): 9985.45 ---------------Time to First Token---------------- Mean TTFT (ms): 267.39 Median TTFT (ms): 177.26 P99 TTFT (ms): 584.29 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 20.98 Median TPOT (ms): 21.06 P99 TPOT (ms): 24.88 ---------------Inter-Token Latency---------------- Mean ITL (ms): 20.92 Median ITL (ms): 17.93 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model nvidia/GLM-4.7-NVFP4 \ --dataset-name random \ --random-input-len 4096 \ --random-output-len 1024 \ --num-prompts 640 \ --max-concurrency 128 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 128 Successful requests: 640 Benchmark duration (s): 172.95 Total input tokens: 1453591 Total generated tokens: 308740 Request throughput (req/s): 3.70 Input token throughput (tok/s): 8404.67 Output token throughput (tok/s): 1785.14 Total token throughput (tok/s): 10189.80 Concurrency: 117.85 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 31848.20 Median E2E Latency (ms): 28554.42 ---------------Time to First Token---------------- Mean TTFT (ms): 1598.40 Median TTFT (ms): 298.88 P99 TTFT (ms): 11015.96 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 65.94 Median TPOT (ms): 65.81 P99 TPOT (ms): 137.73 ---------------Inter-Token Latency---------------- Mean ITL (ms): 62.99 Median ITL (ms): 35.44 ================================================== ``` #### 5.1.4.2 NVIDIA GB200 **Model Deployment (NVIDIA GB200, TP=2 NVFP4 — max tok/s/gpu config):** ```bash Command theme={null} python -m sglang.launch_server \ --model nvidia/GLM-4.7-NVFP4 \ --tp-size 2 \ --mem-fraction-static 0.85 \ --reasoning-parser glm45 \ --tool-call-parser glm47 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model nvidia/GLM-4.7-NVFP4 \ --dataset-name random \ --random-input-len 4096 \ --random-output-len 1024 \ --num-prompts 5 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 1 Successful requests: 5 Benchmark duration (s): 24.74 Total input tokens: 8105 Total generated tokens: 2674 Request throughput (req/s): 0.20 Input token throughput (tok/s): 327.65 Output token throughput (tok/s): 108.10 Total token throughput (tok/s): 435.75 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4944.47 Median E2E Latency (ms): 6347.31 ---------------Time to First Token---------------- Mean TTFT (ms): 211.41 Median TTFT (ms): 207.25 P99 TTFT (ms): 226.46 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.86 Median TPOT (ms): 8.84 P99 TPOT (ms): 8.96 ---------------Inter-Token Latency---------------- Mean ITL (ms): 8.87 Median ITL (ms): 8.85 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model nvidia/GLM-4.7-NVFP4 \ --dataset-name random \ --random-input-len 4096 \ --random-output-len 1024 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 60.40 Total input tokens: 179772 Total generated tokens: 39657 Request throughput (req/s): 1.32 Input token throughput (tok/s): 2976.52 Output token throughput (tok/s): 656.61 Total token throughput (tok/s): 3633.13 Concurrency: 13.97 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 10611.51 Median E2E Latency (ms): 9956.84 ---------------Time to First Token---------------- Mean TTFT (ms): 338.14 Median TTFT (ms): 215.25 P99 TTFT (ms): 915.40 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 20.87 Median TPOT (ms): 21.36 P99 TPOT (ms): 27.05 ---------------Inter-Token Latency---------------- Mean ITL (ms): 20.77 Median ITL (ms): 16.53 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model nvidia/GLM-4.7-NVFP4 \ --dataset-name random \ --random-input-len 4096 \ --random-output-len 1024 \ --num-prompts 640 \ --max-concurrency 128 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 128 Successful requests: 640 Benchmark duration (s): 181.89 Total input tokens: 1453591 Total generated tokens: 309221 Request throughput (req/s): 3.52 Input token throughput (tok/s): 7991.59 Output token throughput (tok/s): 1700.04 Total token throughput (tok/s): 9691.63 Concurrency: 118.86 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 33690.47 Median E2E Latency (ms): 30421.55 ---------------Time to First Token---------------- Mean TTFT (ms): 1353.16 Median TTFT (ms): 383.52 P99 TTFT (ms): 8940.53 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 69.88 Median TPOT (ms): 71.77 P99 TPOT (ms): 131.75 ---------------Inter-Token Latency---------------- Mean ITL (ms): 67.23 Median ITL (ms): 33.46 ================================================== ``` #### 5.1.5 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **4K/1K (Throughput)**: Realistic mixed workload typical of production deployments (RAG context + medium response). Long enough input that prefill matters, long enough output that decode steady-state dominates. Used for the inline B200 / GB200 results above. * **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 1319 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * Test Result (NVIDIA B200, TP=2 NVFP4) ```text Output theme={null} Accuracy: 0.946 Latency: 178.284 s Output throughput: 769.204 token/s ``` * Test Result (NVIDIA GB200, TP=2 NVFP4) ```text Output theme={null} Accuracy: 0.951 Latency: 175.190 s Invalid: 0.000 ``` # GLM-4.7-Flash Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.7-Flash ## 1. Model Introduction [GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) is a lightweight and high-speed model in the GLM-4.7 series developed by Zhipu AI, featuring state-of-the-art capabilities in reasoning, function calling, and efficient local deployment. As a compact variant in the GLM-4.7 family, GLM-4.7-Flash is a **30B-A3B MoE** model designed to balance performance and efficiency: * **Lightweight Architecture**: 30B total parameters with only 3B active parameters, enabling efficient inference * **Enhanced Reasoning**: Inherits the reasoning capabilities from GLM-4.7 with optimized performance * **Superior Coding**: Strong code generation and understanding capabilities * **Advanced Tool Use**: Robust tool calling and agent capabilities for complex workflows * **Optimized for Local Deployment**: Designed for single-GPU deployment scenarios For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7). **Key Features:** * **Efficient MoE Architecture**: 30B-A3B sparse activation for optimal performance/efficiency trade-off * **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs * **Hardware Optimization**: Specifically tuned for NVIDIA H100/H200/B200 GPUs * **High Performance**: Optimized for both throughput and latency scenarios **Available Models:** * **BF16 (Full precision)**: [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) **License:** Please refer to the [official GLM-4.7-Flash model card](https://huggingface.co/zai-org/GLM-4.7-Flash) for license details. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips * **EAGLE Speculative Decoding:** Supported for GLM-4.7-Flash. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. Enable via the interactive command generator above. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser GLM-4.7-Flash supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.7-Flash \ --reasoning-parser glm45 \ --attention-backend triton \ --tp 1 \ --host 0.0.0.0 \ --port 8000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/GLM-4.7-Flash", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling **Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation. GLM-4.7-Flash supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.7-Flash \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --attention-backend triton \ --tp 1 \ --host 0.0.0.0 \ --port 8000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-4.7-Flash", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls (tool call deltas may stream in multiple chunks) if hasattr(delta, 'tool_calls') and delta.tool_calls: for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls if tool_calls_accumulator: print("\n=============== Tool Calls =================", flush=True) for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking for the weather in Beijing. I have the get_weather function available which can provide weather information for a location. The required parameter is "location" and the user has provided "Beijing". There's an optional parameter "unit" for temperature unit, but the user hasn't specified which unit they prefer, and since it's optional, I should not ask about it or make up a value for it. I'll call the function with just the location parameter.I'll check the current weather in Beijing for you. =============== Tool Calls ================= Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="zai-org/GLM-4.7-Flash", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 (1x) * Model: GLM-4.7-Flash * Tensor Parallelism: 1 * SGLang Version: 0.5.7 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Test Scenarios Three core scenarios reflect real-world usage patterns:
Scenario Input Length Output Length Use Case
**Chat** 1K 1K Most common conversational AI workload
**Reasoning** 1K 8K Long-form generation, complex reasoning tasks
**Summarization** 8K 1K Document summarization, RAG retrieval
#### 5.1.2 Concurrency Levels Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier): * **Low Concurrency**: `--max-concurrency 1` (Latency-optimized) * **Medium Concurrency**: `--max-concurrency 16` (Balanced) * **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) #### 5.1.3 Number of Prompts For each concurrency level, configure `num_prompts` to simulate realistic user loads: * **Quick Test**: `num_prompts = concurrency × 1` (minimal test) * **Recommended**: `num_prompts = concurrency × 5` (standard benchmark) * **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade) *** #### 5.1.4 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.7-Flash \ --attention-backend triton \ --tp 1 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 38.94 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.26 Input token throughput (tok/s): 156.67 Output token throughput (tok/s): 108.37 Peak output token throughput (tok/s): 125.00 Peak concurrent requests: 2 Total token throughput (tok/s): 265.03 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3891.12 Median E2E Latency (ms): 3061.48 P90 E2E Latency (ms): 7172.25 P99 E2E Latency (ms): 9042.62 ---------------Time to First Token---------------- Mean TTFT (ms): 131.36 Median TTFT (ms): 94.55 P99 TTFT (ms): 435.93 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.75 Median TPOT (ms): 8.82 P99 TPOT (ms): 9.39 ---------------Inter-Token Latency---------------- Mean ITL (ms): 8.93 Median ITL (ms): 8.98 P95 ITL (ms): 9.83 P99 ITL (ms): 10.20 Max ITL (ms): 18.50 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 52.73 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40775 Request throughput (req/s): 1.52 Input token throughput (tok/s): 752.27 Output token throughput (tok/s): 773.83 Peak output token throughput (tok/s): 1040.00 Peak concurrent requests: 21 Total token throughput (tok/s): 1526.10 Concurrency: 13.98 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 9217.90 Median E2E Latency (ms): 9642.50 P90 E2E Latency (ms): 15147.02 P99 E2E Latency (ms): 18237.06 ---------------Time to First Token---------------- Mean TTFT (ms): 299.02 Median TTFT (ms): 105.98 P99 TTFT (ms): 1109.29 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 18.03 Median TPOT (ms): 18.00 P99 TPOT (ms): 26.51 ---------------Inter-Token Latency---------------- Mean ITL (ms): 17.52 Median ITL (ms): 16.07 P95 ITL (ms): 18.14 P99 ITL (ms): 89.43 Max ITL (ms): 763.13 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 91.48 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 250941 Request throughput (req/s): 5.47 Input token throughput (tok/s): 2730.87 Output token throughput (tok/s): 2761.82 Peak output token throughput (tok/s): 4199.00 Peak concurrent requests: 109 Total token throughput (tok/s): 5492.69 Concurrency: 90.54 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16566.04 Median E2E Latency (ms): 16134.36 P90 E2E Latency (ms): 30167.60 P99 E2E Latency (ms): 34034.04 ---------------Time to First Token---------------- Mean TTFT (ms): 433.94 Median TTFT (ms): 123.26 P99 TTFT (ms): 1760.09 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 32.26 Median TPOT (ms): 33.56 P99 TPOT (ms): 38.78 ---------------Inter-Token Latency---------------- Mean ITL (ms): 31.99 Median ITL (ms): 24.06 P95 ITL (ms): 79.62 P99 ITL (ms): 103.03 Max ITL (ms): 1369.20 ================================================== ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 525.43 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 44462 Total generated tokens (retokenized): 44451 Request throughput (req/s): 0.02 Input token throughput (tok/s): 11.61 Output token throughput (tok/s): 84.62 Peak output token throughput (tok/s): 125.00 Peak concurrent requests: 2 Total token throughput (tok/s): 96.23 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 52540.19 Median E2E Latency (ms): 53694.45 P90 E2E Latency (ms): 94742.08 P99 E2E Latency (ms): 101224.18 ---------------Time to First Token---------------- Mean TTFT (ms): 97.45 Median TTFT (ms): 95.28 P99 TTFT (ms): 105.64 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.94 Median TPOT (ms): 11.25 P99 TPOT (ms): 13.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.80 Median ITL (ms): 11.51 P95 ITL (ms): 15.83 P99 ITL (ms): 16.86 Max ITL (ms): 19.96 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 473.92 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 318306 Total generated tokens (retokenized): 317860 Request throughput (req/s): 0.17 Input token throughput (tok/s): 83.70 Output token throughput (tok/s): 671.65 Peak output token throughput (tok/s): 1040.00 Peak concurrent requests: 19 Total token throughput (tok/s): 755.35 Concurrency: 13.80 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 81746.73 Median E2E Latency (ms): 78508.54 P90 E2E Latency (ms): 155292.49 P99 E2E Latency (ms): 166769.99 ---------------Time to First Token---------------- Mean TTFT (ms): 117.50 Median TTFT (ms): 101.97 P99 TTFT (ms): 182.88 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 20.36 Median TPOT (ms): 20.48 P99 TPOT (ms): 22.63 ---------------Inter-Token Latency---------------- Mean ITL (ms): 20.52 Median ITL (ms): 20.42 P95 ITL (ms): 23.41 P99 ITL (ms): 26.29 Max ITL (ms): 90.48 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 714.72 Total input tokens: 158939 Total input text tokens: 158939 Total generated tokens: 1301025 Total generated tokens (retokenized): 1289431 Request throughput (req/s): 0.45 Input token throughput (tok/s): 222.38 Output token throughput (tok/s): 1820.33 Peak output token throughput (tok/s): 3200.00 Peak concurrent requests: 68 Total token throughput (tok/s): 2042.71 Concurrency: 55.68 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 124364.58 Median E2E Latency (ms): 129250.98 P90 E2E Latency (ms): 219175.80 P99 E2E Latency (ms): 247741.77 ---------------Time to First Token---------------- Mean TTFT (ms): 149.40 Median TTFT (ms): 114.78 P99 TTFT (ms): 288.60 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 30.51 Median TPOT (ms): 31.75 P99 TPOT (ms): 33.32 ---------------Inter-Token Latency---------------- Mean ITL (ms): 30.56 Median ITL (ms): 30.82 P95 ITL (ms): 33.20 P99 ITL (ms): 80.54 Max ITL (ms): 117.72 ================================================== ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 58.27 Total input tokens: 41941 Total input text tokens: 41941 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.17 Input token throughput (tok/s): 719.73 Output token throughput (tok/s): 72.42 Peak output token throughput (tok/s): 112.00 Peak concurrent requests: 2 Total token throughput (tok/s): 792.15 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5825.08 Median E2E Latency (ms): 4624.26 P90 E2E Latency (ms): 12690.22 P99 E2E Latency (ms): 13177.96 ---------------Time to First Token---------------- Mean TTFT (ms): 296.01 Median TTFT (ms): 195.59 P99 TTFT (ms): 717.88 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 12.63 Median TPOT (ms): 13.07 P99 TPOT (ms): 16.68 ---------------Inter-Token Latency---------------- Mean ITL (ms): 13.13 Median ITL (ms): 13.17 P95 ITL (ms): 17.02 P99 ITL (ms): 17.47 Max ITL (ms): 19.84 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 89.59 Total input tokens: 300020 Total input text tokens: 300020 Total generated tokens: 41669 Total generated tokens (retokenized): 41656 Request throughput (req/s): 0.89 Input token throughput (tok/s): 3348.77 Output token throughput (tok/s): 465.10 Peak output token throughput (tok/s): 752.00 Peak concurrent requests: 19 Total token throughput (tok/s): 3813.87 Concurrency: 14.39 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16120.74 Median E2E Latency (ms): 16246.55 P90 E2E Latency (ms): 27279.72 P99 E2E Latency (ms): 34577.93 ---------------Time to First Token---------------- Mean TTFT (ms): 1943.94 Median TTFT (ms): 382.19 P99 TTFT (ms): 8980.41 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 27.87 Median TPOT (ms): 28.26 P99 TPOT (ms): 40.55 ---------------Inter-Token Latency---------------- Mean ITL (ms): 27.27 Median ITL (ms): 21.74 P95 ITL (ms): 23.32 P99 ITL (ms): 232.65 Max ITL (ms): 4282.01 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 167.01 Total input tokens: 1273893 Total input text tokens: 1273893 Total generated tokens: 170000 Total generated tokens (retokenized): 169226 Request throughput (req/s): 1.92 Input token throughput (tok/s): 7627.82 Output token throughput (tok/s): 1017.93 Peak output token throughput (tok/s): 1984.00 Peak concurrent requests: 69 Total token throughput (tok/s): 8645.75 Concurrency: 59.68 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 31147.52 Median E2E Latency (ms): 30603.34 P90 E2E Latency (ms): 54889.44 P99 E2E Latency (ms): 67665.30 ---------------Time to First Token---------------- Mean TTFT (ms): 428.87 Median TTFT (ms): 441.69 P99 TTFT (ms): 1232.68 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 58.06 Median TPOT (ms): 62.79 P99 TPOT (ms): 82.23 ---------------Inter-Token Latency---------------- Mean ITL (ms): 57.93 Median ITL (ms): 33.30 P95 ITL (ms): 247.98 P99 ITL (ms): 409.63 Max ITL (ms): 1421.21 ================================================== ``` #### 5.1.5 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * Result ```text Output theme={null} Accuracy: 0.845 Invalid: 0.000 Latency: 8.431 s Output throughput: 2195.387 token/s ``` # GLM-5 Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5 ## 1. Model Introduction [GLM-5](https://huggingface.co/zai-org/GLM-5) is the most powerful language model in the GLM series developed by Zhipu AI, targeting complex systems engineering and long-horizon agentic tasks. Scaling from GLM-4.5's 355B parameters (32B active) to 744B parameters (40B active), GLM-5 integrates DeepSeek Sparse Attention (DSA) to largely reduce deployment cost while preserving long-context capacity. With advances in both pre-training (28.5T tokens) and post-training via [slime](https://github.com/THUDM/slime) (a novel asynchronous RL infrastructure), GLM-5 delivers significant improvements over GLM-4.7 and achieves best-in-class performance among open-source models on reasoning, coding, and agentic tasks. **Key Features:** * **Systems Engineering & Agentic Tasks**: Purpose-built for complex systems engineering and long-horizon agentic tasks * **State-of-the-Art Performance**: Best-in-class among open-source models on reasoning (HLE, AIME, GPQA), coding (SWE-bench, Terminal-Bench), and agentic tasks (BrowseComp, Vending Bench 2) * **DeepSeek Sparse Attention (DSA)**: Reduces deployment cost while preserving long-context capacity * **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs * **Speculative Decoding**: EAGLE-based speculative decoding support for lower latency **Available Models:** * **BF16 (Full precision)**: [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5) * **FP8 (8-bit quantized)**: [zai-org/GLM-5-FP8](https://huggingface.co/zai-org/GLM-5-FP8) **License:** MIT ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5 on NVIDIA H100, H200, B200, and AMD MI300X/MI325X/MI355X GPUs. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5. ### 3.2 Configuration Tips * Speculative decoding (MTP) can significantly reduce latency for interactive use cases. * **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests. * The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload. * BF16 model always requires **2x GPUs** compared to FP8 on NVIDIA hardware.
Hardware FP8 BF16
H100 tp=16 tp=32
H200 tp=8 tp=16
B200 tp=8 tp=16
MI300X/MI325X — tp=8
MI355X — tp=8
* **B200 (FP8)**: Use `--ep 1 --attention-backend dsa --dsa-decode-backend trtllm --dsa-prefill-backend trtllm --moe-runner-backend flashinfer_trtllm --enable-flashinfer-allreduce-fusion` for optimized DSA and MoE backends on Blackwell. Also add `--quantization fp8` for FP8 weight quantization. * **AMD GPUs**: Use `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang` for the DSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). EAGLE speculative decoding is not currently supported on AMD for GLM-5. * For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common. * **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5. Use `--enable-prefill-cp --cp-strategy interleave` with `--attn-dp-size 1`. * Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature. ## 4. Model Invocation Deploy GLM-5 with the following command (FP8 on H200, all features enabled): ```shell Command theme={null} sglang serve \ --model-path zai-org/GLM-5-FP8 \ --tp 8 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-flashinfer-allreduce-fusion \ --mem-fraction-static 0.85 \ --host 0.0.0.0 \ --port 30000 ``` ### 4.1 MI300X/MI325X/MI355X (ROCm) Server Command The following ROCm command is an additional option for AMD GPUs and does not replace the NVIDIA instructions above. ```shell Command theme={null} sglang serve \ --model-path zai-org/GLM-5 \ --tp 8 \ --trust-remote-code \ --dsa-prefill-backend tilelang \ --dsa-decode-backend tilelang \ --chunked-prefill-size 131072 \ --mem-fraction-static 0.80 \ --watchdog-timeout 1200 \ --host 0.0.0.0 \ --port 30000 ``` ### 4.2 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.3 Advanced Usage #### 4.3.1 Reasoning Parser GLM-5 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response. To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time: * **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed. * **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process. **Example 1: Thinking Mode (Default)** Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Thinking mode is enabled by default, no extra parameters needed response = client.chat.completions.create( model="zai-org/GLM-5-FP8", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user wants me to solve a math problem: "What is 15% of 240?". Step 1: Understand the problem. I need to calculate a percentage of a number. Formula: Percentage × Number = Result. Step 2: Convert the percentage to a decimal or fraction. 15% = 15/100 or 0.15. Step 3: Perform the multiplication. Method A: Decimal multiplication. 0.15 × 240. Break it down: 10% of 240 = 24. 5% is half of 10%, so 12. 15% = 10% + 5% = 24 + 12 = 36. Method B: Fraction multiplication. 15/100 × 240. Simplify 240/100 = 2.4. 15 × 2.4. 10 × 2.4 = 24. 5 × 2.4 = 12. 24 + 12 = 36. Method C: Direct multiplication. 240 × 0.15. 240 × 0.10 = 24. 240 × 0.05 = 12. 24 + 12 = 36. Step 4: Final Verification. Is 36 reasonable? 10% is 24. 20% is 48. 15% is halfway between 10% and 20%. Halfway between 24 and 48 is 36. The result is correct. Step 5: Structure the final response. I will present the calculation clearly, perhaps showing the fractional or decimal method, or the mental math shortcut (10% + 5%). =============== Content ================= Here is the step-by-step solution: **Step 1: Convert the percentage to a decimal.** To convert 15% to a decimal, divide by 100. $$15\% = \frac{15}{100} = 0.15$$ **Step 2: Multiply the decimal by the number.** Now, multiply 0.15 by 240. $$0.15 \times 240$$ **Step 3: Perform the calculation.** You can break this down to make it easier: $$0.15 = 0.10 + 0.05$$ * First, find 10% of 240: $$0.10 \times 240 = 24$$ * Next, find 5% (which is half of 10%): $$\frac{24}{2} = 12$$ * Add the two results together: $$24 + 12 = 36$$ **Answer:** 15% of 240 is **36**. ``` **Example 2: Instruct Mode (Thinking Off)** To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Disable thinking mode via chat_template_kwargs response = client.chat.completions.create( model="zai-org/GLM-5-FP8", messages=[ {"role": "user", "content": "What is 15% of 240?"} ], extra_body={"chat_template_kwargs": {"enable_thinking": False}}, max_tokens=2048, stream=True ) # In Instruct mode, the model responds directly without reasoning_content for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} To find **15% of 240**, follow these steps: ### Step 1: Convert the Percentage to a Decimal First, convert the percentage to a decimal by dividing by 100. \[ 15\% = \frac{15}{100} = 0.15 \] ### Step 2: Multiply by the Number Next, multiply the decimal by the number you want to find the percentage of. \[ 0.15 \times 240 \] ### Step 3: Perform the Multiplication Calculate the multiplication: \[ 0.15 \times 240 = 36 \] ### Final Answer \[ \boxed{36} \] ``` #### 4.3.2 Tool Calling GLM-5 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`. **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-5-FP8", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking for the weather in Beijing. I have access to a get_weather function that can provide current weather information. Let me check what parameters are required: - location: required, should be "Beijing" - unit: optional (not in required array), can be "celsius" or "fahrenheit" Since the user didn't specify a unit preference and it's optional, I should not ask about it or make up a value. I'll just call the function with the required location parameter.I'll get the current weather in Beijing for you. =============== Content ================= Tool Call: get_weather Arguments: Tool Call: None Arguments: { Tool Call: None Arguments: "location": "Be Tool Call: None Arguments: ijing" Tool Call: None Arguments: } ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: H200 (8x) * Model: GLM-5-FP8 * Tensor Parallelism: 8 * SGLang Version: commit 947927bdb #### 5.1.1 Latency Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-5-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 35.78 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4213 Request throughput (req/s): 0.28 Input token throughput (tok/s): 170.54 Output token throughput (tok/s): 117.96 Peak output token throughput (tok/s): 148.00 Peak concurrent requests: 2 Total token throughput (tok/s): 288.50 Concurrency: 1.00 Accept length: 3.48 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3576.31 Median E2E Latency (ms): 2935.97 P90 E2E Latency (ms): 5908.97 P99 E2E Latency (ms): 8588.08 ---------------Time to First Token---------------- Mean TTFT (ms): 290.88 Median TTFT (ms): 282.34 P99 TTFT (ms): 332.27 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.54 Median TPOT (ms): 6.97 P99 TPOT (ms): 9.04 ---------------Inter-Token Latency---------------- Mean ITL (ms): 7.80 Median ITL (ms): 6.81 P95 ITL (ms): 13.51 P99 ITL (ms): 26.99 Max ITL (ms): 29.50 ================================================== ``` #### 5.1.2 Throughput Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-5-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 1000 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 411.74 Total input tokens: 502493 Total input text tokens: 502493 Total generated tokens: 500251 Total generated tokens (retokenized): 499614 Request throughput (req/s): 2.43 Input token throughput (tok/s): 1220.41 Output token throughput (tok/s): 1214.97 Peak output token throughput (tok/s): 2648.00 Peak concurrent requests: 105 Total token throughput (tok/s): 2435.38 Concurrency: 96.30 Accept length: 3.50 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 39648.76 Median E2E Latency (ms): 39058.12 P90 E2E Latency (ms): 57009.82 P99 E2E Latency (ms): 68880.33 ---------------Time to First Token---------------- Mean TTFT (ms): 20613.80 Median TTFT (ms): 21429.21 P99 TTFT (ms): 29543.17 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 38.73 Median TPOT (ms): 36.52 P99 TPOT (ms): 67.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 38.13 Median ITL (ms): 16.57 P95 ITL (ms): 86.01 P99 ITL (ms): 164.88 Max ITL (ms): 1307.02 ================================================== ``` ### 5.2 Accuracy Benchmark The accuracy benchmark results below are shared with GLM-5.1, as GLM-5.1 was not independently benchmarked at the time of this writing. A separate GLM-5.1 benchmark run is planned. #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. * Test Result ```text Output theme={null} Accuracy: 0.955 Invalid: 0.000 Latency: 32.470 s Output throughput: 642.044 token/s ``` #### 5.2.2 MMLU Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run mmlu \ --base-url http://127.0.0.1:30000/v1 \ --num-threads 64 --max-tokens 2048 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. * Test Result ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.860 subject: anatomy, #q:135, acc: 0.874 subject: astronomy, #q:152, acc: 0.941 subject: business_ethics, #q:100, acc: 0.880 subject: clinical_knowledge, #q:265, acc: 0.932 subject: college_biology, #q:144, acc: 0.972 subject: college_chemistry, #q:100, acc: 0.640 subject: college_computer_science, #q:100, acc: 0.900 subject: college_mathematics, #q:100, acc: 0.810 subject: college_medicine, #q:173, acc: 0.873 subject: college_physics, #q:102, acc: 0.912 subject: computer_security, #q:100, acc: 0.880 subject: conceptual_physics, #q:235, acc: 0.928 subject: econometrics, #q:114, acc: 0.807 subject: electrical_engineering, #q:145, acc: 0.897 subject: elementary_mathematics, #q:378, acc: 0.937 subject: formal_logic, #q:126, acc: 0.778 subject: global_facts, #q:100, acc: 0.710 subject: high_school_biology, #q:310, acc: 0.961 subject: high_school_chemistry, #q:203, acc: 0.847 subject: high_school_computer_science, #q:100, acc: 0.960 subject: high_school_european_history, #q:165, acc: 0.891 subject: high_school_geography, #q:198, acc: 0.960 subject: high_school_government_and_politics, #q:193, acc: 0.984 subject: high_school_macroeconomics, #q:390, acc: 0.923 subject: high_school_mathematics, #q:270, acc: 0.696 subject: high_school_microeconomics, #q:238, acc: 0.962 subject: high_school_physics, #q:151, acc: 0.821 subject: high_school_psychology, #q:545, acc: 0.956 subject: high_school_statistics, #q:216, acc: 0.889 subject: high_school_us_history, #q:204, acc: 0.941 subject: high_school_world_history, #q:237, acc: 0.945 subject: human_aging, #q:223, acc: 0.857 subject: human_sexuality, #q:131, acc: 0.908 subject: international_law, #q:121, acc: 0.934 subject: jurisprudence, #q:108, acc: 0.907 subject: logical_fallacies, #q:163, acc: 0.933 subject: machine_learning, #q:112, acc: 0.830 subject: management, #q:103, acc: 0.942 subject: marketing, #q:234, acc: 0.940 subject: medical_genetics, #q:100, acc: 0.990 subject: miscellaneous, #q:783, acc: 0.959 subject: moral_disputes, #q:346, acc: 0.873 subject: moral_scenarios, #q:895, acc: 0.837 subject: nutrition, #q:306, acc: 0.922 subject: philosophy, #q:311, acc: 0.897 subject: prehistory, #q:324, acc: 0.929 subject: professional_accounting, #q:282, acc: 0.844 subject: professional_law, #q:1534, acc: 0.714 subject: professional_medicine, #q:272, acc: 0.941 subject: professional_psychology, #q:612, acc: 0.913 subject: public_relations, #q:110, acc: 0.791 subject: security_studies, #q:245, acc: 0.878 subject: sociology, #q:201, acc: 0.940 subject: us_foreign_policy, #q:100, acc: 0.920 subject: virology, #q:166, acc: 0.596 subject: world_religions, #q:171, acc: 0.936 Total latency: 165.275 Average accuracy: 0.877 ``` ### 5.3 AMD GPU Benchmarks #### 5.3.1 GSM8K Benchmark (MI325/MI35x) * MI325/MI35x Test (GLM-5 BF16, `tp=8`, TileLang DSA backends) ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. ```text Output theme={null} Accuracy: 0.970 Invalid: 0.000 ``` Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911). # GLM-5.1 Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.1 ## 1. Model Introduction **Available Models:** * **BF16 (Full precision)**: [zai-org/GLM-5.1](https://huggingface.co/zai-org/GLM-5.1) * **FP8 (8-bit quantized)**: [zai-org/GLM-5.1-FP8](https://huggingface.co/zai-org/GLM-5.1-FP8) * **NVFP4 (4-bit quantized)**: [nvidia/GLM-5.1-NVFP4](https://huggingface.co/nvidia/GLM-5.1-NVFP4) **License:** MIT ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5.1 on NVIDIA H100, H200, B300, GB300, and AMD MI300X/MI325X/MI355X GPUs. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.1. ### 3.2 Configuration Tips * Speculative decoding (MTP) can significantly reduce latency for interactive use cases. * **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests. * The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
Hardware NVFP4 FP8 BF16 MXFP4
H100 — tp=16 — —
H200 — tp=8 — —
B300 tp=8 — — —
GB300 tp=4 — — —
MI300X/MI325X — tp=8 tp=8 —
MI355X — tp=8 tp=8 tp=4
* **H100 and H200**: FP8 is the recommended deployment path. * **B300 and GB300**: NVFP4 is the recommended deployment path. Use `nvidia/GLM-5.1-NVFP4` with `--quantization modelopt_fp4`. Use `tp=8` on B300 and `tp=4` on GB300. The CUDA 13 image variant is required for B300 and GB300. * **AMD GPUs**: BF16 and FP8 checkpoints run on MI300X/MI325X/MI355X at tp=8. On MI355X (gfx950), the MXFP4 checkpoint `amd/GLM-5.1-MXFP4` is also supported at tp=4 with `--kv-cache-dtype fp8_e4m3`. All AMD paths pass `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`, `--chunked-prefill-size 131072`, and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (\~89 GB/GPU vs \~175 GB/GPU). EAGLE speculative decoding is supported on AMD GPUs: MI300X/MI325X (gfx942) and MI355X (gfx950), but it **requires `--disable-custom-all-reduce`** — the aiter custom all-reduce kernel deadlocks during EAGLE verify at high concurrency, so without this flag the server will hang. * For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5.1 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common. * **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.1. Use `--enable-prefill-cp --cp-strategy interleave`. Attention DP is supported, with `tp_size / attn_dp_size` CP ranks in each DP group. Dense FFNs retain their configured TP width. * Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` to enable the [IndexCache](https://github.com/THUDM/IndexCache) method for GLM-5.1. This can improve serving efficiency with only a small accuracy loss. If you are running rigorous accuracy evaluations, do not enable this feature. ## 4. Model Invocation Deploy GLM-5.1 with the following command (FP8 on H200, all features enabled): ```shell Command theme={null} sglang serve \ --model-path zai-org/GLM-5.1-FP8 \ --tp 8 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --mem-fraction-static 0.85 \ --host 0.0.0.0 \ --port 30000 ``` ### 4.1 B300/GB300 (NVFP4) Server Command #### B300 ```shell Command theme={null} sglang serve \ --model-path nvidia/GLM-5.1-NVFP4 \ --tp 8 \ --quantization modelopt_fp4 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --trust-remote-code \ --mem-fraction-static 0.80 \ --host 0.0.0.0 \ --port 30000 ``` #### GB300 ```shell Command theme={null} sglang serve \ --model-path nvidia/GLM-5.1-NVFP4 \ --tp 4 \ --quantization modelopt_fp4 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --trust-remote-code \ --mem-fraction-static 0.80 \ --host 0.0.0.0 \ --port 30000 ``` ### 4.2 MI300X/MI325X/MI355X (ROCm) Server Command The following ROCm commands are additional options for AMD GPUs and do not replace the NVIDIA instructions above. #### MXFP4 (MI355X / gfx950) On MI355X (gfx950), set `SGLANG_DSA_TRITON_PREFILL=1` to enable a faster Triton attention kernel for the prefill phase (opt-in, off by default). Keep `--dsa-prefill-backend tilelang` as shown. The EAGLE speculative-decoding flags below are optional but recommended on gfx950. ```shell Command theme={null} # SGLANG_DSA_TRITON_PREFILL=1 is optional; it enables a faster Triton prefill kernel on gfx950 SGLANG_DSA_TRITON_PREFILL=1 sglang serve \ --model-path amd/GLM-5.1-MXFP4 \ --tp 4 \ --trust-remote-code \ --kv-cache-dtype fp8_e4m3 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --dsa-prefill-backend tilelang \ --dsa-decode-backend tilelang \ --chunked-prefill-size 131072 \ --mem-fraction-static 0.85 \ --watchdog-timeout 1200 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --disable-custom-all-reduce \ --host 0.0.0.0 \ --port 30000 ``` #### FP8 (Recommended) ```shell Command theme={null} sglang serve \ --model-path zai-org/GLM-5.1-FP8 \ --tp 8 \ --trust-remote-code \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --dsa-prefill-backend tilelang \ --dsa-decode-backend tilelang \ --chunked-prefill-size 131072 \ --mem-fraction-static 0.80 \ --watchdog-timeout 1200 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --disable-custom-all-reduce \ --host 0.0.0.0 \ --port 30000 ``` #### BF16 ```shell Command theme={null} sglang serve \ --model-path zai-org/GLM-5.1 \ --tp 8 \ --trust-remote-code \ --dsa-prefill-backend tilelang \ --dsa-decode-backend tilelang \ --chunked-prefill-size 131072 \ --mem-fraction-static 0.80 \ --watchdog-timeout 1200 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --disable-custom-all-reduce \ --host 0.0.0.0 \ --port 30000 ``` ### 4.3 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.4 Advanced Usage #### 4.4.1 Reasoning Parser GLM-5.1 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response. To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time: * **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed. * **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process. **Example 1: Thinking Mode (Default)** Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Thinking mode is enabled by default, no extra parameters needed response = client.chat.completions.create( model="zai-org/GLM-5.1-FP8", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= 1. **Understand the Goal:** The user wants to find 15% of 240, and they want the solution explained step-by-step. 2. **Identify the Core Mathematical Concept:** "Percent" means "per hundred" or "out of 100". Finding "X% of Y" translates to the mathematical operation: $(X / 100) \times Y$. 3. **Step-by-Step Breakdown:** * *Step 1: Convert the percentage to a decimal (or fraction).* 15% means 15 out of 100, which is $15/100$ or $0.15$. * *Step 2: Multiply the decimal by the given number.* Multiply $0.15$ by $240$. * *Step 3: Perform the calculation.* * $0.15 \times 240$ * I can break this down further to make it easy to follow: * $0.10 \times 240 = 24$ (which is 10%) * $0.05 \times 240 = 12$ (which is 5%, half of 10%) * $24 + 12 = 36$ * Alternatively, standard multiplication: * $240 \times 15 = 3600$ * Move decimal two places left -> $36$ * *Step 4: State the final answer clearly.* 4. **Draft the Response (incorporating the steps clearly):** * *Introduction:* State the problem clearly. * *Step 1:* Explain how to convert 15% to a decimal. * *Step 2:* Explain the multiplication step. * *Step 3:* Show the actual math (I'll provide the standard multiplication and the "mental math" trick as it adds value). * *Conclusion:* Give the final answer. 5. **Refine the Output (Self-Correction/Polishing during drafting):** * *Drafting Step 1:* To find 15% of 240, first convert 15% into a decimal. Since percent means "per hundred," you divide 15 by 100. 15 ÷ 100 = 0.15. * *Drafting Step 2:* Next, multiply this decimal by the number you are finding the percentage of (which is 240). So, calculate 0.15 × 240. * *Drafting Step 3 (Standard way):* 0.15 × 240 = 36. * *Adding the alternative mental math way:* It's often helpful to break it down into 10% and 5%. * 10% of 240 = 24 (move the decimal point one place to the left) * 5% is half of 10%, so half of 24 = 12 * Add them together: 24 + 12 = 36. * *Final Answer:* 15% of 240 is 36. 6. **Final Review against User Prompt:** Does it solve the problem? Yes. Is it step-by-step? Yes. Is it clear? Yes. (Proceed to generate output). =============== Content ================= Here is the step-by-step solution to find 15% of 240: **Step 1: Convert the percentage to a decimal.** To convert a percentage to a decimal, divide it by 100 (or simply move the decimal point two places to the left). * 15% = 15 ÷ 100 = **0.15** **Step 2: Multiply the decimal by the number.** Now, multiply the decimal (0.15) by the number you are finding the percentage of (240). * 0.15 × 240 = **36** *(Alternative mental math method for Step 2)*: If you don't want to multiply by 0.15 directly, you can break 15% down into 10% and 5%: * **10% of 240** = 24 (just move the decimal point one place to the left) * **5% of 240** = 12 (5% is half of 10%, so just divide 24 by 2) * **Add them together**: 24 + 12 = **36** **Answer:** 15% of 240 is **36**. ``` **Example 2: Instruct Mode (Thinking Off)** To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Disable thinking mode via chat_template_kwargs response = client.chat.completions.create( model="zai-org/GLM-5.1-FP8", messages=[ {"role": "user", "content": "What is 15% of 240?"} ], extra_body={"chat_template_kwargs": {"enable_thinking": False}}, max_tokens=2048, stream=True ) # In Instruct mode, the model responds directly without reasoning_content for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} 15% of 240 is 36. Here is how to calculate it: 1. Convert the percentage to a decimal: 15% = 0.15 2. Multiply the decimal by the number: 0.15 × 240 = 36 ``` #### 4.4.2 Tool Calling GLM-5.1 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`. **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-5.1-FP8", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user wants to know the weather in Beijing. I'll call the get_weather function with "Beijing" as the location. =============== Content ================= Tool Call: get_weather Arguments: Tool Call: None Arguments: { Tool Call: None Arguments: "location": "Be Tool Call: None Arguments: ijing" Tool Call: None Arguments: } ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: H200 (8x) * Model: GLM-5.1-FP8 * Tensor Parallelism: 8 * SGLang Version: commit 947927bdb #### 5.1.1 Latency Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-5.1-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 35.78 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4213 Request throughput (req/s): 0.28 Input token throughput (tok/s): 170.54 Output token throughput (tok/s): 117.96 Peak output token throughput (tok/s): 148.00 Peak concurrent requests: 2 Total token throughput (tok/s): 288.50 Concurrency: 1.00 Accept length: 3.48 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3576.31 Median E2E Latency (ms): 2935.97 P90 E2E Latency (ms): 5908.97 P99 E2E Latency (ms): 8588.08 ---------------Time to First Token---------------- Mean TTFT (ms): 290.88 Median TTFT (ms): 282.34 P99 TTFT (ms): 332.27 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.54 Median TPOT (ms): 6.97 P99 TPOT (ms): 9.04 ---------------Inter-Token Latency---------------- Mean ITL (ms): 7.80 Median ITL (ms): 6.81 P95 ITL (ms): 13.51 P99 ITL (ms): 26.99 Max ITL (ms): 29.50 ================================================== ``` #### 5.1.2 Throughput Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-5.1-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 1000 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 411.74 Total input tokens: 502493 Total input text tokens: 502493 Total generated tokens: 500251 Total generated tokens (retokenized): 499614 Request throughput (req/s): 2.43 Input token throughput (tok/s): 1220.41 Output token throughput (tok/s): 1214.97 Peak output token throughput (tok/s): 2648.00 Peak concurrent requests: 105 Total token throughput (tok/s): 2435.38 Concurrency: 96.30 Accept length: 3.50 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 39648.76 Median E2E Latency (ms): 39058.12 P90 E2E Latency (ms): 57009.82 P99 E2E Latency (ms): 68880.33 ---------------Time to First Token---------------- Mean TTFT (ms): 20613.80 Median TTFT (ms): 21429.21 P99 TTFT (ms): 29543.17 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 38.73 Median TPOT (ms): 36.52 P99 TPOT (ms): 67.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 38.13 Median ITL (ms): 16.57 P95 ITL (ms): 86.01 P99 ITL (ms): 164.88 Max ITL (ms): 1307.02 ================================================== ``` ### 5.2 Accuracy Benchmark The accuracy benchmark results below are shared with GLM-5, as GLM-5.1 was not independently benchmarked at the time of this writing. A separate benchmark run is planned. #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. * Test Result ```text Output theme={null} Accuracy: 0.955 Invalid: 0.000 Latency: 32.470 s Output throughput: 642.044 token/s ``` #### 5.2.2 MMLU Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run mmlu \ --base-url http://127.0.0.1:30000/v1 \ --num-threads 64 --max-tokens 2048 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. * Test Result ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.860 subject: anatomy, #q:135, acc: 0.874 subject: astronomy, #q:152, acc: 0.941 subject: business_ethics, #q:100, acc: 0.880 subject: clinical_knowledge, #q:265, acc: 0.932 subject: college_biology, #q:144, acc: 0.972 subject: college_chemistry, #q:100, acc: 0.640 subject: college_computer_science, #q:100, acc: 0.900 subject: college_mathematics, #q:100, acc: 0.810 subject: college_medicine, #q:173, acc: 0.873 subject: college_physics, #q:102, acc: 0.912 subject: computer_security, #q:100, acc: 0.880 subject: conceptual_physics, #q:235, acc: 0.928 subject: econometrics, #q:114, acc: 0.807 subject: electrical_engineering, #q:145, acc: 0.897 subject: elementary_mathematics, #q:378, acc: 0.937 subject: formal_logic, #q:126, acc: 0.778 subject: global_facts, #q:100, acc: 0.710 subject: high_school_biology, #q:310, acc: 0.961 subject: high_school_chemistry, #q:203, acc: 0.847 subject: high_school_computer_science, #q:100, acc: 0.960 subject: high_school_european_history, #q:165, acc: 0.891 subject: high_school_geography, #q:198, acc: 0.960 subject: high_school_government_and_politics, #q:193, acc: 0.984 subject: high_school_macroeconomics, #q:390, acc: 0.923 subject: high_school_mathematics, #q:270, acc: 0.696 subject: high_school_microeconomics, #q:238, acc: 0.962 subject: high_school_physics, #q:151, acc: 0.821 subject: high_school_psychology, #q:545, acc: 0.956 subject: high_school_statistics, #q:216, acc: 0.889 subject: high_school_us_history, #q:204, acc: 0.941 subject: high_school_world_history, #q:237, acc: 0.945 subject: human_aging, #q:223, acc: 0.857 subject: human_sexuality, #q:131, acc: 0.908 subject: international_law, #q:121, acc: 0.934 subject: jurisprudence, #q:108, acc: 0.907 subject: logical_fallacies, #q:163, acc: 0.933 subject: machine_learning, #q:112, acc: 0.830 subject: management, #q:103, acc: 0.942 subject: marketing, #q:234, acc: 0.940 subject: medical_genetics, #q:100, acc: 0.990 subject: miscellaneous, #q:783, acc: 0.959 subject: moral_disputes, #q:346, acc: 0.873 subject: moral_scenarios, #q:895, acc: 0.837 subject: nutrition, #q:306, acc: 0.922 subject: philosophy, #q:311, acc: 0.897 subject: prehistory, #q:324, acc: 0.929 subject: professional_accounting, #q:282, acc: 0.844 subject: professional_law, #q:1534, acc: 0.714 subject: professional_medicine, #q:272, acc: 0.941 subject: professional_psychology, #q:612, acc: 0.913 subject: public_relations, #q:110, acc: 0.791 subject: security_studies, #q:245, acc: 0.878 subject: sociology, #q:201, acc: 0.940 subject: us_foreign_policy, #q:100, acc: 0.920 subject: virology, #q:166, acc: 0.596 subject: world_religions, #q:171, acc: 0.936 Total latency: 165.275 Average accuracy: 0.877 ``` ### 5.3 AMD GPU Benchmarks #### 5.3.1 GSM8K Benchmark (MI325/MI35x) * MI325/MI35x Test (GLM-5.1 BF16, `tp=8`, TileLang DSA backends) ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. ```text Output theme={null} Accuracy: 0.970 Invalid: 0.000 ``` Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911). # GLM-5.2 Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.2 Deploy GLM-5.2 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points: * **Low-Latency** — fastest reply for a single user. Pick for chat. * **Balanced** — good speed with several users at once. Use for typical multi-user serving. * **High-Throughput** — most tokens per second across many users. Best for batch jobs. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.2. Speed numbers are measured with `--random-range-ratio 1.0`, `--flush-cache`, on `main @ 09ca4fc` (H200 FP8 cells: `v0.5.14 @ 49e384ce`). Spec cells pin the EAGLE acceptance length via the serve env `SGLANG_SIMULATE_ACC_LEN` (low-latency 5-1-6 = 3.5; FP8 balanced 1-1-2 = 2; NVFP4 balanced 2-1-3 = 2); high-throughput has no spec. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## Agentic Long-Context Deployment Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order. * **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks). * **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`. * **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image. ## 1. Model Introduction **GLM-5.2** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It ships in two precisions — **FP8** (`zai-org/GLM-5.2-FP8`) and full **BF16** (`zai-org/GLM-5.2`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (\~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, NVIDIA also publishes an **NVFP4** build (`nvidia/GLM-5.2-NVFP4`) that quantizes only the MoE experts' linear weights and activations to 4-bit (the shared expert stays unquantized), holding accuracy within \~1 point of the FP8 baseline on GPQA Diamond, SciCode, and IFBench. For AMD MI355X (gfx950), AMD publishes an **MXFP4** build (`amd/GLM-5.2-MXFP4`, Quark-quantized) — see the AMD GPUs configuration tip below; this recipe is inferred from the validated `amd/GLM-5.1-MXFP4` MI355X recipe and not yet benchmarked on GLM-5.2 (`verified: false`).
Model Architecture Context
GLM-5.2-FP8 MoE · DSA · 256 experts (top-8) · MTP · FP8 1,048,576
GLM-5.2 MoE · DSA · 256 experts (top-8) · MTP · BF16 1,048,576
GLM-5.2-NVFP4 MoE · DSA · 256 experts (top-8) · MTP · NVFP4 1,048,576
GLM-5.2-MXFP4 MoE · DSA · 256 experts (top-8) · MTP · MXFP4 1,048,576
**Recommended generation:** `temperature=1.0`, `top_p=0.95` (the checkpoint's `generation_config.json` defaults; informational — do not hardcode in client code). **Resources:** [GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) · [GLM-5.2 (BF16)](https://huggingface.co/zai-org/GLM-5.2) · [GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4) · [GLM-5.2-MXFP4](https://huggingface.co/amd/GLM-5.2-MXFP4). ## 2. Configuration Tips * **DeepSeek Sparse Attention (DSA).** GLM-5.2 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required. On Hopper, pairing `--kv-cache-dtype fp8_e4m3` with `--dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv` selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.2's 64 query heads match the kernel's native tile) — see the [DeepSeek-V3.2 page](../DeepSeek/DeepSeek-V3_2) for kernel details; the optional `SGLANG_ENABLE_DSA_Q8KV8_*` performance env vars are documented in `python/sglang/srt/environ.py`. * **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). **Tune the draft length to the accept length.** GLM-5.2's MTP head is strong — accept length runs high (4+ in many workloads, near-saturating at 5–6 in low-latency runs). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens` accordingly: while accept length stays close to the draft-token count there is headroom to push them higher (more accepted tokens per step); if it falls well below, lower them — every rejected draft token is wasted verification compute. * **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4). * **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP. * **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.2`, \~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, \~2.1 TB HBM) — **verified**; on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8), and those **multi-node BF16 recipes are still proposed/inferred** (`verified: false`). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. On B300, BF16 low-latency matches FP8 (the sm103 FP8 path is not yet optimized), but FP8 wins at the balanced/high-throughput points. * **PD Disaggregation (prefill/decode).** GLM-5.2 is a DSA model and runs under prefill/decode disaggregation — toggle the **PD Disagg** card in the [Playground above](#playground) (pick a Prefill/Decode role + transfer backend, then front the roles with `sglang_router.launch_router --pd-disaggregation`). The Mooncake backend **auto-detects the InfiniBand HCA**, so no device flag is needed by default; only add `--disaggregation-ib-device mlx5_0` (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (`--privileged --ulimit memlock=-1`, or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) — without IB exposure Mooncake silently falls back to TCP. * **Chunked-prefill size is regime-dependent.** At long input (8K+) the default `--chunked-prefill-size 2048` is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to `--chunked-prefill-size 32768` on the balanced recipe gave roughly **+34–78% output throughput and −39–59% TTFT** on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is **neutral for high-throughput** (decode-bound there) — keep the default. `--max-running-requests` tracks KV capacity, not a tuning free-for-all: \~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near `--max-running-requests 80` and let high-throughput run wider. * **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.2-FP8`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.2`, \~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there (or a multi-node BF16 layout once validated). Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). FP8 uses about half the memory of BF16 (\~89 GB/GPU vs \~175 GB/GPU). GLM-5.2 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2). * **MI355X MXFP4 (gfx950-only).** AMD publishes a Quark-quantized **`amd/GLM-5.2-MXFP4`** build for MI355X. It needs `--trust-remote-code` (Quark's custom quant config) and runs at `tp=4` (the 4-bit MoE weights fit a 4-GPU slice) with `--kv-cache-dtype fp8_e4m3`, the DSA triton backends (`--dsa-prefill-backend triton --dsa-decode-backend triton`, also SGLang's ROCm default), and the same `--chunked-prefill-size` and `--watchdog-timeout` as the FP8/BF16 recipes above. At `tp=4` this shape — 16 query heads per rank on an FP8 KV cache — is the one the gfx950 sparse-MLA kernels are tuned for; the `tp=8` recipes above stay on tilelang, which is what their published numbers were measured with. This recipe is carried over from the validated `amd/GLM-5.1-MXFP4` MI355X deployment (same DSA architecture family) and has not yet been benchmarked on GLM-5.2, so the Deploy panel marks it unverified. **gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to \~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3). * **MTP / EAGLE speculative decoding on AMD.** Five-step MTP is validated for `amd/GLM-5.2-MXFP4` on MI355X with `lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260930`. Choose **Low-Latency** for TP8/EP1, or **High-Throughput** for TP4/EP4 with HiCache (`--enable-hierarchical-cache --hicache-ratio 1.0`). HiCache uses the default `kernel`, `page_first`, and `write_through` settings. MTP remains unvalidated for GLM-5.2 on MI300X/MI325X and for the other MI355X checkpoint precisions. ## 3. Advanced Usage ### 3.1 Reasoning GLM-5.2 is a hybrid-reasoning model. Enable the `glm45` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Thinking is on by default; turn it off with `chat_template_kwargs: {"enable_thinking": False}` (the template variable is `enable_thinking`, not `thinking`). **Reasoning effort.** Pass `chat_template_kwargs: {"reasoning_effort": ...}` to inject a `Reasoning Effort: ` system line (only while thinking is on). **The template wires only two effective levels — `Max` and `High` — and if you don't pass `reasoning_effort` at all you get `Max`, the highest.** `"high"` is the *only* value that lowers effort; every other value (including `"low"` and `"medium"`) falls through to `Max`: | `reasoning_effort` | Injected system line | Effect | | - | - | - | | *(not passed / unset)* | `Reasoning Effort: Max` | **default — highest reasoning** | | `"high"` | `Reasoning Effort: High` | dials reasoning **down** | | `"low"`, `"medium"`, any other value | `Reasoning Effort: Max` | falls through to `Max` (not a distinct level) | ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="zai-org/GLM-5.2-FP8", messages=[{"role": "user", "content": "What is 15% of 240?"}], extra_body={"chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "high"}}, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Reasoning: 1. **Identify the core question:** The user wants to find 15% of 240. 2. **Convert the percentage to a decimal:** 15% = 0.15 3. **Multiply by the total:** 0.15 * 240 = 36 (Quick mental math: 10% of 240 = 24; 5% = 12; 24 + 12 = 36.) Answer: 15% of 240 is **36**. Here is how you can calculate it: 0.15 × 240 = 36 ``` ### 3.2 Tool Calling Enable the `glm47` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. GLM-5.2 emits the newer `………` format, so it needs the **`glm47`** parser — the older `glm45` parser does not parse it (the call would be left as raw text in `content`). On thinking mode the turn also fills `reasoning_content`, so print both fields. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }] resp = client.chat.completions.create( model="zai-org/GLM-5.2-FP8", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Reasoning: The user wants to know the weather in Paris. I'll call the get_weather function with "Paris" as the city. Tool calls: [ { "id": "call_13fcd52146934b7781d06d4a", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"} } ] ``` ### 3.3 HiCache (Hierarchical KV Caching) For long-context, prefix-heavy workloads, enable hierarchical KV caching to spill cold KV blocks to host memory (toggle the **Hierarchical KV Cache** card in the [Playground above](#playground)). Useful given GLM-5.2's 1M-token window; pair `--hicache-ratio` with a write policy that matches your reuse pattern. ### 3.4 Claude Code Integration GLM-5.2's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.2 server with only environment variables — no code change. Launch the server with `--reasoning-parser glm45 --tool-call-parser glm47` (any recipe from the Deployment panel above works), then: ```bash Command theme={null} export ANTHROPIC_BASE_URL="http://127.0.0.1:30000" export ANTHROPIC_AUTH_TOKEN="dummy" export API_TIMEOUT_MS="3000000" export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000" export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 export CLAUDE_CODE_ATTRIBUTION_HEADER=0 export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]" export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]" export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]" claude ``` Two of these matter specifically for GLM-5.2: * **`CLAUDE_CODE_ATTRIBUTION_HEADER=0`** — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.2's chat template renders `tools` **before** `system`, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse. * **`glm-5.2[1m]`** as the model name — the `[1m]` suffix is the client-side hint that enables Claude Code's 1M-context beta, matching GLM-5.2's 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the `model` field, so any name is accepted server-side. For the full setup (streaming, tool-use, count\_tokens, persisting env in `~/.claude/settings.json`, troubleshooting), see [Anthropic-Compatible API](../../../docs/basic_usage/anthropic_api). ### 3.5 Context Parallelism Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.2. For prefill CP on CUDA, use `interleave` as shown below. Attention DP is supported: each DP group uses `tp_size / attn_dp_size` CP ranks. Dense FFNs retain the configured TP width; use `--moe-dense-tp-size 1` for local dense computation. Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.2, please append the following arguments: ```bash theme={null} --attn-cp-size 8 \ --enable-prefill-cp \ --cp-strategy interleave \ ``` which splits the sequence equally across `--attn-cp-size` ranks during attention forward. The trade off for prefill CP is that it will introduce extra all-gather operation before indexer-topk and attention kernels, so it will increase latency for decode (in unified deployment) or short prefill. When deploying with PD Disaggregation, the prefill node can choose to enable [LayerSplit](https://z.ai/blog/scaling-pain) technique with ```bash theme={null} --enable-dsa-cache-layer-split \ --attn-cp-size 8 \ --cp-strategy interleave \ ``` With LayerSplit, the kv cache on each rank can be sharded over the CP attention group, and prefetched when necessary. This can reduce kv cache memory by up to 75%, thus increasing the throughput on prefill side. # GLM-5.3 Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3 Deploy GLM-5.3 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points: * **Low-Latency** — fastest reply for a single user. Pick for chat. * **Balanced** — good speed with several users at once. Use for typical multi-user serving. * **High-Throughput** — most tokens per second across many users. Best for batch jobs. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.3. ## Playground The Playground is where you experiment with **SGLang features beyond the deployment matrix**. It lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction **GLM-5.3** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It keeps the same base architecture while updating post-training for complex coding and long-horizon tasks. It ships in two precisions — **FP8** (`zai-org/GLM-5.3`) and full **BF16** (`zai-org/GLM-5.3-BF16`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (\~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, RadixArk publishes an **experimental NVFP4** build (`RadixArk/GLM-5.3-NVFP4`, Model Optimizer) that quantizes only the routed experts' linear weights and activations to 4-bit (attention, shared experts, dense layers, MTP, embeddings, and the LM head stay unquantized), cutting the weight footprint to \~0.45 TB so a 4-GPU GB300 node can serve it with TP4. **The NVFP4 cells are experimental** — benchmark data (accuracy / throughput) is still pending; treat them as provisional until the benchmark cards fill in.
Model Architecture Context
GLM-5.3 MoE · DSA · 256 experts (top-8) · MTP · FP8 1,048,576
GLM-5.3-BF16 MoE · DSA · 256 experts (top-8) · MTP · BF16 1,048,576
GLM-5.3-NVFP4 MoE · DSA · 256 experts (top-8) · MTP · NVFP4 1,048,576
**Recommended generation:** `temperature=1.0`, `top_p=0.95` (the checkpoint's `generation_config.json` defaults; informational — do not hardcode in client code). **Resources:** [GLM-5.3](https://huggingface.co/zai-org/GLM-5.3) · [GLM-5.3-BF16](https://huggingface.co/zai-org/GLM-5.3-BF16) · [GLM-5.3-NVFP4](https://huggingface.co/RadixArk/GLM-5.3-NVFP4). ## 2. Configuration Tips * **DeepSeek Sparse Attention (DSA).** GLM-5.3 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required. On Hopper, pairing `--kv-cache-dtype fp8_e4m3` with `--dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv` selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.3's 64 query heads match the kernel's native tile) — see the [DeepSeek-V3.2 page](../DeepSeek/DeepSeek-V3_2) for kernel details; the optional `SGLANG_ENABLE_DSA_Q8KV8_*` performance env vars are documented in `python/sglang/srt/environ.py`. * **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens`: lower the draft length when rejected draft tokens create excess verification work. * **DFlash2 (block-diffusion draft).** The **Speculative** card in the [Playground above](#playground) also offers **DFlash2**, which replaces the in-checkpoint MTP layer with the separately trained block-diffusion drafter [`incoai/GLM-5.3-DFlash2`](https://huggingface.co/incoai/GLM-5.3-DFlash2). It proposes a whole block per step and the target verifies the block in one forward pass, so output quality stays the target's. The block size — 8, i.e. 7 draft tokens per verification step — comes from the draft checkpoint's own `dflash_config`, so no `--speculative-num-draft-tokens` is passed; the draft is a small dense model and runs on `fa4` instead of the target's DSA backends. Note that DFLASH runs on **CUDA/NPU only** and rejects **DP-Attention**, so turn DP-Attention off in the **Attention** card before selecting it on a high-throughput base. The draft repository is public but licensed CC BY-NC-ND 4.0 for research and evaluation. * **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4). * **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP. * **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.3-BF16`, \~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, \~2.1 TB HBM); on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. * **PD Disaggregation (prefill/decode).** GLM-5.3 is a DSA model and runs under prefill/decode disaggregation — toggle the **PD Disagg** card in the [Playground above](#playground) (pick a Prefill/Decode role + transfer backend, then front the roles with `sglang_router.launch_router --pd-disaggregation`). The Mooncake backend **auto-detects the InfiniBand HCA**, so no device flag is needed by default; only add `--disaggregation-ib-device mlx5_0` (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (`--privileged --ulimit memlock=-1`, or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) — without IB exposure Mooncake silently falls back to TCP. * **Chunked-prefill size is regime-dependent.** For long-input balanced workloads, start with `--chunked-prefill-size 32768` and tune it together with `--max-running-requests` for your input length and KV capacity. Keep the default chunked-prefill size for the high-throughput recipe unless profiling shows that prefill is the bottleneck. * **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.3`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.3-BF16`, \~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there. Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). GLM-5.3 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2). * **MTP / EAGLE speculative decoding** is disabled for AMD in the Deploy panel because the gfx950 spec-decode draft kernel is not yet validated on this hardware (and at `--speculative-num-steps > 3` hits a separate build issue). Until MTP is validated on gfx950, omit the `--speculative-*` flags and serve without MTP. ## 3. Advanced Usage ### 3.1 Reasoning GLM-5.3 is a reasoning model, and generated commands enable `--reasoning-parser auto` (which resolves to `glm45` for GLM-5.3) by default so thinking is separated from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Without the parser the server returns the thinking and the answer as one `content` string with a stray `` between them, because the chat template opens `` in the generation prompt. You can disable **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground) when an integration needs that raw format. The chat template defaults `clear_thinking` to `false`; for multi-turn chat, pass `chat_template_kwargs: {"clear_thinking": True}` so previous reasoning is cleared before the next response. **Reasoning effort.** Pass `chat_template_kwargs: {"reasoning_effort": ...}` to select `low`, `high`, or `max`. If you omit it or pass another value, the template uses `max`. | `reasoning_effort` | Injected system line | Effect | | - | - | - | | *(not passed / unset)*, `"max"` | `Reasoning Effort: Max` | default — highest reasoning effort | | `"high"` | `Reasoning Effort: High` | high reasoning effort | | `"low"` | `Reasoning Effort: Low` | low reasoning effort | | any other value | `Reasoning Effort: Max` | falls back to the default | ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="zai-org/GLM-5.3", messages=[{"role": "user", "content": "What is 15% of 240?"}], extra_body={"chat_template_kwargs": {"clear_thinking": True, "reasoning_effort": "high"}}, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Reasoning: 1. **Identify the core question:** The user wants to find 15% of 240. 2. **Convert the percentage to a decimal:** 15% = 0.15 3. **Multiply by the total:** 0.15 * 240 = 36 (Quick mental math: 10% of 240 = 24; 5% = 12; 24 + 12 = 36.) Answer: 15% of 240 is **36**. Here is how you can calculate it: 0.15 × 240 = 36 ``` ### 3.2 Tool Calling Generated commands enable `--tool-call-parser auto` by default, so structured calls are returned in `message.tool_calls` with `finish_reason: "tool_calls"`. `auto` resolves to **`glm47`** for GLM-5.3: the model emits the newer `………` format, which the older `glm45` parser does not parse (the call would be left as raw text in `content`). Running with no tool-call parser fails the same way, and `finish_reason` stays `"stop"`, so an agent loop never sees the call. You can disable **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground) when tool calling is not needed. On thinking mode the turn also fills `reasoning_content`, so print both fields. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }] resp = client.chat.completions.create( model="zai-org/GLM-5.3", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Reasoning: The user wants to know the weather in Paris. I'll call the get_weather function with "Paris" as the city. Tool calls: [ { "id": "call_13fcd52146934b7781d06d4a", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"} } ] ``` ### 3.3 HiCache (Hierarchical KV Caching) For long-context, prefix-heavy workloads, enable hierarchical KV caching to spill cold KV blocks to host memory (toggle the **Hierarchical KV Cache** card in the [Playground above](#playground)). Useful given GLM-5.3's 1M-token window; pair `--hicache-ratio` with a write policy that matches your reuse pattern. ### 3.4 Claude Code Integration GLM-5.3's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.3 server with only environment variables — no code change. Launch the server with `--reasoning-parser auto --tool-call-parser auto` (any recipe from the Deployment panel above works), then: ```bash Command theme={null} export ANTHROPIC_BASE_URL="http://127.0.0.1:30000" export ANTHROPIC_AUTH_TOKEN="dummy" export API_TIMEOUT_MS="3000000" export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000" export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 export CLAUDE_CODE_ATTRIBUTION_HEADER=0 export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.3[1m]" export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.3[1m]" export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.3[1m]" claude ``` Two of these matter specifically for GLM-5.3: * **`CLAUDE_CODE_ATTRIBUTION_HEADER=0`** — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.3's chat template renders `tools` **before** `system`, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse. * **`glm-5.3[1m]`** as the model name — the `[1m]` suffix is the client-side hint that enables Claude Code's 1M-context beta, matching GLM-5.3's 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the `model` field, so any name is accepted server-side. For the full setup (streaming, tool-use, count\_tokens, persisting env in `~/.claude/settings.json`, troubleshooting), see [Anthropic-Compatible API](../../../docs/basic_usage/anthropic_api). ### 3.5 Context Parallelism Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.3. For prefill CP on CUDA, use `interleave` as shown below. Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.3, please append the following arguments: ```bash theme={null} --attn-cp-size 8 \ --enable-prefill-cp \ --cp-strategy interleave \ ``` which splits the sequence equally across `--attn-cp-size` ranks during attention forward. The trade off for prefill CP is that it will introduce extra all-gather operation before indexer-topk and attention kernels, so it will increase latency for decode (in unified deployment) or short prefill. You can combine interleave CP with attention DP. Each attention-DP group owns `tp_size / attn_dp_size` CP ranks; for example, `--tp 8 --attn-dp-size 2` uses CP4 within each DP group. Dense FFNs retain their configured TP width: the default runs them over the full TP group, while `--moe-dense-tp-size 1` runs them independently on local token shards. When deploying with PD Disaggregation, a **prefill worker using the Mooncake transfer backend** can enable the [LayerSplit](https://z.ai/blog/scaling-pain) technique with ```bash theme={null} --enable-dsa-cache-layer-split \ --enable-prefill-cp \ --attn-cp-size 8 \ --cp-strategy interleave \ ``` With LayerSplit, the kv cache on each rank can be sharded over the CP attention group, and prefetched when necessary. This can reduce kv cache memory by up to 75%, thus increasing the throughput on prefill side. # GLM-5.3-Flash Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3-Flash Deploy GLM-5.3-Flash with SGLang using recipes for H100, H200, B200, B300, GB200, GB300, and MI355X, with MTP and multimodal serving. ## Deployment Use an SGLang build that includes GLM-5.3-Flash support (v0.5.20 or later). ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` For the listed NVIDIA hardware, the deployment panel can render a complete `docker run` command. See [Install SGLang with Docker](/docs/get-started/install#method-3-using-docker) for host setup. For MI355X, follow the [AMD GPU installation guide](/docs/hardware-platforms/amd_gpu) and run the generated **Python** command on four GPUs. Select FP8 or [MXFP4](https://huggingface.co/OneNexus/GLM-5.3-Flash-MXFP4); SGLang downloads the model and tokenizer automatically. The cookbook opens on **B300**. Choose your hardware, then choose the operating point that matches your workload: * **Low Latency** starts with MTP 5/1/6 speculative decoding and tensor parallelism to shorten interactive responses. * **High Throughput** starts with speculative decoding off, which avoids draft-and-verify overhead under sustained batches. Every listed hardware platform exposes both strategies. A **Verified** badge means that exact hardware and command were tested. **Final Verification In Progress** means the recipe runs and is queued for measurement on the final weights. **Not Verified** means the command is a supported starting point that still needs workload validation. A choice is disabled only when the underlying runtime combination is known to be unsupported. The recommended selection is only a starting point. The same panel also lets you override the KV/DSA pairing, multimodal feature transport, Breakable Cuda Graph, and HiCache tiers. The MI355X commands are marked **Not Verified** because the selector defaults to HiCache off, whereas the measured AgentX configuration adds the host tier described in [Extend the cache hierarchy](#extend-the-cache-hierarchy). Do not treat the CUDA benchmark rows as MI355X measurements. **Breakable Cuda Graph** defaults to **Off**. On NVIDIA, select **On** to add `--cuda-graph-backend-prefill breakable` to the generated command. This requires a build that includes [PR #38522](https://github.com/sgl-project/sglang/pull/38522); the option is disabled on MI355X. Generated commands leave `--mamba-full-memory-ratio` at its `0.9` default, which is a generic starting point rather than a workload-tuned split: too low starves the KDA state pool and clamps `max_running_requests`, too high over-provisions it and shrinks the KV pool. Use the repo-local [`compute-mamba-ratio`](https://github.com/sgl-project/sglang/blob/main/.claude/skills/compute-mamba-ratio/SKILL.md) skill to compute the balanced ratio — or the `--max-mamba-cache-size` pin to use instead — from your average request length and the two pool sizes printed in one boot log. See [Size both memory pools](#size-both-memory-pools) for what each pool caps. ## Playground Use the Playground for lower-level tuning such as attention parallelism, MoE communication, PD disaggregation, speculative decoding, and reasoning or tool parsers. It inherits every selection from the deployment panel and shows only the command-line diff. The MoE backend defaults to `flashinfer_trtllm` on B300, B200, GB300 and GB200, to DeepGemm on Hopper, and to AITER on MI355X. ## 1. Model introduction GLM-5.3-Flash is a natively multimodal Mixture-of-Experts model built around a hybrid attention architecture — 320B total parameters with 18B active. Its 45 text layers combine MLA attention, DSA sparse attention, and KDA linear attention, and a 24-layer vision encoder handles image and video input. The checkpoint uses 288 routed experts with 8 active experts per token and includes a native MTP draft layer for speculative decoding. See the [GLM-5.3-Flash blog](https://z.ai/blog/glm-5.3-flash) and the [GLM-5 technical report](https://arxiv.org/abs/2602.15763) for training details.
Attribute Description
Architecture MoE with hybrid attention (MLA, DSA, KDA), mHC, and MTP — 320B total / 18B active parameters
Precision FP8 weights (zai-org/GLM-5.3-Flash); MXFP4 on MI355X (OneNexus/GLM-5.3-Flash-MXFP4); NVFP4 on Blackwell from RadixArk/GLM-5.3-Flash-NVFP4 and nvidia/GLM-5.3-Flash-NVFP4; FP8 KV cache by default on Blackwell and MI355X, BF16 KV cache on H100 and H200
Context 1M tokens
Inputs Text, image, and video
Generation defaults temperature=1.0, top\_p=0.95, thinking enabled
The deployment recipes inherit the checkpoint's 1,048,576-token context limit and generation configuration. Override sampling only when your application has its own evaluated settings. ## 2. Configuration tips ### Choose a strategy Start with **Low Latency** for chat and agent workloads. It drafts from the checkpoint's MTP head at a fixed depth (5 steps, top-k 1, 6 draft tokens) with natural acceptance. Measure **High Throughput** for heavily batched traffic where disabling speculative decoding can be more efficient. SGLang serves MTP through `--speculative-algorithm EAGLE` (upstream folds the older NEXTN spelling into EAGLE), so generated commands use that flag value. Strategy labels describe the workload goal, not a hardware restriction. Both strategies stay available when you switch hardware; only the verification badge changes. On MI355X, the measured AgentX frontier used Low Latency (EAGLE 5/1/6); High Throughput is an unmeasured speculation-off starting point. ### Change the speculative algorithm The **Speculative** card in the Playground changes the algorithm without leaving the selected strategy: * **EAGLE / MTP 5-1-6** is exactly what Low Latency serves, so a Low Latency base starts on this chip. Pick it from a High Throughput base to keep that recipe's other settings and add the MTP head. * **Off (greedy)** strips the whole `--speculative-*` family, which is what High Throughput already starts from. * **DFlash2** swaps the in-checkpoint MTP head for the trained block-diffusion draft in [`incoai/GLM-5.3-Flash-DFlash2`](https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2). The draft proposes a whole block per step and the target verifies it in one forward pass, so output quality stays the target's. Its block size comes from the draft checkpoint, and the draft runs on `fa4` rather than the target's DSA backends. The hidden-state capture it needs ([PR #36708](https://github.com/sgl-project/sglang/pull/36708)) shipped with the GLM-5.3-Flash support in v0.5.20, so the image pinned above is enough. The draft repository is access-gated: request access on its model page, then download it alongside the target before serving. This combination is not yet measured on the cookbook hardware, so treat it as a starting point. Neither algorithm runs with DP-Attention; the card disables the affected chips and names the reason. DFlash2's FA4 draft path is also disabled on MI355X. ### Size both memory pools GLM-5.3-Flash maintains a paged KV pool for attention and a separate KDA state pool. The KDA state pool can limit concurrency before the KV pool is full. If startup reduces `max_running_requests` because of KDA state capacity, increase `--mamba-full-memory-ratio` or set `--max-mamba-cache-size` for the expected concurrency, then tune `--max-running-requests` to the workload. Keep the prefix cache enabled for every strategy. Keep the checkpoint's KDA lower-bound setting unchanged. In particular, do not override `linear_lower_bound` through `--json-model-override-args`. ### Keep the KV and DSA backends paired On Blackwell, the recipes default to an FP8 KV cache with TRT-LLM DSA: on GB300 this pairing measured 2.9–5.7% higher throughput and about 1.8x the KV token capacity at identical pool bytes, with GSM8K accuracy within noise of BF16. BF16 KV with TileLang DSA remains selectable in the deployment panel and is the default on H100 and H200, where FP8 KV with TRT-LLM DSA is disabled. MI355X instead uses FP8 KV with TileLang DSA in the recorded ROCm recipe. Switch the dtype and both DSA backends together; TileLang DSA with FP8 KV is not a valid CUDA combination, and TRT-LLM DSA is not the MI355X backend. ### Extend the cache hierarchy Keep **HiCache** off when GPU memory is sufficient. Select **L1 + L2** to spill reusable cache entries into host memory. Select **+ L3** only after configuring Mooncake on every serving node; the generated command exposes the required configuration path. These options remain selectable but are marked **Not Verified** until the resulting command is validated on the chosen hardware. For agentic MXFP4 serving on MI355X, select **Low Latency** with **L1 + L2**. The host tier uses `--hicache-size 180` (about 190 GB of host memory per rank once the KDA state and DSA indexer pools are included, 750 GB per node) and requires a build that includes [PR #42178](https://github.com/sgl-project/sglang/pull/42178). With less free host memory, keep HiCache off; the server does not start when the host tier does not fit. ### Multimodal memory All strategies enable multimodal serving. Image and video input use `Glm5NextProcessor`, which the pinned Transformers release ships (the MI355X checks predate that release and used commit `e4052f55`). On MI355X, text, image, and an image-bearing follow-up were qualified with that build; video was not evaluated. The processor samples video at 2 FPS and caps video input at 240,000 visual tokens. Install `torchcodec` in the serving environment before sending video requests. For very long videos on 4x GB300, use encoder disaggregation to isolate the vision encoder's memory spikes from language decoding. The default multimodal feature transport is automatic, and on a single CUDA node auto resolves to CPU transport. The MI355X selection explicitly emits CPU transport, matching its image-serving check. CUDA IPC is opt-in: pass `--mm-feature-transport cuda_ipc` when lower transfer latency matters more than the GPU memory the IPC pool reserves. CUDA VMM transport applies only to multi-node GB200/GB300 systems on the MNNVL fabric, where auto selects it. ## 3. Advanced usage ### 3.1 Reasoning Thinking is enabled by the checkpoint's generation configuration, and generated commands enable `--reasoning-parser auto` (which resolves to `glm45` for GLM-5.3-Flash) by default. The OpenAI-compatible API then places thinking in `message.reasoning_content` and the final answer in `message.content`. You can disable **Reasoning Parser** in the Playground when an integration needs the raw response format. To disable thinking for a request, pass `chat_template_kwargs: {"thinking": false}` in the request body. ### 3.2 Tool calling Generated commands enable `--tool-call-parser auto` (which resolves to `glm47` for GLM-5.3-Flash) by default, so structured calls are returned in `message.tool_calls`. You can disable **Tool Call Parser** in the Playground when tool calling is not needed. On follow-up turns, read both `reasoning_content` and `content` because a thinking model can use either field around tool execution. ### 3.3 Multimodal serving The base recipes accept image and video content through the OpenAI-compatible chat API. Keep the processor defaults unless you have measured a different sampling or resize policy. Inputs above the video token budget are clamped to the processor's limit rather than rejected. ### 3.4 Encoder disaggregation Encoder disaggregation separates vision preprocessing from language inference. The verified topology uses one 4x GB300 node shared by an encoder-only TP4 process on port 30001 and a language-only TP4 process on port 30000. Start the encoder first. ```bash Command theme={null} sglang serve \ --model-path zai-org/GLM-5.3-Flash \ --tp-size 4 \ --encoder-only \ --host 0.0.0.0 \ --port 30001 ``` ```bash Command theme={null} sglang serve \ --model-path zai-org/GLM-5.3-Flash \ --tp-size 4 \ --attention-backend dsa \ --dsa-prefill-backend tilelang \ --dsa-decode-backend tilelang \ --linear-attn-backend triton \ --kv-cache-dtype bfloat16 \ --quantization fp8 \ --moe-runner-backend flashinfer_trtllm \ --max-running-requests 64 \ --chunked-prefill-size 8192 \ --max-prefill-tokens 8192 \ --disable-prefill-cuda-graph \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --language-only \ --encoder-urls http://localhost:30001 \ --mem-fraction-static 0.78 \ --host 0.0.0.0 \ --port 30000 ``` This topology served image requests and videos up to 238,080 visual tokens. In a concurrent long-video workload, the largest observed decode gap fell from 5.53 seconds in unified serving to 1.79 seconds with encoder disaggregation. Keep `--mem-fraction-static 0.78` on the language process so the encoder retains room for vision workspaces. Install `torchcodec` for video and see the [encoder disaggregation guide](/docs/advanced_features/epd_disaggregation) for the generic architecture and operational model. ### 3.5 Prefill Context Parallelism Use prefill context parallelism (CP) to distribute DSA prefill computation for long prompts, including multimodal inputs. This hybrid model uses **interleave CP for DSA layers** and **head-wise tensor parallelism for KDA layers**, sharing the same communication group. Layer boundaries all-gather the context shards before KDA and reduce-scatter its output back onto those shards. The mHC residual streams and their coefficients stay local, with each output's reduction completed before the residual update. In the Playground's **Attention** card, select **TP=8** and **CP=8** on B300 or B200 to add `--attn-cp-size 8 --enable-prefill-cp --cp-strategy interleave`. For CP4, set TP=4 and CP=4. With CP enabled, the **MoE** card restricts EP to **1** or the CP size; an incompatible EP selection falls back to 1. Choosing EP=CP also adds `--moe-dense-tp-size` with the CP size. Prefill CP and PD disaggregation cannot be enabled together for this model. ```bash Command theme={null} sglang serve \ --model-path zai-org/GLM-5.3-Flash \ --tp-size 4 \ --attn-cp-size 4 \ --enable-prefill-cp \ --cp-strategy interleave \ --moe-dense-tp-size 4 \ --ep-size 1 \ --speculative-algorithm EAGLE \ --speculative-num-steps 5 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 6 \ --dsa-prefill-backend trtllm \ --dsa-decode-backend trtllm \ --kv-cache-dtype fp8_e4m3 \ --moe-runner-backend flashinfer_trtllm \ --reasoning-parser auto \ --tool-call-parser auto \ --host 0.0.0.0 \ --port 30000 ``` ### 3.6 PD disaggregation (preview) PD splits prefill and decode into separate server groups behind a router. For this hybrid model, the transfer moves both the paged DSA KV and the KDA recurrent state. The Playground's **PD Disagg** card defaults to **Off**, with **Mooncake** as its only transfer backend. Select **Prefill** or **Decode** to add `--disaggregation-mode prefill` or `--disaggregation-mode decode`, respectively, together with `--disaggregation-transfer-backend mooncake`. Selecting either role disables prefill CP. The generated prefill and decode commands use ports 30000 and 30100, respectively. PD serving is mechanically validated with dummy weights only — startup, bootstrap, state transfer, and request flow all work on 4x GB300. It has not been load- or accuracy-tested. Treat it as a preview until the real-weight gate completes. ```bash Prefill (GPU 0-1) theme={null} sglang serve \ --model-path zai-org/GLM-5.3-Flash \ --tp-size 2 \ --dsa-prefill-backend tilelang \ --dsa-decode-backend tilelang \ --kv-cache-dtype bfloat16 \ --moe-runner-backend triton \ --disaggregation-mode prefill \ --disaggregation-bootstrap-port 8998 \ --disaggregation-transfer-backend nixl \ --host 0.0.0.0 \ --port 31000 ``` ```bash Decode (GPU 2-3) theme={null} sglang serve \ --model-path zai-org/GLM-5.3-Flash \ --tp-size 2 \ --base-gpu-id 2 \ --dsa-prefill-backend tilelang \ --dsa-decode-backend tilelang \ --kv-cache-dtype bfloat16 \ --moe-runner-backend triton \ --disaggregation-mode decode \ --disaggregation-transfer-backend nixl \ --host 0.0.0.0 \ --port 32000 ``` ```bash Router theme={null} python -m sglang_router.launch_router \ --pd-disaggregation \ --mini-lb \ --prefill http://127.0.0.1:31000 8998 \ --decode http://127.0.0.1:32000 \ --host 0.0.0.0 \ --port 30000 ``` The positional `8998` after `--prefill` must equal the prefill server's `--disaggregation-bootstrap-port`. Operational notes: * Give each role a distinct `--nccl-port` when both share one node. * Single-node NIXL needs `UCX_NET_DEVICES=lo` and `UCX_TLS=tcp,cuda_copy,cuda_ipc,self,sm` in both server environments. * The validated arm used the triton MoE runner; the flashinfer\_trtllm runner from the deployment recipes is untested under PD. Known limitations: * Speculative decoding does not start under PD at the current cut (draft-graph capture width assert on the prefill role; decode-side memory pressure at TP2). Run PD without speculative flags. * Prefill and decode with different TP sizes transfer state through slice paths, but numeric correctness is unverified. Keep both roles at the same TP size. # GLM Glyph Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-Glyph ## 1. Model Introduction [Glyph](https://huggingface.co/zai-org/Glyph) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding. **Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X **Key Features:** * **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving * **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs * **High Performance**: Optimized for both throughput and latency scenarios **Available Models:** * **BF16 (Full precision)**: [zai-org/Glyph](https://huggingface.co/zai-org/Glyph) * **FP8 (8-bit quantized)**: [zai-org/Glyph-FP8](https://huggingface.co/zai-org/Glyph-FP8) **License:** Please refer to the [official Glyph model card](https://huggingface.co/zai-org/Glyph) for license details. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and other options. ### 3.2 Configuration Tips * **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count. See the [GLM-4.5 cookbook page](/cookbook/autoregressive/GLM/GLM-4.5) for the full Thinking Budget usage example. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Thinking Mode Glyph supports thinking mode for enhanced reasoning. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model-path zai-org/Glyph \ --reasoning-parser glm45 \ --tp 4 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/Glyph", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. **Disable Thinking Mode:** To disable thinking mode for a specific request: ```python Example theme={null} response = client.chat.completions.create( model="zai-org/Glyph", messages=[{"role": "user", "content": "What is the capital of France?"}], extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) ``` #### 4.2.2 Tool Calling Glyph supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model-path zai-org/Glyph \ --reasoning-parser glm45 \ --tool-call-parser glm45 \ --tp 4 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/Glyph", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="zai-org/Glyph", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Model: Glyph * SGLang Version: 0.5.6.post1 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Scenario Benchmark * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model zai-org/Glyph \ --tp 2 ``` ##### 5.1.1.1 Low Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 17.03 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.59 Input token throughput (tok/s): 358.17 Output token throughput (tok/s): 247.74 Peak output token throughput (tok/s): 251.00 Peak concurrent requests: 3 Total token throughput (tok/s): 605.91 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1702.14 Median E2E Latency (ms): 1361.72 ---------------Time to First Token---------------- Mean TTFT (ms): 22.35 Median TTFT (ms): 22.61 P99 TTFT (ms): 23.76 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 3.99 Median TPOT (ms): 3.99 P99 TPOT (ms): 4.01 ---------------Inter-Token Latency---------------- Mean ITL (ms): 3.99 Median ITL (ms): 3.99 P95 ITL (ms): 4.03 P99 ITL (ms): 4.12 Max ITL (ms): 7.46 ================================================== ``` ##### 5.1.1.2 Medium Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 16.27 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40804 Request throughput (req/s): 4.92 Input token throughput (tok/s): 2438.06 Output token throughput (tok/s): 2507.94 Peak output token throughput (tok/s): 3069.00 Peak concurrent requests: 26 Total token throughput (tok/s): 4946.00 Concurrency: 13.44 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2733.43 Median E2E Latency (ms): 2892.98 ---------------Time to First Token---------------- Mean TTFT (ms): 33.10 Median TTFT (ms): 27.73 P99 TTFT (ms): 49.34 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.33 Median TPOT (ms): 5.39 P99 TPOT (ms): 5.86 ---------------Inter-Token Latency---------------- Mean ITL (ms): 5.30 Median ITL (ms): 4.89 P95 ITL (ms): 5.54 P99 ITL (ms): 21.17 Max ITL (ms): 25.14 ================================================== ``` ##### 5.1.1.3 High Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 25.67 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 252657 Request throughput (req/s): 19.48 Input token throughput (tok/s): 9733.69 Output token throughput (tok/s): 9843.99 Peak output token throughput (tok/s): 13398.00 Peak concurrent requests: 127 Total token throughput (tok/s): 19577.68 Concurrency: 89.49 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4593.75 Median E2E Latency (ms): 4431.03 ---------------Time to First Token---------------- Mean TTFT (ms): 48.66 Median TTFT (ms): 35.88 P99 TTFT (ms): 120.61 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.10 Median TPOT (ms): 9.55 P99 TPOT (ms): 11.00 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.01 Median ITL (ms): 6.51 P95 ITL (ms): 23.19 P99 ITL (ms): 25.54 Max ITL (ms): 52.93 ================================================== ``` #### 5.1.2 Reasoning Scenario Benchmark ##### 5.1.2.1 Low Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 201.53 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 44462 Total generated tokens (retokenized): 44455 Request throughput (req/s): 0.05 Input token throughput (tok/s): 30.27 Output token throughput (tok/s): 220.63 Peak output token throughput (tok/s): 251.00 Peak concurrent requests: 2 Total token throughput (tok/s): 250.90 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 20151.45 Median E2E Latency (ms): 21576.31 ---------------Time to First Token---------------- Mean TTFT (ms): 2362.23 Median TTFT (ms): 23.03 P99 TTFT (ms): 21310.14 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.00 Median TPOT (ms): 4.00 P99 TPOT (ms): 4.01 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.00 Median ITL (ms): 4.00 P95 ITL (ms): 4.05 P99 ITL (ms): 4.08 Max ITL (ms): 5.67 ================================================== ``` ##### 5.1.2.2 Medium Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 118.67 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 318306 Total generated tokens (retokenized): 318270 Request throughput (req/s): 0.67 Input token throughput (tok/s): 334.27 Output token throughput (tok/s): 2682.26 Peak output token throughput (tok/s): 3264.00 Peak concurrent requests: 19 Total token throughput (tok/s): 3016.53 Concurrency: 13.74 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 20387.23 Median E2E Latency (ms): 20466.09 ---------------Time to First Token---------------- Mean TTFT (ms): 132.47 Median TTFT (ms): 27.19 P99 TTFT (ms): 583.15 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.09 Median TPOT (ms): 5.13 P99 TPOT (ms): 5.19 ---------------Inter-Token Latency---------------- Mean ITL (ms): 5.09 Median ITL (ms): 5.08 P95 ITL (ms): 5.18 P99 ITL (ms): 5.57 Max ITL (ms): 522.26 ================================================== ``` ##### 5.1.2.3 High Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 150.00 Total input tokens: 158939 Total input text tokens: 158939 Total input vision tokens: 0 Total generated tokens: 1301025 Total generated tokens (retokenized): 1300901 Request throughput (req/s): 2.13 Input token throughput (tok/s): 1059.59 Output token throughput (tok/s): 8673.49 Peak output token throughput (tok/s): 11899.00 Peak concurrent requests: 71 Total token throughput (tok/s): 9733.09 Concurrency: 54.71 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 25645.42 Median E2E Latency (ms): 26913.26 ---------------Time to First Token---------------- Mean TTFT (ms): 163.75 Median TTFT (ms): 93.67 P99 TTFT (ms): 426.19 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.27 Median TPOT (ms): 6.39 P99 TPOT (ms): 6.59 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.27 Median ITL (ms): 0.17 P95 ITL (ms): 32.94 P99 ITL (ms): 67.89 Max ITL (ms): 136.00 ================================================== ``` #### 5.1.3 Summarization Scenario Benchmark #### 5.1.3.1 Low Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 17.44 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.57 Input token throughput (tok/s): 2405.19 Output token throughput (tok/s): 242.00 Peak output token throughput (tok/s): 250.00 Peak concurrent requests: 2 Total token throughput (tok/s): 2647.19 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1742.54 Median E2E Latency (ms): 1412.47 ---------------Time to First Token---------------- Mean TTFT (ms): 53.48 Median TTFT (ms): 45.05 P99 TTFT (ms): 98.57 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.01 Median TPOT (ms): 4.01 P99 TPOT (ms): 4.03 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.01 Median ITL (ms): 4.01 P95 ITL (ms): 4.06 P99 ITL (ms): 4.09 Max ITL (ms): 4.95 ================================================== ``` ##### 5.1.3.2 Medium Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 16.90 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41669 Total generated tokens (retokenized): 41668 Request throughput (req/s): 4.73 Input token throughput (tok/s): 17753.58 Output token throughput (tok/s): 2465.75 Peak output token throughput (tok/s): 3005.00 Peak concurrent requests: 25 Total token throughput (tok/s): 20219.33 Concurrency: 13.68 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2890.33 Median E2E Latency (ms): 3069.55 ---------------Time to First Token---------------- Mean TTFT (ms): 41.46 Median TTFT (ms): 31.75 P99 TTFT (ms): 93.18 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.52 Median TPOT (ms): 5.58 P99 TPOT (ms): 6.14 ---------------Inter-Token Latency---------------- Mean ITL (ms): 5.48 Median ITL (ms): 5.13 P95 ITL (ms): 5.93 P99 ITL (ms): 20.76 Max ITL (ms): 36.01 ================================================== ``` ##### 5.1.3.3 High Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 35.54 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 170000 Total generated tokens (retokenized): 169994 Request throughput (req/s): 9.01 Input token throughput (tok/s): 35848.57 Output token throughput (tok/s): 4783.96 Peak output token throughput (tok/s): 8396.00 Peak concurrent requests: 80 Total token throughput (tok/s): 40632.53 Concurrency: 59.26 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6580.96 Median E2E Latency (ms): 6248.74 ---------------Time to First Token---------------- Mean TTFT (ms): 345.27 Median TTFT (ms): 96.06 P99 TTFT (ms): 2823.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 12.26 Median TPOT (ms): 12.53 P99 TPOT (ms): 23.58 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.76 Median ITL (ms): 6.57 P95 ITL (ms): 27.66 P99 ITL (ms): 91.24 Max ITL (ms): 2609.64 ================================================== ``` ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * Test Result ```text Output theme={null} Accuracy: 0.890 Invalid: 0.000 Latency: 3.718 s Output throughput: 5245.606 token/s ``` # GLM-OCR Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-OCR ## 1. Model Introduction [GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) is a multimodal OCR model for complex document understanding, built on the GLM-V encoder–decoder architecture. It introduces Multi-Token Prediction (MTP) loss and stable full-task reinforcement learning to improve training efficiency, recognition accuracy, and generalization. The model integrates the CogViT visual encoder pre-trained on large-scale image–text data, a lightweight cross-modal connector with efficient token downsampling, and a GLM-0.5B language decoder. Combined with a two-stage pipeline of layout analysis and parallel recognition based on PP-DocLayout-V3, GLM-OCR delivers robust and high-quality OCR performance across diverse document layouts. **Hardware Support:** NVIDIA B200/H100/H200 **Key Features:** * **State-of-the-Art Performance**: Achieves 94.62 on OmniDocBench V1.5, ranking #1, and delivers SOTA results across major document understanding benchmarks, including formula recognition, table recognition, and information extraction. * **Optimized for Real-World Scenarios**: Specifically optimized for practical business cases, maintaining stable and accurate performance on complex tables, code documents, seals, and other challenging layouts. * **Efficient Inference**: With only 0.9B parameters, GLM-OCR supports deployment via vLLM and SGLang, significantly reducing inference latency and compute cost—well suited for high-concurrency and edge deployments. * **Easy to Use**: Fully open-sourced with a complete SDK and inference toolchain, enabling one-line invocation and seamless integration into existing systems. For more details, please refer to the [official GLM-OCR model card](https://huggingface.co/zai-org/GLM-OCR). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options. You can optionally enable MTP (Multi-Token Prediction) for faster inference using EAGLE speculative decoding. ### 3.2 Configuration Tips * **CUDA IPC Transport**: The `SGLANG_USE_CUDA_IPC_TRANSPORT=1` environment variable enables CUDA IPC for transferring multimodal features, which significantly improves TTFT. * **MTP (Multi-Token Prediction)**: Enable MTP to use EAGLE speculative decoding for faster inference. This feature predicts multiple tokens at once to reduce latency. * **Memory Management**: For memory-constrained environments, you may need to adjust `--mem-fraction-static` and/or `--max-running-requests`. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 OCR Image Processing GLM-OCR supports OCR tasks on various document types. Here's a basic example: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Please extract all text from this image." } ] } ] start = time.time() response = client.chat.completions.create( model="zai-org/GLM-OCR", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 2.29s Generated text: CINNAMON SUGAR 1 x 17,000 17,000 SUB TOTAL 17,000 GRAND TOTAL 17,000 CASH IDR 20,000 CHANGE DUE 3,000 ``` #### 4.2.2 Complex Document Processing GLM-OCR excels at processing complex documents including: * **Tables**: Accurate extraction of tabular data with structure preservation * **Formulas**: Mathematical formula recognition * **Code Documents**: Source code extraction from screenshots * **Seals and Stamps**: Recognition of seals and stamps in documents * **Multi-layout Documents**: Mixed content with text, images, and tables ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) # Example: Processing a document with tables messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "YOUR_DOCUMENT_IMAGE_URL" } }, { "type": "text", "text": "Please extract the table content from this document and format it as markdown." } ] } ] response = client.chat.completions.create( model="zai-org/GLM-OCR", messages=messages, max_tokens=4096 ) print(response.choices[0].message.content) ``` ## 5. Benchmark ### 5.1 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.1.1 OCRBench Benchmark * Benchmark Command ```bash Command theme={null} python3 -m lmms_eval \ --model openai_compatible \ --model_args "model_version=zai-org/GLM-OCR" \ --tasks ocrbench \ --batch_size 128 \ --log_samples \ --log_samples_suffix "openai_compatible" \ --output_path ./logs ``` * Test Result
Tasks Version Filter n-shot Metric Value Stderr
ocrbench Yaml none 0 ocrbench\_accuracy ↑ 0.806 N/A
#### 5.1.2 OmniDocBench V1.5 GLM-OCR achieves **94.62** on OmniDocBench V1.5, ranking #1 among all models, demonstrating state-of-the-art performance across major document understanding benchmarks. # GigaChat 3.5 Source: https://docs.sglang.io/cookbook/autoregressive/GigaChat/GigaChat3.5 Deploy GigaChat 3.5 with SGLang — the 432B-A28B hybrid MLA + GDN Mixture-of-Experts instant model with MTP speculative decoding and tool calling, on H100 and H200 in FP8. ## Deployment For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). GigaChat 3.5 support is on `main` ([PR #29189](https://github.com/sgl-project/sglang/pull/29189), merged 2026-09-21) and not yet in a `pip` release — install from source: ```bash Command theme={null} pip install --upgrade pip pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' ``` Then run the **Python** output of the command panel below. No released `lmsysorg/sglang` tag carries GigaChat 3.5 yet. The nightly `lmsysorg/sglang:dev` is rebuilt from `main` and carries the model in builds after 2026-09-21, but that path has not been run end-to-end, so the command panel shows only the Python run mode. Run the Python command inside the nightly image; the Docker tab returns to the panel after a verified run. ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. This page covers the **instant** checkpoint (direct answers, no thinking). For the thinking checkpoint, see [GigaChat 3.5 Reasoning](/cookbook/autoregressive/GigaChat/GigaChat3.5-Reasoning). Pick your hardware and recipe to generate the launch command. Two serving strategies cover the operating points: * **Low-Latency** — MTP speculative decoding on, Mamba state pool pinned. Fastest reply per user; pick for chat and agent loops. * **High-Throughput** — MTP off. At saturation the draft + verify overhead outweighs the speedup, so this is the recipe for batch jobs and many concurrent users. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been run end-to-end; the Playground lets you turn on additional knobs — the tool-call parser, the MTP preset, parallelism — on top of whichever cell the Deploy panel is currently showing. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end. ## 1. Model Introduction **GigaChat 3.5** is a hybrid Mixture-of-Experts model from GigaChat — **432B** total parameters, **28B** active per token, **256K** context (YaRN, factor 8 over a 32K base), released under the **MIT** license. Its **40** layers interleave two attention types: **10** full-attention **MLA** layers (DeepSeek-style latent attention, every 4th layer) and **30** **GDN** linear-attention layers (Qwen3-Next gated delta-net, constant-size state). Only the MLA layers hold a KV cache; the linear layers keep their state in a Mamba-style pool. The MoE block routes to **8 of 256** experts plus **1** shared expert. The checkpoint ships **2** stacked **MTP** draft heads for speculative decoding, and tool calls are emitted in GigaChat's **GCML** format. This is the **instant** release: it answers directly, without a thinking block. The separately trained **Reasoning** release always thinks first and ships 3 MTP heads — it has [its own page](/cookbook/autoregressive/GigaChat/GigaChat3.5-Reasoning). **Resources:** HuggingFace — [GigaChat3.5-432B-A28B](https://huggingface.co/ai-sage/GigaChat3.5-432B-A28B) (FP8, served here) · [GigaChat3.5-432B-A28B-bf16](https://huggingface.co/ai-sage/GigaChat3.5-432B-A28B-bf16) (BF16, for fine-tuning; no single-node recipe). ## 2. Configuration Tips **Two pools.** GDN layers keep per-request state in a Mamba-style pool, MLA layers use a paged KV pool; `--mamba-full-memory-ratio` (default 0.9) splits the post-weight budget. On 8×H100 at `--mem-fraction-static 0.8` that is \~51 GB of weights per GPU, 347 state slots and 542k KV tokens. The resolver forces `--mamba-radix-cache-strategy no_buffer` (3 slots per request, overlap scheduler off), so the unpinned server admits 115 concurrent requests. **MTP needs a pinned pool.** Each running request holds extra state copies under speculative decoding, and the default ratio then admits only a few dozen. The Low-Latency recipe pins `--max-mamba-cache-size 240` with `--max-running-requests 80`; the KV pool drops to 185k tokens. Keep `--mem-fraction-static 0.8` on 80 GB — it is already tight in eager prefill at 16 concurrent 8k-token prompts. **Capacity levers** (toggles in the [Playground](#playground)). `--mamba-ssm-dtype bfloat16` halves every state slot at no cost in accuracy or speed. `--kv-cache-dtype fp8_e4m3` doubles the KV pool but decodes \~2.3× slower on H100 and cannot be combined with MTP on 80 GB. **MTP flags.** `--speculative-algorithm EAGLE` uses GigaChat's own NextN heads through the multi-layer EAGLE worker, selected automatically. This checkpoint has **2** heads: `--speculative-num-steps 2`, `--speculative-num-draft-tokens 3`. **Parsers.** `--tool-call-parser gigachat35` for structured tool calls (§3.1). No reasoning parser, not even `auto`: the checkpoint never emits ``, and the forced `gigachat35` splitter would move the whole answer into `reasoning_content`. ## 3. Advanced Usage ### 3.1 Tool Calling Enable the `gigachat35` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured calls via `message.tool_calls`. GigaChat emits tool calls in its **GCML** format — a `<｜GCML｜tool_calls>` … `` block holding one `<｜GCML｜invoke name="…">` per call — inside the assistant turn; the parser reads them out and strips the block from `content`. `--tool-call-parser auto` also resolves to `gigachat35` for this checkpoint, since its chat template carries the GCML marker. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string", "description": "The city name"}}, "required": ["city"], }, }, } ] resp = client.chat.completions.create( model="ai-sage/GigaChat3.5-432B-A28B", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=tools, ) msg = resp.choices[0].message print("Content:", msg.content) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Content: Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_aa689732d4884654a1f76762', function=Function(arguments='{"city": "Paris"}', name='get_weather'), type='function', index=0)] ``` Appending a `tool` message with `{"city": "Paris", "temp_c": 9, "condition": "light rain"}` and re-requesting returns the final answer with no further tool call: ```text Output theme={null} Content: The current weather in Paris is light rain with a temperature of 9°C. Tool calls: None ``` GigaChat tends to call tools **sequentially** — one hop, then the next — so a multi-tool request may come back with a single call. Drive it with an agent loop that appends each `tool` result and re-requests until `finish_reason` is no longer `tool_calls`. Parallel calls (several invokes in one block) are supported by the format when the model chooses them. ### 3.2 MTP (Speculative Decoding) GigaChat 3.5 ships its own NextN draft heads, so speculative decoding needs no separate draft model: `--speculative-algorithm EAGLE` runs the stacked heads through SGLang's multi-layer EAGLE worker, which is selected automatically for this architecture. The **Low-Latency** recipe has it on; in the [Playground](#playground) the **Speculative** card adds the same preset to any other cell. ```bash Flags theme={null} --speculative-algorithm EAGLE \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 \ --max-running-requests 80 \ --max-mamba-cache-size 240 ``` * **Steps follow the checkpoint.** This checkpoint has 2 draft heads, so use `--speculative-num-steps 2` and `--speculative-num-draft-tokens 3` (steps + 1); the worker rejects other values at startup. * **Pin the state pool.** Under MTP every running request holds extra GDN state copies for verification. Without `--max-mamba-cache-size` and `--max-running-requests` the resolver clamps concurrency to a few dozen requests; §2 explains what the pin costs in context. The panel shows an amber callout whenever a speculative command lacks `--max-running-requests`. * **Where it pays off.** Low concurrency: accepted draft tokens skip target forwards, so single-request decode runs about 2× faster with the output distribution unchanged. At high concurrency the verify work and the smaller KV pool eat the gain, which is why the High-Throughput recipe leaves MTP off. Measured numbers are in the benchmark card above. # GigaChat 3.5 Reasoning Source: https://docs.sglang.io/cookbook/autoregressive/GigaChat/GigaChat3.5-Reasoning Deploy GigaChat 3.5 Reasoning with SGLang — the 432B-A28B hybrid MLA + GDN Mixture-of-Experts thinking model with reasoning parsing, MTP speculative decoding and tool calling, on H100 and H200 in FP8. ## Deployment For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). GigaChat 3.5 support is on `main` ([PR #29189](https://github.com/sgl-project/sglang/pull/29189), merged 2026-09-21) and not yet in a `pip` release — install from source: ```bash Command theme={null} pip install --upgrade pip pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' ``` Then run the **Python** output of the command panel below. No released `lmsysorg/sglang` tag carries GigaChat 3.5 yet. The nightly `lmsysorg/sglang:dev` is rebuilt from `main` and carries the model in builds after 2026-09-21, but that path has not been run end-to-end, so the command panel shows only the Python run mode. Run the Python command inside the nightly image; the Docker tab returns to the panel after a verified run. ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. This page covers the **Reasoning** checkpoint (always thinks before answering). For the direct-answer checkpoint, see [GigaChat 3.5](/cookbook/autoregressive/GigaChat/GigaChat3.5). Pick your hardware and recipe to generate the launch command. Two serving strategies cover the operating points: * **Low-Latency** — MTP speculative decoding on, Mamba state pool pinned to 80 requests. Fastest reply per user: measured 2× decode speed at batch 1 and 1.5× at 16 concurrent 8k-token requests. Pick for chat and agent loops. * **High-Throughput** — MTP off, pools left to the default ratio, so the KV pool is \~2.7× larger. Past a few dozen concurrent long-context requests the pinned recipe starts queueing prompts on KV space, and at saturation the draft + verify overhead no longer pays; this is the recipe for batch jobs and many concurrent users. H100 numbers were measured on 8×H100 SXM at `b63f8416b3` (the PR #29189 merge on `main`) with `bench_serving`, random ISL 8192 / OSL 1024, `--random-range-ratio 1.0`, `--warmup-requests 4`, `--flush-cache`; latencies are P50. GSM8K was measured with the 5-shot chat harness `sglang.test.run_eval --eval-name gsm8k --api chat` (1314 scored questions, `max_tokens` 8192), so the model thinks before every answer exactly as in production. That harness was deprecated upstream after these runs; the **Reproduce** button emits its successor, `sgl-eval run gsm8k` (single-shot), which can land a point or two away from the numbers shown. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been run end-to-end; the Playground lets you turn on additional knobs — the reasoning and tool-call parsers, the MTP preset, parallelism — on top of whichever cell the Deploy panel is currently showing. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end. ## 1. Model Introduction **GigaChat 3.5 Reasoning** is the thinking release of GigaChat's hybrid Mixture-of-Experts model — **432B** total parameters, **28B** active per token, **256K** context (YaRN, factor 8 over a 32K base), released under the **MIT** license. It shares the base architecture with [GigaChat 3.5](/cookbook/autoregressive/GigaChat/GigaChat3.5): **40** layers interleaving **10** full-attention **MLA** layers (DeepSeek-style latent attention, every 4th layer) with **30** **GDN** linear-attention layers (Qwen3-Next gated delta-net, constant-size state), and an MoE block routing to **8 of 256** experts plus **1** shared expert. Only the MLA layers hold a KV cache; the linear layers keep their state in a Mamba-style pool. What differs: the chat template opens every assistant turn with ``, so the model always reasons before answering (there is no template switch to turn it off), and the checkpoint ships **3** stacked **MTP** draft heads instead of 2. Tool calls use the same **GCML** format. The model card reports the largest gains over the instant release in mathematics, code, instruction following and structured output. **Recommended generation:** the model card suggests `temperature=0.6` with a generous `max_tokens` (thinking alone can run to thousands of tokens). Informational only — the sample code below relies on the checkpoint's `generation_config.json` defaults. **Resources:** HuggingFace — [GigaChat3.5-432B-A28B-Reasoning](https://huggingface.co/ai-sage/GigaChat3.5-432B-A28B-Reasoning) (FP8, served here) · [GigaChat3.5-432B-A28B-Reasoning-bf16](https://huggingface.co/ai-sage/GigaChat3.5-432B-A28B-Reasoning-bf16) (BF16, for fine-tuning; no single-node recipe). ## 2. Configuration Tips **Two pools.** GDN layers keep per-request state in a Mamba-style pool, MLA layers use a paged KV pool; `--mamba-full-memory-ratio` (default 0.9) splits the post-weight budget. On 8×H100 at `--mem-fraction-static 0.8` that is \~51 GB of weights per GPU, 348 state slots and 542k KV tokens. The resolver forces `--mamba-radix-cache-strategy no_buffer` (3 slots per request, overlap scheduler off), so the unpinned server admits 116 concurrent requests. **MTP needs a pinned pool and 0.83.** Each running request holds extra state copies under speculative decoding, and the default ratio then admits only a few dozen. The Low-Latency recipe pins `--max-mamba-cache-size 240` with `--max-running-requests 80` and runs at `--mem-fraction-static 0.83` (KV pool 199k tokens); at 0.85 the server ran out of memory in eager prefill at 16 concurrent 8k-token prompts. To size by workload instead of a pin: `r = (3 + draft_tokens) × 16 MB / (11.5 KB × L)` for an average request length `L`, and with MTP keep running 8k-token requests under \~20, since the EAGLE worker holds each prompt's hidden states outside both pools. **Capacity levers** (toggles in the [Playground](#playground)). `--mamba-ssm-dtype bfloat16` halves every state slot at no cost in accuracy or accept length — on the Low-Latency recipe the KV pool grows from 199k to 496k tokens. `--kv-cache-dtype fp8_e4m3` doubles the KV pool but decodes \~2.3× slower on H100 and cannot be combined with MTP on 80 GB. **MTP flags.** `--speculative-algorithm EAGLE` uses GigaChat's own NextN heads through the multi-layer EAGLE worker, selected automatically. This checkpoint has **3** heads: `--speculative-num-steps 3`, `--speculative-num-draft-tokens 4`. **Parsers.** `--reasoning-parser gigachat35` splits thinking from the answer (§3.1), `--tool-call-parser gigachat35` gives structured tool calls (§3.2); `auto` resolves to `gigachat35` for both. Do not use the reasoning parser on the Instant checkpoint. ## 3. Advanced Usage ### 3.1 Reasoning The chat template opens every assistant turn with ``, and there is no template switch to turn thinking off. Enable the `gigachat35` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to split the thinking from the final answer into `reasoning_content` vs `content`. Without the parser, the raw think block is returned inline in `content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="ai-sage/GigaChat3.5-432B-A28B-Reasoning", messages=[{"role": "user", "content": "What is 17 times 24?"}], max_tokens=4096, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Reasoning: I need to calculate 17 times 24. I can do this by breaking it down into smaller parts. First, I know that 10 times 24 is 240, and 7 times 24 is... let me figure that out. 7 times 20 is 140, and 7 times 4 is 28, so 140 plus 28 is 168. Then, I add the two results: 240 plus 168. 240 plus 168: 240 plus 100 is 340, plus 60 is 400, plus 8 is 408. Wait, that doesn't seem right. Let me add them properly. 240 + 168: 0 + 8 is 8, 4 + 6 is 10, carry over the 1, so 2 + 1 is 3, plus the carried over 1 is 4, so it should be 408. But I think I remember that 17 times 24 is 408, but let me double-check. [... ~1,400 more thinking tokens ...] Answer: 17 multiplied by 24 equals 408. This can be calculated in several ways: ### Method 1: Breaking Down the Numbers - $17 \times 24 = 17 \times (20 + 4)$ - First, $17 \times 20 = 340$ - Then, $17 \times 4 = 68$ - Add the results: $340 + 68 = 408$ ### Method 2: Using the Distributive Property - $17 \times 24 = (10 + 7) \times 24$ - $10 \times 24 = 240$ - $7 \times 24 = 168$ - Add the results: $240 + 168 = 408$ [...] ``` The full response was 1,641 completion tokens, 24.6 s at batch size 1 on 8×H100 without MTP. The model can spend hundreds of tokens thinking, so keep `max_tokens` generous. A response with an empty `content` and `finish_reason: "length"` means the think block was cut off before it closed — raise the limit. ### 3.2 Tool Calling Enable the `gigachat35` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured calls via `message.tool_calls`. GigaChat emits tool calls in its **GCML** format — a `<｜GCML｜tool_calls>` … `` block holding one `<｜GCML｜invoke name="…">` per call — inside the assistant turn; the parser reads them out and strips the block from `content`. `--tool-call-parser auto` also resolves to `gigachat35` for this checkpoint, since its chat template carries the GCML marker. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string", "description": "The city name"}}, "required": ["city"], }, }, } ] resp = client.chat.completions.create( model="ai-sage/GigaChat3.5-432B-A28B-Reasoning", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Reasoning: Okay, the user is asking about the weather in Paris. Let me check what tools I have available. There's the get_weather function which requires the city name. Since the user provided "Paris" directly, I can use that function. I just need to make sure to format the tool call correctly with the city parameter. No other information is needed here, so I'll call get_weather with city set to Paris. Content: Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_ef4e35b959c648fbaeab9afd', function=Function(arguments='{"city": "Paris"}', name='get_weather'), type='function', index=0)] ``` Appending a `tool` message with `{"city": "Paris", "temp_c": 9, "condition": "light rain"}` and re-requesting returns the final answer with no further tool call: ```text Output theme={null} Reasoning: Content: The current weather in Paris is **9°C** with **light rain**. Tool calls: None ``` The tool-call turn carries the model's thinking in `reasoning_content`, so print it alongside `content` and `tool_calls`. GigaChat tends to call tools **sequentially** — one hop, then the next — so a multi-tool request may come back with a single call. Drive it with an agent loop that appends each `tool` result and re-requests until `finish_reason` is no longer `tool_calls`. Parallel calls (several invokes in one block) are supported by the format when the model chooses them. ### 3.3 MTP (Speculative Decoding) GigaChat 3.5 ships its own NextN draft heads, so speculative decoding needs no separate draft model: `--speculative-algorithm EAGLE` runs the stacked heads through SGLang's multi-layer EAGLE worker, which is selected automatically for this architecture. The **Low-Latency** recipe has it on; in the [Playground](#playground) the **Speculative** card adds the same preset to any other cell. ```bash Flags theme={null} --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --max-running-requests 80 \ --max-mamba-cache-size 240 ``` * **Steps follow the checkpoint.** This checkpoint has 3 draft heads, so use `--speculative-num-steps 3` and `--speculative-num-draft-tokens 4` (steps + 1); the worker rejects other values at startup. * **Pin the state pool.** Under MTP every running request holds extra GDN state copies for verification. Without `--max-mamba-cache-size` and `--max-running-requests` the resolver clamps concurrency to a few dozen requests; §2 explains what the pin costs in context and why the recipe runs at `--mem-fraction-static 0.83`. The panel shows an amber callout whenever a speculative command lacks `--max-running-requests`. * **Where it pays off.** Long think blocks are long, low-entropy decode, exactly what draft heads predict well: on 8×H100 single-request decode runs about 2× faster and 16 concurrent 8k-token requests about 1.5× faster, with GSM8K unchanged. At 64 concurrent requests the verify work and the smaller KV pool make it slower than the plain recipe, which is why the High-Throughput recipe leaves MTP off. Measured numbers are in the benchmark card above. # DiffusionGemma Source: https://docs.sglang.io/cookbook/autoregressive/Google/DiffusionGemma ## 1. Model Introduction DiffusionGemma is a uniform-state (renoising) block-diffusion language model from Google. An encoder builds causal context, and a decoder denoises a fixed-length bidirectional canvas of `canvas_length` tokens. The `Gemma4Renoise` sampler runs `max_denoising_steps` reverse steps over the canvas, feeding the previous step's logits back as self-conditioning and emitting the greedy argmax of the processed logits. **Key Features:** * **Uniform-State Renoising**: The canvas starts from random tokens and is refined each step by accepting confident positions and re-noising the rest, with no mask token. * **Encoder / Decoder Canvas**: The encoder produces causal context KV, the decoder attends bidirectionally over the canvas. * **Self-Conditioning**: Each step conditions on the previous step's logits. * **EntropyBound Acceptance**: Each step accepts the lowest-entropy canvas positions within an entropy budget and re-noises the rest. * **StableAndConfident Stopping**: A canvas stops early once it is stable and confident. * **MoE Architecture**: The 26B-A4B model uses a Mixture-of-Experts architecture for efficient inference. * **Multimodal Input**: Accepts text and image inputs (via a \~550M vision encoder) and generates text output. **Available Models:**
Model Architecture Parameters
[google/diffusiongemma-26B-A4B-it](https://huggingface.co/google/diffusiongemma-26B-A4B-it) MoE, uniform-state diffusion (text + image) 25.2B total / 3.8B active
**Architecture Specifications:** | Spec | Value | | - | - | | Total Parameters | 25.2B | | Active Parameters | 3.8B | | Layers | 30 | | Sliding Window | 1024 tokens | | Context Length | Up to 256K tokens | | Canvas Length | 256 | | Vocabulary Size | 262K | | Experts | 8 active / 128 total + 1 shared | | Supported Modalities | Text, Image | | Vision Encoder | \~550M parameters | **License:** Refer to the model card for license details. ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. The checkpoint ships its own modeling code, so `--trust-remote-code` is required when serving. ## 3. Model Deployment ### 3.1 Basic Configuration The required runtime settings are applied automatically for `Gemma4Renoise` (the Triton attention backend, eager mode, and unchunked prefill, needed because the full-attention head\_dim is 512 and the canvas uses bidirectional attention), so a default launch works: ```bash Command theme={null} sglang serve \ --model-path google/diffusiongemma-26B-A4B-it \ --dllm-algorithm Gemma4Renoise \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` ### 3.2 Configuration Tips **dLLM-Specific Parameters:**
Parameter Description Recommended Value
`--dllm-algorithm` Diffusion decoding algorithm `Gemma4Renoise`
`--trust-remote-code` Required to load the checkpoint's modeling code Always enabled
`--dllm-algorithm-config` Optional YAML overriding the renoise schedule Checkpoint defaults
The attention backend, eager mode, and unchunked prefill are selected automatically for `Gemma4Renoise`, so they do not need to be passed on the command line. Sampling is governed by the renoise schedule. Request-level `logprobs`, penalties, `logit_bias`, and grammar / structured output (`json_schema` / `regex` / `ebnf` / `structural_tag`) are not applied and are rejected with a 400. Core sampling controls (`temperature`, `top_k`, `top_p`) are accepted but have no effect. Streaming is block-level: one fully-denoised canvas per chunk. **Gemma4Renoise Config** (defaults follow the checkpoint's `generation_config.json`): ```yaml Config theme={null} # Number of reverse denoising steps per canvas. max_denoising_steps: 48 # Optional. Makes the renoise sampling reproducible (also shared across TP ranks). seed: 1234 sampler_config: # Entropy budget. Accept the lowest-entropy canvas positions within this bound each step (the rest are re-noised). entropy_bound: 0.1 # Linear temperature schedule applied over the denoising steps. temperature_schedule: t_min: 0.4 t_max: 0.8 # Stop early once the canvas is stable and confident. stopping_config: confidence_threshold: 0.005 stability_threshold: 1 ``` ## 4. Model Invocation ### 4.1 Deployment Start the server with the command from [Section 3.1](#3-1-basic-configuration). ### 4.2 Basic Usage ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="google/diffusiongemma-26B-A4B-it", messages=[ {"role": "user", "content": "What are the key differences between TCP and UDP?"} ], max_tokens=1024 ) print(response.choices[0].message.content) ``` ### 4.3 Streaming Streaming emits one fully-denoised canvas per chunk. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="google/diffusiongemma-26B-A4B-it", messages=[ {"role": "user", "content": "Write a Python function to compute the Fibonacci sequence."} ], max_tokens=2048, stream=True ) for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` ## 5. Benchmark ### 5.1 Speed Benchmark Not benchmarked for speed. ### 5.2 Accuracy Benchmark Full test splits, every item scored (no failed-request exclusions). Text MCQ benchmarks use greedy generate-and-parse, MATH uses boxed-answer extraction plus sympy equivalence. MMLU, ARC-Challenge, and MATH-500 are the mean of two independent server launches.
Benchmark Score
GSM8K 95.4%
ARC-Challenge 91.6%
HumanEval 92.7% pass\@1
MMLU 76.2%
MMLU-Pro 73.7%
GSM-Symbolic 92.2%
MATH-500 72.1%
AIME-2026 10.0%
HMMT-Feb-2025 10.0%
GPQA-main 59.2%
Multimodal, full standard split per task (MMMU / MMMU-Pro / MMStar / AI2D as multiple-choice, MathVista testmini, DocVQA by ANLS, ChartQA by relaxed accuracy):
Multimodal benchmark Score
MMMU (val, MC) 64.9%
MMMU-Pro (standard 10-opt, MC) 57.3%
MathVista (testmini) 68.4%
DocVQA (val) 85.9%
ChartQA (test) 61.7%
AI2D (test) 78.7%
MMStar (val) 65.9%
# EmbeddingGemma Source: https://docs.sglang.io/cookbook/autoregressive/Google/EmbeddingGemma Serve Google's EmbeddingGemma text embedding model with SGLang. ## Overview [EmbeddingGemma](https://huggingface.co/google/embeddinggemma-300m) is Google's 300M-parameter text embedding model. SGLang detects its bidirectional Gemma 3 encoder, applies normalized mean pooling, and serves embeddings through the OpenAI-compatible `/v1/embeddings` endpoint. On NVIDIA CUDA, SGLang uses breakable CUDA graph (BCG) for its complete prefill by default. It also disables prefix caching and chunked prefill, which are incompatible with this bidirectional encoder. ## Prerequisites * NVIDIA CUDA GPU. * A Hugging Face account that has accepted the [EmbeddingGemma license](https://huggingface.co/google/embeddinggemma-300m). * A Hugging Face access token. Export it before starting the server so it can download the gated checkpoint: ```bash theme={null} export HF_TOKEN= ``` Install an SGLang build that includes EmbeddingGemma support: ```bash theme={null} pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' ``` ## Start the server The standard configuration detects EmbeddingGemma and enables embedding mode, BCG, and the checkpoint's BF16 dtype automatically: ```bash theme={null} sglang serve \ --model-path google/embeddinggemma-300m \ --host 0.0.0.0 ``` ### Hopper performance defaults On H100 and H200, SGLang automatically selects FA3 and captures BCG through 16,384 tokens, covering eight 2K embedding requests in one replay. No extra performance flags are required for this workload. To capture larger aggregate prefills, raise the BCG tier explicitly: ```bash theme={null} sglang serve \ --model-path google/embeddinggemma-300m \ --cuda-graph-max-bs-prefill 32768 \ --host 0.0.0.0 ``` EmbeddingGemma automatically enables batch tokenization for list-valued embedding requests, so do not add a separate tokenizer batching flag. ## Create embeddings Send one string or a batch of strings to the OpenAI-compatible endpoint: ```bash theme={null} curl http://127.0.0.1:30000/v1/embeddings \ -H 'Content-Type: application/json' \ -d '{ "model": "google/embeddinggemma-300m", "input": [ "A short guide to serving text embeddings.", "Vector search retrieves semantically similar documents." ], "encoding_format": "float" }' ``` See [OpenAI-compatible embedding APIs](/docs/basic_usage/openai_api_embeddings) for Python and OpenAI client examples. ## Deployment behavior EmbeddingGemma performs bidirectional attention over the complete input, so reusing a prefix KV cache or splitting the input into chunked prefills would produce incorrect attention states. SGLang applies the required settings automatically: * disables RadixAttention prefix caching; * disables chunked prefill; * disables the decode CUDA graph because this is an embedding-only model; * uses BCG for CUDA prefill; * uses the FlashAttention raw-K/V path when the prefill backend is FA3 or FA4 on supported Hopper and Blackwell CUDA GPUs. No prefill CUDA-graph override is required for this recipe. Keep BCG enabled to use the optimized EmbeddingGemma path. # EmbeddingGemma 2 Source: https://docs.sglang.io/cookbook/autoregressive/Google/EmbeddingGemma2 Serve Google's multimodal EmbeddingGemma 2 model (text, image, video, audio) with SGLang. ## Overview [EmbeddingGemma 2](https://huggingface.co/google/embeddinggemma-2) is Google's multimodal embedding model. It maps text (including code), images, video, audio, and interleaved combinations of them into one shared, L2-normalized 768-dimensional vector space. | Property | Value | | :- | :- | | Parameters | 740M total: 270M text backbone, 170M vision encoder, 300M audio encoder | | Architecture | 24-layer bidirectional Gemma 4 encoder, 5:1 local:global attention, local window radius 512 | | Context | 8,192 tokens shared by all modalities in a request | | Output | 768 dimensions; Matryoshka truncation to 512, 256, or 128 | | Pooling | Mean pooling over all tokens, then L2 normalization | SGLang detects the `EmbeddingGemma2Model` architecture and configures it automatically: * enables embedding mode, so `--is-embedding` is not needed; * selects the Triton attention backend, which implements the bidirectional sliding window; * disables RadixAttention prefix caching and chunked prefill, because bidirectional attention makes every token depend on the whole input; * caps the context length at 8,192 tokens. ## Prerequisites * An NVIDIA CUDA GPU. The model weighs about 1.5 GB in BF16, so any data-center GPU works. * A Transformers build that includes EmbeddingGemma 2 (`model_type: embedding_gemma2`). Older releases fail at startup with `Transformers does not recognize this architecture`. * An SGLang build that includes EmbeddingGemma 2 support: ```bash theme={null} pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' ``` Run the model in BF16 (the checkpoint default) or FP32. Do not use FP16: the model's activations exceed its range and produce NaN or degraded embeddings. ## Start the server ```bash theme={null} sglang serve \ --model-path google/embeddinggemma-2 \ --host 0.0.0.0 ``` ### Load only the encoders you need The vision and audio encoders load only when their modality is allowed. Set a modality's per-request limit to `0` to skip loading its encoder. Images and video share the vision encoder. | Serving | `--limit-mm-data-per-request` | Parameters loaded | | :- | :- | :- | | Text only | `'{"image": 0, "video": 0, "audio": 0}'` | 270M | | Text and images/video | `'{"audio": 0}'` | 440M | | Text and audio | `'{"image": 0, "video": 0}'` | 570M | | All modalities | (default) | 740M | ```bash theme={null} sglang serve \ --model-path google/embeddinggemma-2 \ --limit-mm-data-per-request '{"image": 0, "video": 0, "audio": 0}' \ --host 0.0.0.0 ``` A request for a disabled modality fails with HTTP 400, for example `Image count 1 exceeds limit 0 per request.` ### Enable Matryoshka dimensions To let clients request shorter vectors through the `dimensions` field, declare the supported sizes when starting the server: ```bash theme={null} sglang serve \ --model-path google/embeddinggemma-2 \ --json-model-override-args '{"matryoshka_dimensions": [128, 256, 512, 768]}' \ --host 0.0.0.0 ``` SGLang truncates the pooled vector and then re-normalizes it, as the model card requires. Other sizes are rejected with HTTP 400. Queries and documents must use the same dimension. ## Task prompts EmbeddingGemma 2 is trained with short task prefixes on text inputs. Prepend them yourself; images, video, and audio take no prefix. For documents without a title, use `title: none`. | Use case | Query | Document | | :- | :- | :- | | Retrieval | `task: search result \| query: {content}` | `title: {title} \| text: {content}` | | Question answering | `task: question answering \| query: {content}` | `title: {title} \| text: {content}` | | Fact checking | `task: fact checking \| query: {content}` | `title: {title} \| text: {content}` | | Code retrieval | `task: code retrieval \| query: {content}` | `title: {title} \| text: {content}` | | Classification | `task: classification \| query: {content}` | | | Clustering | `task: clustering \| query: {content}` | | | Semantic similarity | `task: sentence similarity \| query: {content}` | | ## Create embeddings ### Native `/encode` API `/encode` supports every modality, including several media items in one embedding. Text, one embedding per string: ```bash theme={null} curl http://127.0.0.1:30000/encode \ -H 'Content-Type: application/json' \ -d '{ "text": [ "task: search result | query: What causes the aurora borealis?", "title: none | text: The aurora is caused by solar wind particles colliding with the upper atmosphere." ] }' ``` An image, video, or audio clip on its own. SGLang inserts the placeholder for you: ```bash theme={null} curl http://127.0.0.1:30000/encode \ -H 'Content-Type: application/json' \ -d '{"image_data": "https://example.com/cat.jpg"}' ``` Use `video_data` or `audio_data` the same way. Audio should be mono at 16 kHz. Interleaved input, one embedding for the whole sequence. Mark each item's position with `<|image|>`, `<|video|>`, or `<|audio|>`, and pass the items in the same order: ```bash theme={null} curl http://127.0.0.1:30000/encode \ -H 'Content-Type: application/json' \ -d '{ "text": "Waterproof running shoes. <|image|> Breathable mesh upper. <|image|> Narration: <|audio|>", "image_data": ["https://example.com/shoe.jpg", "https://example.com/mesh.jpg"], "audio_data": "https://example.com/narration.wav" }' ``` #### How requests map to embeddings | Request shape | Result | | :- | :- | | `text` is a string, media is a list | 1 embedding containing all the media | | `text` is a list of N strings, media is a list of N entries | N embeddings, paired by position; an entry may be `null` or a list of several items | | `text` is a list, media is a single item (not a list) | the item is attached to every request, so every string needs its placeholder | | No `text`, media is a flat list `[a, b]` | 1 embedding of `a` and `b` together | | No `text`, media is a list of lists `[[a], [b]]` | 2 embeddings | Requests are rejected with HTTP 400 instead of being silently changed when: * the number of `<|image|>`, `<|video|>`, or `<|audio|>` placeholders differs from the number of items of that modality (text without any placeholder plus an image is also a mismatch); * a batched media list has a different length than the `text` list. ### OpenAI-compatible `/v1/embeddings` ```bash theme={null} curl http://127.0.0.1:30000/v1/embeddings \ -H 'Content-Type: application/json' \ -d '{ "model": "google/embeddinggemma-2", "input": [ {"text": "a product photo: <|image|>", "image": "https://example.com/shoe.jpg"}, {"image": "https://example.com/mesh.jpg"}, {"text": "task: search result | query: waterproof trail shoes"} ] }' ``` Each input item produces one embedding. In this API, an item carries at most one `image` or `video` and no audio. For several media items in one embedding, or for audio, use `/encode`. ## Input limits All modalities share the 8,192-token context. Requests that exceed it fail with HTTP 400 (`The input (N tokens) is longer than the model's context length (8192 tokens).`). | Modality | Cost | Max, single modality | | :- | :- | :- | | Text | 1 token per subword | 8,192 tokens | | Image | up to 280 tokens per image | about 29 images | | Video | up to 140 tokens per sampled frame | about 58 frames | | Audio | 25 tokens per second | about 327 seconds | Video is sampled at 1 frame per second and capped at 32 frames by default, spread uniformly over longer clips, so a video of any length fits by default. A 2-minute 720p clip, for example, uses 3,906 tokens. Only the sampled frames are decoded. To change the sampling rate or frame cap for every request, use `--mm-process-config`: ```bash theme={null} sglang serve \ --model-path google/embeddinggemma-2 \ --mm-process-config '{"video": {"fps": 2, "max_frames": 48}}' \ --host 0.0.0.0 ``` The number of frames that fit depends on resolution. At 720p, 64 frames fit; 70 frames exceed the context. # Gemma 4 Source: https://docs.sglang.io/cookbook/autoregressive/Google/Gemma4 ## 1. Model Introduction Gemma 4 is Google's next-generation family of open models, building on the Gemma 3 architecture with improved performance, MoE variants, and multimodal support for text, vision, and audio. **Key Features:** * **Hybrid Attention**: Combines sliding window and full attention layers for efficient long-context processing * **Multimodal**: Supports text, image, and audio inputs via dedicated vision and audio encoders * **MoE Variant**: The 26B-A4B model uses a Mixture-of-Experts architecture for efficient inference * **Per-Layer Embeddings (PLE)**: Layer-specific token embeddings for enhanced representations * **Reasoning**: Built-in thinking mode with `gemma4` reasoning parser * **Tool Calling**: Function call support with streaming via `gemma4` tool call parser * **Fused Operations**: Triton-optimized RMSNorm + residual + scalar kernels **Available Models:**
Model Architecture Parameters
[google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it) Dense \~2B
[google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) Dense \~4B
[google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it) Dense 12B
[google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) Dense 31B
[google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) MoE 26B total / 4B active
## 2. SGLang Installation Gemma 4 (including the encoder-free unified 12B, [sgl-project/sglang#27167](https://github.com/sgl-project/sglang/pull/27167)) is supported on SGLang main. Install it together with the matching transformers commit: ```bash Command theme={null} # Install SGLang from main pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' # Install transformers with Gemma 4 support (encoder-free unified family included) pip install 'git+https://github.com/huggingface/transformers.git@1423d22f7a3b62e8c70ad67b58ec25cd9b675897' ``` ### Docker `lmsysorg/sglang:latest` (CUDA 13.0, multi-arch `amd64` + `arm64`) runs on both Hopper (H200) and Blackwell (B200 / GB200 / GB300): ```bash Command theme={null} docker run --gpus all --ipc=host --shm-size 32g \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -p 30000:30000 \ lmsysorg/sglang:latest \ sglang serve --model-path google/gemma-4-12B-it \ --reasoning-parser gemma4 --tool-call-parser gemma4 \ --host 0.0.0.0 --port 30000 ``` For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration The Gemma 4 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)). The recommended launch configurations vary by hardware and model size. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model variant. ### 3.2 Configuration Tips * SGLang automatically selects the Triton attention backend for Gemma 4 models (required for bidirectional image-token attention during prefill). * **Attention backend on Blackwell (B200/sm100)**: SGLang defaults to the `trtllm_mha` backend on sm100, which is fastest for text but applies *causal* attention to image tokens. For multimodal (image) workloads on B200, pass `--attention-backend triton` to restore bidirectional image-token attention and full vision quality. Text-only and audio workloads are unaffected by the default. * **Gemma 4 26B-A4B on B200**: Use `--mem-fraction-static 0.75` to leave workspace headroom for the Triton MoE path. * For the 26B-A4B MoE model, consider `--tp 2` for high-throughput workloads. * **Speculative Decoding (MTP)**: Each Gemma 4 variant ships with a paired `*-assistant` draft model that enables NEXTN multi-token prediction. Enable it via the selector above, or pass `--speculative-algorithm NEXTN --speculative-draft-model-path google/gemma-4--it-assistant --speculative-num-steps 5 --speculative-num-draft-tokens 6 --speculative-eagle-topk 1`. MTP can significantly reduce latency for interactive use cases. The 26B-A4B MoE model requires `--tp 2` when MTP is enabled. * **QAT checkpoints**: Toggle **Checkpoint → QAT** in the selector to target the `qat-q4_0-unquantized` releases. These keep bf16 weights, so memory and TP requirements match the standard checkpoints, and each has a matching `*-qat-q4_0-unquantized-assistant` draft model for MTP. * Hardware requirements:
Model Hardware TP
gemma-4-E2B-it 1x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X 1
gemma-4-E4B-it 1x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X 1
gemma-4-12B-it 1x H200 / 1x B200 / 1x B300 1
gemma-4-31B-it 2x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X 2 (H200) / 1 (B200/B300/AMD)
gemma-4-26B-A4B-it 1x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X 1
### 3.3 AMD GPU Deployment (MI300X / MI325X / MI355X) SGLang automatically selects the correct attention backend on AMD GPUs. For the small E-models (`gemma-4-E2B-it`, `gemma-4-E4B-it`), disable AITER on AMD GPUs and use the same command line otherwise: ```bash Command theme={null} SGLANG_USE_AITER=0 sglang serve --model-path google/gemma-4-E4B-it \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --host 0.0.0.0 --port 30000 ``` For `gemma-4-31B-it` and `gemma-4-26B-A4B-it`, the same commands above work on MI300X, MI325X, and MI355X without additional command-line changes. > **Status**: AMD benchmarks are available in [Section 5.1](#5-1-speed-benchmark). ## 4. Model Invocation Deploy gemma-4-26B-A4B-it (MoE) with all features enabled: ```bash Command theme={null} sglang serve --model-path google/gemma-4-26B-A4B-it \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --host 0.0.0.0 --port 30000 ``` #### Speculative Decoding (MTP) Server Commands Each Gemma 4 variant ships with a paired `*-assistant` draft model for NEXTN multi-token prediction. Use the commands below to enable MTP for the corresponding target model. These match the configuration generated when you toggle **Speculative Decoding (MTP) → Enabled** in the [interactive selector](#3-1-basic-configuration). ```bash Command theme={null} # Gemma 4 E2B + MTP sglang serve \ --model-path google/gemma-4-E2B-it \ --speculative-algorithm NEXTN \ --speculative-draft-model-path google/gemma-4-E2B-it-assistant \ --speculative-num-steps 5 \ --speculative-num-draft-tokens 6 \ --speculative-eagle-topk 1 \ --mem-fraction-static 0.85 ``` ```bash Command theme={null} # Gemma 4 E4B + MTP sglang serve \ --model-path google/gemma-4-E4B-it \ --speculative-algorithm NEXTN \ --speculative-draft-model-path google/gemma-4-E4B-it-assistant \ --speculative-num-steps 5 \ --speculative-num-draft-tokens 6 \ --speculative-eagle-topk 1 \ --mem-fraction-static 0.85 ``` ```bash Command theme={null} # Gemma 4 12B + MTP (~35% faster single-stream decode on H200) sglang serve \ --model-path google/gemma-4-12B-it \ --speculative-algorithm NEXTN \ --speculative-draft-model-path google/gemma-4-12B-it-assistant \ --speculative-num-steps 5 \ --speculative-num-draft-tokens 6 \ --speculative-eagle-topk 1 \ --mem-fraction-static 0.85 ``` ```bash Command theme={null} # Gemma 4 31B + MTP sglang serve \ --model-path google/gemma-4-31B-it \ --tp-size 2 \ --speculative-algorithm NEXTN \ --speculative-draft-model-path google/gemma-4-31B-it-assistant \ --speculative-num-steps 5 \ --speculative-num-draft-tokens 6 \ --speculative-eagle-topk 1 \ --mem-fraction-static 0.85 ``` ```bash Command theme={null} # Gemma 4 26B-A4B + MTP sglang serve \ --model-path google/gemma-4-26B-A4B-it \ --tp-size 2 \ --speculative-algorithm NEXTN \ --speculative-draft-model-path google/gemma-4-26B-A4B-it-assistant \ --speculative-num-steps 5 \ --speculative-num-draft-tokens 6 \ --speculative-eagle-topk 1 \ --mem-fraction-static 0.85 ``` ### 4.1 Basic Usage ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="google/gemma-4-26B-A4B-it", messages=[ {"role": "user", "content": "What are the key differences between TCP and UDP?"} ], max_tokens=1024 ) print(response.choices[0].message.content) ```
Example Output ```text Output theme={null} The fundamental difference between **TCP (Transmission Control Protocol)** and **UDP (User Datagram Protocol)** lies in how they prioritize data integrity versus speed. ### 1. Connection Type * **TCP (Connection-Oriented):** Before any data is sent, TCP performs a "three-way handshake." The sender and receiver exchange signals to establish a formal connection. * **UDP (Connectionless):** UDP does not establish a connection. It simply starts blasting packets to the destination IP address without checking if the receiver is ready. ### 2. Reliability and Error Checking * **TCP (Reliable):** If a packet is lost or arrives corrupted, TCP detects the error and retransmits the missing data. * **UDP (Unreliable):** If a packet is lost or corrupted, it is simply discarded. There is no mechanism to ask for a retransmission. ### 3. Ordering of Data * **TCP (Ordered):** Segments are assigned sequence numbers and reassembled in the correct order. * **UDP (Unordered):** Packets may arrive in a different order than sent. ### 4. Speed and Overhead * **TCP (Slower):** Managing connections, tracking, and retransmissions adds significant overhead. * **UDP (Faster):** No handshake, no tracking — extremely fast and ideal for real-time needs. | Feature | TCP | UDP | | :--- | :--- | :--- | | **Connection** | Connection-oriented | Connectionless | | **Reliability** | Guaranteed delivery | Best-effort | | **Ordering** | Maintains strict order | No guaranteed order | | **Speed** | Slower (High overhead) | Faster (Low overhead) | ```
### 4.2 Vision Input Gemma 4 multimodal variants accept images alongside text: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="google/gemma-4-26B-A4B-it", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://farm4.staticflickr.com/3175/2653711032_804ff86d81_z.jpg" } }, { "type": "text", "text": "Describe this image in detail." } ] } ], max_tokens=1024 ) print(response.choices[0].message.content) ```
Example Output ```text Output theme={null} A vertical, full shot shows a girl and a boy standing in front of a giant teddy bear. The boy, who is on the left, is of South Asian descent, has short dark hair, and is smiling at the camera. He is wearing a navy blue sweatshirt with a white collar, blue jeans, and white, black, and red sneakers. The girl, on the right, is also of South Asian descent and has long, dark hair. She is smiling at the camera and is wearing a pink t-shirt, a white long-sleeve shirt underneath, blue jeans, and pink sneakers. The giant teddy bear is light brown and is standing behind the two children. The bear has large, dark eyes and a black nose. In the background, on the left, there is a large wooden basket filled with small teddy bears. To the left of the basket, an American flag is hanging on the wall. On the right side of the image, there is a green leafy plant. The floor is a dark purple carpet. The lighting is bright and even. ```
### 4.3 Reasoning (Thinking Mode) Gemma 4 supports hybrid reasoning. Thinking is **not enabled by default** — pass `chat_template_kwargs: {"enable_thinking": true}` via `extra_body` to activate it. The reasoning parser separates thinking and content, returning the thinking process via `reasoning_content` in the streaming response. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="google/gemma-4-26B-A4B-it", messages=[ {"role": "user", "content": "Solve step by step: If a train travels at 60 km/h for 2.5 hours, how far does it go?"} ], max_tokens=4096, stream=True, extra_body={"chat_template_kwargs": {"enable_thinking": True}} ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ```
Example Output ```text Output theme={null} =============== Thinking ================= * Input: Speed = 60 km/h, Time = 2.5 hours. * Goal: Find the distance traveled. * Distance = Speed × Time. * Step 1: Identify given values. Speed = 60 km/h, Time = 2.5 hours * Step 2: Formula. Distance = Speed × Time * Step 3: Calculation. 60 × 2.5 Mental math: 60 × 2 = 120; 60 × 0.5 = 30; 120 + 30 = 150. * Step 4: Final Result. 150 km. =============== Content ================= To find the distance traveled, you can follow these steps: ### 1. Identify the given information: * **Speed:** 60 km/h * **Time:** 2.5 hours ### 2. Use the distance formula: Distance = Speed × Time ### 3. Substitute the values: Distance = 60 km/h × 2.5 hours ### 4. Perform the calculation: * 60 × 2 = 120 * 60 × 0.5 = 30 * 120 + 30 = 150 **Final Answer: The train travels 150 km.** ```
### 4.4 Tool Calling Gemma 4 supports function calling with the `gemma4` tool call parser. Enable it during deployment with `--tool-call-parser gemma4`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] response = client.chat.completions.create( model="google/gemma-4-26B-A4B-it", messages=[ {"role": "user", "content": "What's the weather in Tokyo?"} ], tools=tools, stream=True ) thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if hasattr(delta, 'tool_calls') and delta.tool_calls: if has_thinking and thinking_started: print("\n=============== Tool Calls ================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") if delta.content: print(delta.content, end="", flush=True) print() ```
Example Output ```text Output theme={null} =============== Tool Calls ================ Tool Call: get_weather Arguments: {"location": "Tokyo"} ```
### 4.5 Audio Input The audio-capable Gemma 4 variants (`gemma-4-E2B-it`, `gemma-4-E4B-it`, `gemma-4-12B-it`) accept raw audio alongside text. Pass the waveform as a base64 `audio_url` data URI (16 kHz mono WAV works well), or as OpenAI's `input_audio` part with the base64 bytes in `data` and a `format` of `wav`: ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("sample.wav", "rb") as f: audio_b64 = base64.b64encode(f.read()).decode() response = client.chat.completions.create( model="google/gemma-4-12B-it", messages=[ { "role": "user", "content": [ {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}}, {"type": "text", "text": "Transcribe the speech in this audio exactly."}, ], } ], max_tokens=256, temperature=0, ) print(response.choices[0].message.content) ```
Example Output ```text Output theme={null} Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel. ```
For best ASR quality, use the recommended transcription prompt structure: ```text Prompt theme={null} Transcribe the following speech segment in {LANGUAGE} into {LANGUAGE} text. Follow these specific instructions for formatting the answer: * Only output the transcription, with no newlines. * When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three. ``` For speech translation (AST), ask for the transcription in the source language first, then the translation: *"Transcribe the following speech segment in , then translate it into . ..."* ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: H200 * SGLang Version: gemma4 branch #### gemma-4-E2B-it (1x H200, TP=1) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-E2B-it ``` **Latency Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 10 --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 17.44 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.57 Output token throughput (tok/s): 242.03 Total token throughput (tok/s): 591.94 Mean TTFT (ms): 50.19 Median TTFT (ms): 54.22 Mean TPOT (ms): 3.99 Median ITL (ms): 4.05 ================================================== ``` **Latency Benchmark (Image)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang-oai-chat \ --host 0.0.0.0 --port 30000 \ --dataset-name image --image-count 2 --image-resolution 720p \ --random-input-len 128 --random-output-len 1024 \ --num-prompts 10 --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 18.05 Total input tokens: 6097 Total input vision tokens: 5340 Total generated tokens: 4220 Request throughput (req/s): 0.55 Output token throughput (tok/s): 233.84 Total token throughput (tok/s): 571.69 Mean TTFT (ms): 109.59 Median TTFT (ms): 112.62 Mean TPOT (ms): 4.01 Median ITL (ms): 4.04 ================================================== ``` **Throughput Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 1000 --max-concurrency 100 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 51.73 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 19.33 Output token throughput (tok/s): 9876.36 Peak output token throughput (tok/s): 13863.00 Total token throughput (tok/s): 19791.14 Mean TTFT (ms): 86.57 Mean TPOT (ms): 9.56 Median ITL (ms): 5.99 ================================================== ``` **Throughput Benchmark (Image)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang-oai-chat \ --host 0.0.0.0 --port 30000 \ --dataset-name image --image-count 2 --image-resolution 720p \ --random-input-len 128 --random-output-len 1024 \ --num-prompts 1000 --max-concurrency 100 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 89.07 Total input tokens: 617799 Total input vision tokens: 534000 Total generated tokens: 510855 Request throughput (req/s): 11.23 Output token throughput (tok/s): 5735.75 Peak output token throughput (tok/s): 12823.00 Total token throughput (tok/s): 12672.23 Mean TTFT (ms): 636.46 Mean TPOT (ms): 16.34 Median ITL (ms): 5.68 ================================================== ``` #### gemma-4-E4B-it (1x H200, TP=1) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-E4B-it ``` **Latency Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 24.49 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.41 Output token throughput (tok/s): 172.32 Total token throughput (tok/s): 421.45 Mean TTFT (ms): 52.76 Median TTFT (ms): 53.66 Mean TPOT (ms): 5.64 Median ITL (ms): 5.74 ================================================== ``` **Latency Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 25.04 Total input tokens: 6124 Total input vision tokens: 5340 Total generated tokens: 4220 Request throughput (req/s): 0.40 Output token throughput (tok/s): 168.54 Total token throughput (tok/s): 413.13 Mean TTFT (ms): 110.15 Median TTFT (ms): 108.24 Mean TPOT (ms): 5.66 Median ITL (ms): 5.73 ================================================== ``` **Throughput Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 72.95 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 13.71 Output token throughput (tok/s): 7002.68 Peak output token throughput (tok/s): 9878.00 Total token throughput (tok/s): 14032.60 Mean TTFT (ms): 166.33 Mean TPOT (ms): 13.36 Median ITL (ms): 8.88 ================================================== ``` **Throughput Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 108.99 Total input tokens: 616952 Total input vision tokens: 534000 Total generated tokens: 510855 Request throughput (req/s): 9.18 Output token throughput (tok/s): 4687.38 Peak output token throughput (tok/s): 9277.00 Total token throughput (tok/s): 10348.25 Mean TTFT (ms): 626.17 Mean TPOT (ms): 20.00 Median ITL (ms): 8.64 ================================================== ``` #### gemma-4-31B-it (2x H200, TP=2) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-31B-it --tp 2 ``` **Latency Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 53.05 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.19 Output token throughput (tok/s): 79.55 Total token throughput (tok/s): 194.55 Mean TTFT (ms): 72.77 Median TTFT (ms): 75.05 Mean TPOT (ms): 12.32 Median ITL (ms): 12.53 ================================================== ``` **Latency Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 53.78 Total input tokens: 6162 Total input vision tokens: 5340 Total generated tokens: 4220 Request throughput (req/s): 0.19 Output token throughput (tok/s): 78.46 Total token throughput (tok/s): 193.03 Mean TTFT (ms): 143.35 Median TTFT (ms): 146.85 Mean TPOT (ms): 12.37 Median ITL (ms): 12.48 ================================================== ``` **Throughput Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 182.00 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 5.49 Output token throughput (tok/s): 2806.82 Peak output token throughput (tok/s): 3798.00 Total token throughput (tok/s): 5624.56 Mean TTFT (ms): 324.67 Mean TPOT (ms): 33.95 Median ITL (ms): 25.44 ================================================== ``` **Throughput Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 236.46 Total input tokens: 621630 Total input vision tokens: 534000 Total generated tokens: 510855 Request throughput (req/s): 4.23 Output token throughput (tok/s): 2160.42 Peak output token throughput (tok/s): 3745.00 Total token throughput (tok/s): 4789.30 Mean TTFT (ms): 952.02 Mean TPOT (ms): 44.17 Median ITL (ms): 26.81 ================================================== ``` #### gemma-4-26B-A4B-it (MoE, 1x H200, TP=1) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-26B-A4B-it ``` > **Tip**: Consider `--tp 2` for high-throughput workloads. **Latency Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 25.00 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.40 Output token throughput (tok/s): 168.81 Total token throughput (tok/s): 412.85 Mean TTFT (ms): 103.74 Median TTFT (ms): 46.57 Mean TPOT (ms): 5.60 Median ITL (ms): 5.78 ================================================== ``` **Latency Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 25.31 Total input tokens: 6164 Total input vision tokens: 5340 Total generated tokens: 4220 Request throughput (req/s): 0.40 Output token throughput (tok/s): 166.70 Total token throughput (tok/s): 410.20 Mean TTFT (ms): 129.22 Median TTFT (ms): 132.54 Mean TPOT (ms): 5.68 Median ITL (ms): 5.75 ================================================== ``` **Throughput Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 138.98 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 7.20 Output token throughput (tok/s): 3675.81 Peak output token throughput (tok/s): 4799.00 Total token throughput (tok/s): 7365.91 Mean TTFT (ms): 153.77 Mean TPOT (ms): 25.95 Median ITL (ms): 20.23 ================================================== ``` **Throughput Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 186.38 Total input tokens: 621146 Total input vision tokens: 534000 Total generated tokens: 510855 Request throughput (req/s): 5.37 Output token throughput (tok/s): 2740.86 Peak output token throughput (tok/s): 4962.00 Total token throughput (tok/s): 6073.47 Mean TTFT (ms): 854.71 Mean TPOT (ms): 34.64 Median ITL (ms): 19.08 ================================================== ``` #### gemma-4-31B-it (1x MI300X, TP=1) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-31B-it ``` > **Note**: The 31B dense model fits on a single MI300X (192 GB VRAM) at TP=1, unlike H200 (141 GB) which requires TP=2. **Latency Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 10 --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 103.55 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.10 Output token throughput (tok/s): 40.75 Total token throughput (tok/s): 99.67 Mean TTFT (ms): 152.35 Median TTFT (ms): 169.66 Mean TPOT (ms): 24.13 Median ITL (ms): 24.23 ================================================== ``` **Throughput Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 1000 --max-concurrency 100 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 441.59 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 2.26 Output token throughput (tok/s): 1156.85 Peak output token throughput (tok/s): 1759.00 Total token throughput (tok/s): 2318.19 Mean TTFT (ms): 819.22 Mean TPOT (ms): 82.51 Median ITL (ms): 63.45 ================================================== ``` #### gemma-4-26B-A4B-it (MoE, 1x MI300X, TP=1) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-26B-A4B-it ``` **Latency Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 10 --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 43.73 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.23 Output token throughput (tok/s): 96.49 Total token throughput (tok/s): 236.00 Mean TTFT (ms): 185.58 Median TTFT (ms): 90.18 Mean TPOT (ms): 9.78 Median ITL (ms): 9.57 ================================================== ``` **Throughput Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 1000 --max-concurrency 100 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 219.43 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 4.56 Output token throughput (tok/s): 2328.05 Peak output token throughput (tok/s): 3500.00 Total token throughput (tok/s): 4665.16 Mean TTFT (ms): 168.44 Mean TPOT (ms): 41.23 Median ITL (ms): 29.31 ================================================== ``` #### gemma-4-12B-it (1x H200, TP=1) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-12B-it ``` **Latency Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 10 --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 38.66 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.26 Output token throughput (tok/s): 109.15 Total token throughput (tok/s): 266.94 Mean TTFT (ms): 33.08 Median TTFT (ms): 33.71 Mean TPOT (ms): 9.02 Median ITL (ms): 9.19 ================================================== ``` **Latency Benchmark (Image)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang-oai-chat \ --host 0.0.0.0 --port 30000 \ --dataset-name image --image-count 2 --image-resolution 720p \ --random-input-len 128 --random-output-len 1024 \ --num-prompts 10 --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 39.36 Total input vision tokens: 5320 Total generated tokens: 4220 Request throughput (req/s): 0.25 Output token throughput (tok/s): 107.23 Total token throughput (tok/s): 263.62 Mean TTFT (ms): 94.98 Median TTFT (ms): 97.33 Mean TPOT (ms): 9.08 Median ITL (ms): 9.17 ================================================== ``` **Throughput Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 1000 --max-concurrency 100 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 130.44 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 7.67 Output token throughput (tok/s): 3916.46 Total token throughput (tok/s): 7848.15 Mean TTFT (ms): 207.49 Median TTFT (ms): 76.95 Mean TPOT (ms): 24.38 Median ITL (ms): 17.89 ================================================== ``` **Throughput Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 147.57 Total input tokens: 619609 Total input vision tokens: 532000 Total generated tokens: 510855 Request throughput (req/s): 6.78 Output token throughput (tok/s): 3461.79 Total token throughput (tok/s): 7660.54 Mean TTFT (ms): 438.40 Median TTFT (ms): 129.83 Mean TPOT (ms): 27.12 Median ITL (ms): 19.16 ================================================== ``` #### gemma-4-12B-it (1x B200, TP=1) Server Launch Command: ```bash Command theme={null} # Text/audio: the sm100 default (trtllm_mha) is fastest. # For image workloads add --attention-backend triton (bidirectional image attention). sglang serve --model-path google/gemma-4-12B-it --attention-backend triton ``` **Latency Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 30.46 Output token throughput (tok/s): 138.55 Total token throughput (tok/s): 338.85 Mean TTFT (ms): 28.14 Median TTFT (ms): 29.74 Mean TPOT (ms): 7.08 Median ITL (ms): 7.26 ================================================== ``` **Latency Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 31.43 Total input vision tokens: 5320 Total generated tokens: 4220 Request throughput (req/s): 0.32 Output token throughput (tok/s): 134.26 Total token throughput (tok/s): 329.57 Mean TTFT (ms): 115.51 Median TTFT (ms): 74.27 Mean TPOT (ms): 7.14 Median ITL (ms): 7.24 ================================================== ``` **Throughput Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 92.94 Request throughput (req/s): 10.76 Output token throughput (tok/s): 5496.55 Total token throughput (tok/s): 11014.49 Mean TTFT (ms): 120.89 Median TTFT (ms): 45.00 Mean TPOT (ms): 17.23 Median ITL (ms): 14.30 ================================================== ``` **Throughput Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Max request concurrency: 100 Successful requests: 998 Benchmark duration (s): 107.82 Total input tokens: 617971 Total input vision tokens: 530936 Total generated tokens: 508951 Request throughput (req/s): 9.26 Output token throughput (tok/s): 4720.29 Total token throughput (tok/s): 10451.68 Mean TTFT (ms): 425.89 Median TTFT (ms): 109.57 Mean TPOT (ms): 19.45 Median ITL (ms): 15.11 ================================================== ``` > **Performance tuning**: On B200, raising `--scheduler-recv-interval` to 16 lifted text throughput from 5497 to 5673 tok/s output (≈ +3%) at concurrency 100 with no accuracy change, by reducing the scheduler's per-step Python overhead. It is a safe, low-risk knob for high-concurrency serving. ### 5.2 Accuracy Benchmark **Test Environment:** * Hardware: H200 * SGLang Version: gemma4 branch #### MMLU
Model Humanities Social Sciences STEM Other Overall
gemma-4-E2B-it 0.621 0.739 0.830 0.736 **0.720**
gemma-4-E4B-it 0.703 0.862 0.902 0.825 **0.810**
gemma-4-12B-it 0.784 0.888 0.946 0.861 **0.859**
gemma-4-31B-it 0.878 0.921 0.884 0.911 **0.896**
gemma-4-26B-A4B-it 0.853 0.906 0.938 0.886 **0.891**
#### GSM8K
Model Accuracy Invalid Latency (s) Output Throughput (tok/s)
gemma-4-E2B-it 0.170 0.000 3.990 8041.739
gemma-4-E4B-it 0.745 0.000 4.174 4672.030
gemma-4-12B-it 0.431 0.052 55.105 6580.229
gemma-4-31B-it 0.805 0.005 16.148 1559.914
gemma-4-26B-A4B-it 0.450 0.010 13.001 4089.457
> **Note**: These GSM8K numbers use the raw few-shot completion harness (now removed). `gemma-4-12B-it` is reasoning-oriented and is under-elicited by raw few-shot prompting; with the chat template it scores **0.950** on the same 1319 GSM8K test questions (also measured before the sgl-eval migration). #### gemma-4-12B-it with sgl-eval `gemma-4-12B-it` is reasoning-oriented and answers verbosely (step-by-step) rather than emitting a terse final line. Strict last-line `Answer: $LETTER` extraction (as in `sglang.test.run_eval`) therefore undercounts its correct answers. [sgl-eval](https://github.com/sgl-project/sgl-eval) — sgl-project's evaluation CLI, which uses robust answer extraction — gives a faithful score on the served model: | Benchmark | Examples | Accuracy | | - | - | - | | MMLU | 2000 | **0.878** | | GSM8K | 1319 | **0.960** | Reproduce against a running server (`--base-url` points at your endpoint): ```bash Command theme={null} pip install sgl-eval # Sanity-check the endpoint sgl-eval ping --base-url http://localhost:30000/v1 # Run the benchmarks (greedy, single-shot) sgl-eval run gsm8k --base-url http://localhost:30000/v1 sgl-eval run mmlu --base-url http://localhost:30000/v1 --num-examples 2000 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. #### MMMU
Model Overall
gemma-4-E2B-it **0.307**
gemma-4-E4B-it **0.396**
gemma-4-12B-it **0.683**
gemma-4-31B-it **0.589**
gemma-4-26B-A4B-it **0.549**
MMMU detailed scores (per domain) **gemma-4-E2B-it** ```json Config theme={null} {"Overall-Art and Design": {"num": 120, "acc": 0.45}, "Art": {"num": 30, "acc": 0.5}, "Art_Theory": {"num": 30, "acc": 0.467}, "Design": {"num": 30, "acc": 0.5}, "Music": {"num": 30, "acc": 0.333}, "Overall-Business": {"num": 150, "acc": 0.26}, "Accounting": {"num": 30, "acc": 0.367}, "Economics": {"num": 30, "acc": 0.233}, "Finance": {"num": 30, "acc": 0.2}, "Manage": {"num": 30, "acc": 0.233}, "Marketing": {"num": 30, "acc": 0.267}, "Overall-Science": {"num": 150, "acc": 0.273}, "Biology": {"num": 30, "acc": 0.233}, "Chemistry": {"num": 30, "acc": 0.267}, "Geography": {"num": 30, "acc": 0.367}, "Math": {"num": 30, "acc": 0.233}, "Physics": {"num": 30, "acc": 0.267}, "Overall-Health and Medicine": {"num": 150, "acc": 0.273}, "Basic_Medical_Science": {"num": 30, "acc": 0.5}, "Clinical_Medicine": {"num": 30, "acc": 0.233}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.233}, "Pharmacy": {"num": 30, "acc": 0.3}, "Public_Health": {"num": 30, "acc": 0.1}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.4}, "History": {"num": 30, "acc": 0.4}, "Literature": {"num": 30, "acc": 0.567}, "Sociology": {"num": 30, "acc": 0.333}, "Psychology": {"num": 30, "acc": 0.3}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.252}, "Agriculture": {"num": 30, "acc": 0.333}, "Architecture_and_Engineering": {"num": 30, "acc": 0.267}, "Computer_Science": {"num": 30, "acc": 0.233}, "Electronics": {"num": 30, "acc": 0.1}, "Energy_and_Power": {"num": 30, "acc": 0.3}, "Materials": {"num": 30, "acc": 0.2}, "Mechanical_Engineering": {"num": 30, "acc": 0.333}, "Overall": {"num": 900, "acc": 0.307}} ``` **gemma-4-E4B-it** ```json Config theme={null} {"Overall-Art and Design": {"num": 120, "acc": 0.458}, "Art": {"num": 30, "acc": 0.433}, "Art_Theory": {"num": 30, "acc": 0.567}, "Design": {"num": 30, "acc": 0.667}, "Music": {"num": 30, "acc": 0.167}, "Overall-Business": {"num": 150, "acc": 0.287}, "Accounting": {"num": 30, "acc": 0.233}, "Economics": {"num": 30, "acc": 0.467}, "Finance": {"num": 30, "acc": 0.133}, "Manage": {"num": 30, "acc": 0.3}, "Marketing": {"num": 30, "acc": 0.3}, "Overall-Science": {"num": 150, "acc": 0.28}, "Biology": {"num": 30, "acc": 0.333}, "Chemistry": {"num": 30, "acc": 0.133}, "Geography": {"num": 30, "acc": 0.4}, "Math": {"num": 30, "acc": 0.2}, "Physics": {"num": 30, "acc": 0.333}, "Overall-Health and Medicine": {"num": 150, "acc": 0.427}, "Basic_Medical_Science": {"num": 30, "acc": 0.4}, "Clinical_Medicine": {"num": 30, "acc": 0.533}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.4}, "Pharmacy": {"num": 30, "acc": 0.4}, "Public_Health": {"num": 30, "acc": 0.4}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.7}, "History": {"num": 30, "acc": 0.633}, "Literature": {"num": 30, "acc": 0.867}, "Sociology": {"num": 30, "acc": 0.733}, "Psychology": {"num": 30, "acc": 0.567}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.324}, "Agriculture": {"num": 30, "acc": 0.533}, "Architecture_and_Engineering": {"num": 30, "acc": 0.3}, "Computer_Science": {"num": 30, "acc": 0.367}, "Electronics": {"num": 30, "acc": 0.133}, "Energy_and_Power": {"num": 30, "acc": 0.4}, "Materials": {"num": 30, "acc": 0.2}, "Mechanical_Engineering": {"num": 30, "acc": 0.333}, "Overall": {"num": 900, "acc": 0.396}} ``` **gemma-4-12B-it** ```json Config theme={null} {"Overall-Art and Design": {"num": 120, "acc": 0.667}, "Art": {"num": 30, "acc": 0.7}, "Art_Theory": {"num": 30, "acc": 0.867}, "Design": {"num": 30, "acc": 0.767}, "Music": {"num": 30, "acc": 0.333}, "Overall-Business": {"num": 150, "acc": 0.747}, "Accounting": {"num": 30, "acc": 0.767}, "Economics": {"num": 30, "acc": 0.767}, "Finance": {"num": 30, "acc": 0.633}, "Manage": {"num": 30, "acc": 0.7}, "Marketing": {"num": 30, "acc": 0.867}, "Overall-Science": {"num": 150, "acc": 0.647}, "Biology": {"num": 30, "acc": 0.6}, "Chemistry": {"num": 30, "acc": 0.633}, "Geography": {"num": 30, "acc": 0.567}, "Math": {"num": 30, "acc": 0.6}, "Physics": {"num": 30, "acc": 0.833}, "Overall-Health and Medicine": {"num": 150, "acc": 0.68}, "Basic_Medical_Science": {"num": 30, "acc": 0.667}, "Clinical_Medicine": {"num": 30, "acc": 0.633}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.267}, "Pharmacy": {"num": 30, "acc": 0.833}, "Public_Health": {"num": 30, "acc": 1.0}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.817}, "History": {"num": 30, "acc": 0.8}, "Literature": {"num": 30, "acc": 0.9}, "Sociology": {"num": 30, "acc": 0.8}, "Psychology": {"num": 30, "acc": 0.767}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.6}, "Agriculture": {"num": 30, "acc": 0.467}, "Architecture_and_Engineering": {"num": 30, "acc": 0.667}, "Computer_Science": {"num": 30, "acc": 0.733}, "Electronics": {"num": 30, "acc": 0.567}, "Energy_and_Power": {"num": 30, "acc": 0.667}, "Materials": {"num": 30, "acc": 0.567}, "Mechanical_Engineering": {"num": 30, "acc": 0.533}, "Overall": {"num": 900, "acc": 0.683}} ``` **gemma-4-31B-it** ```json Config theme={null} {"Overall-Art and Design": {"num": 120, "acc": 0.667}, "Art": {"num": 30, "acc": 0.667}, "Art_Theory": {"num": 30, "acc": 0.867}, "Design": {"num": 30, "acc": 0.8}, "Music": {"num": 30, "acc": 0.333}, "Overall-Business": {"num": 150, "acc": 0.573}, "Accounting": {"num": 30, "acc": 0.633}, "Economics": {"num": 30, "acc": 0.733}, "Finance": {"num": 30, "acc": 0.433}, "Manage": {"num": 30, "acc": 0.533}, "Marketing": {"num": 30, "acc": 0.533}, "Overall-Science": {"num": 150, "acc": 0.527}, "Biology": {"num": 30, "acc": 0.667}, "Chemistry": {"num": 30, "acc": 0.567}, "Geography": {"num": 30, "acc": 0.5}, "Math": {"num": 30, "acc": 0.267}, "Physics": {"num": 30, "acc": 0.633}, "Overall-Health and Medicine": {"num": 150, "acc": 0.673}, "Basic_Medical_Science": {"num": 30, "acc": 0.733}, "Clinical_Medicine": {"num": 30, "acc": 0.533}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.467}, "Pharmacy": {"num": 30, "acc": 0.8}, "Public_Health": {"num": 30, "acc": 0.833}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.825}, "History": {"num": 30, "acc": 0.833}, "Literature": {"num": 30, "acc": 0.867}, "Sociology": {"num": 30, "acc": 0.767}, "Psychology": {"num": 30, "acc": 0.833}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.405}, "Agriculture": {"num": 30, "acc": 0.667}, "Architecture_and_Engineering": {"num": 30, "acc": 0.2}, "Computer_Science": {"num": 30, "acc": 0.567}, "Electronics": {"num": 30, "acc": 0.333}, "Energy_and_Power": {"num": 30, "acc": 0.533}, "Materials": {"num": 30, "acc": 0.3}, "Mechanical_Engineering": {"num": 30, "acc": 0.233}, "Overall": {"num": 900, "acc": 0.589}} ``` **gemma-4-26B-A4B-it** ```json Config theme={null} {"Overall-Art and Design": {"num": 120, "acc": 0.717}, "Art": {"num": 30, "acc": 0.733}, "Art_Theory": {"num": 30, "acc": 0.833}, "Design": {"num": 30, "acc": 0.867}, "Music": {"num": 30, "acc": 0.433}, "Overall-Business": {"num": 150, "acc": 0.493}, "Accounting": {"num": 30, "acc": 0.533}, "Economics": {"num": 30, "acc": 0.533}, "Finance": {"num": 30, "acc": 0.333}, "Manage": {"num": 30, "acc": 0.5}, "Marketing": {"num": 30, "acc": 0.567}, "Overall-Science": {"num": 150, "acc": 0.473}, "Biology": {"num": 30, "acc": 0.633}, "Chemistry": {"num": 30, "acc": 0.367}, "Geography": {"num": 30, "acc": 0.533}, "Math": {"num": 30, "acc": 0.267}, "Physics": {"num": 30, "acc": 0.567}, "Overall-Health and Medicine": {"num": 150, "acc": 0.62}, "Basic_Medical_Science": {"num": 30, "acc": 0.767}, "Clinical_Medicine": {"num": 30, "acc": 0.533}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.433}, "Pharmacy": {"num": 30, "acc": 0.7}, "Public_Health": {"num": 30, "acc": 0.667}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.758}, "History": {"num": 30, "acc": 0.8}, "Literature": {"num": 30, "acc": 0.833}, "Sociology": {"num": 30, "acc": 0.733}, "Psychology": {"num": 30, "acc": 0.667}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.376}, "Agriculture": {"num": 30, "acc": 0.633}, "Architecture_and_Engineering": {"num": 30, "acc": 0.367}, "Computer_Science": {"num": 30, "acc": 0.533}, "Electronics": {"num": 30, "acc": 0.167}, "Energy_and_Power": {"num": 30, "acc": 0.367}, "Materials": {"num": 30, "acc": 0.367}, "Mechanical_Engineering": {"num": 30, "acc": 0.2}, "Overall": {"num": 900, "acc": 0.549}} ``` #### ASR
Model WER Avg Latency (s) Throughput (req/s)
gemma-4-E2B-it 23.86% 0.212 2.99
gemma-4-E4B-it 29.55% 0.366 2.46
gemma-4-12B-it Supported (see §4.5) — —
gemma-4-31B-it Not Supported — —
gemma-4-26B-A4B-it Not Supported — —
#### FLEUR (EN\_US)
Model WER Avg Latency (s) Throughput (req/s)
gemma-4-E2B-it 7.37% 0.8963s 16.25
gemma-4-E4B-it 6.08% 0.8707s 16.20
gemma-4-12B-it Supported (see §4.5) — —
gemma-4-31B-it Not Supported — —
gemma-4-26B-A4B-it Not Supported — —
### 5.3 Logits correctness validation **gemma-4-E2B-it** ```shell Command theme={null} $ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-E2B-it .... prefill logits (final): tensor([[-25.3063, -2.5718, -10.3674, ..., -25.3779, -25.5181, -25.2337]], device='cuda:0') .... $ python scripts/playground/reference_hf.py --model-path google/gemma-4-E2B-it .... prefill logits (final) tensor([-25.3281, -2.1367, -10.2266, ..., -25.4375, -25.5000, -25.2500], device='cuda:0', dtype=torch.float16) .... ``` **gemma-4-E4B-it** ```shell Command theme={null} $ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-E4B-it .... prefill logits (final): tensor([[-17.6478, 7.9901, -5.6505, ..., -17.5658, -17.6478, -17.7293]], device='cuda:0') .... $ python scripts/playground/reference_hf.py --model-path google/gemma-4-E4B-it .... prefill logits (final) tensor([-17.5625, 8.0469, -5.5742, ..., -17.4688, -17.5625, -17.6719], device='cuda:0', dtype=torch.float16) .... ``` **gemma-4-31B-it** ```shell Command theme={null} $ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-31B-it .... prefill logits (final): tensor([[-2.0748, 1.1245, -7.4356, ..., -2.1059, -2.1525, -2.2303]], device='cuda:0') .... $ python scripts/playground/reference_hf.py --model-path google/gemma-4-31B-it .... prefill logits (final) tensor([-2.1133, 1.2656, -7.4766, ..., -2.1523, -2.2012, -2.2695], device='cuda:0', dtype=torch.float16) .... ```
# Granite 4.2 Source: https://docs.sglang.io/cookbook/autoregressive/IBM/Granite-4.2 Deploy Granite 4.2 3B, 8B, and 30B dense models with SGLang on NVIDIA H200 and B200, including thinking modes and structured tool calling. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick a Granite 4.2 checkpoint to generate the launch command. The verified matrix covers BF16 serving on one NVIDIA H200 or B200 with tensor parallelism 1. The H200 speed results use `lmsysorg/sglang:dev` at SGLang commit `d59c1ddf7` and the B200 results at commit `d10a656ad8`; the launch recipes were verified end to end on both GPUs against the release checkpoints. Each speed point uses 80 fixed-length random requests at 8,192 input and 1,024 output tokens, 8 warmup requests, a flushed cache, greedy sampling, and ignore-EOS. ## Playground The Playground layers SGLang features on top of the verified recipe. Any override changes the badge to **Not Verified** until that exact configuration is tested end to end. ## 1. Model introduction **Granite 4.2** is IBM's dense decoder-only language model family with 3B, 8B, and 30B checkpoints. Each checkpoint uses BF16 weights, has a configured context length of 131,072 tokens, and supports default thinking, non-thinking, low-effort thinking, and structured tool calls through its chat template. The repositories declare the Apache-2.0 license.
Variant Total params Position in family
Granite 4.2 3B 3B Smallest checkpoint
Granite 4.2 8B 8B Mid-size checkpoint
Granite 4.2 30B 30B Largest checkpoint
**Recommended generation:** IBM recommends `temperature=1.0` and `top_p=0.95` for general chat, reasoning, and tool calling. The release checkpoints ship these values in `generation_config.json`; send them per request when you want to be explicit. **Resources:** [Granite 4.2 3B](https://huggingface.co/ibm-granite/granite-4.2-3b) · [Granite 4.2 8B](https://huggingface.co/ibm-granite/granite-4.2-8b) · [Granite 4.2 30B](https://huggingface.co/ibm-granite/granite-4.2-30b). ## 2. Configuration tips * **Thinking is enabled by default.** Set `chat_template_kwargs.enable_thinking` to `false` for a direct answer. Set `enable_thinking` and `low_effort` to `true` for a shorter reasoning trace. * **Give thinking enough tokens.** At `temperature=1.0` the default thinking mode can run past 1,000 tokens on multi-step problems. Use a `max_tokens` of at least 2,048 for thinking requests so the answer is not cut off. * **Reasoning parser.** Launch with `--reasoning-parser auto`, which resolves to `nemotron_3` for these checkpoints, so OpenAI-compatible responses separate the trace into `message.reasoning_content` and the answer into `message.content`. Without a parser flag the reasoning markup stays inline in `message.content`. * **Tool-call parser.** Launch with `--tool-call-parser auto`, which resolves to `qwen3_coder` for these checkpoints, so tool requests are returned through `message.tool_calls`. Without it, raw `` markup stays in `message.content`. * **Single-GPU sizing.** All three BF16 checkpoints loaded and completed chat requests with `--tp 1 --mem-fraction-static 0.8` on one H200 and on one B200. Increase TP only after validating the new topology. * **Image selection.** The validated path uses `lmsysorg/sglang:dev`. A stable image tested during validation had an incompatible dependency set before model loading, so use the recipe's image until a newer tagged release is confirmed. ## 3. Advanced usage The outputs below are verbatim captures from Granite 4.2 3B on the verified server. Sampling is stochastic, so a repeated request can produce different wording. ### 3.1 Thinking modes The `nemotron_3` reasoning parser keeps reasoning and final content in separate fields. Granite 4.2 accepts three chat-template modes: default thinking, non-thinking, and low-effort thinking. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") model = "ibm-granite/granite-4.2-3b" modes = { "thinking": {"enable_thinking": True}, "non-thinking": {"enable_thinking": False}, "low-effort": {"enable_thinking": True, "low_effort": True}, } for name, chat_template_kwargs in modes.items(): response = client.chat.completions.create( model=model, messages=[ {"role": "user", "content": "What is 17 * 23? Answer briefly."} ], extra_body={"chat_template_kwargs": chat_template_kwargs}, ) message = response.choices[0].message print(f"[{name}]") print("Reasoning:", getattr(message, "reasoning_content", None)) print("Answer:", message.content) ``` ```text Output theme={null} [thinking] Reasoning: Okay, the user asked "What is 17 * 23? Answer briefly." I need to calculate 17 multiplied by 23. Let me do the multiplication. 17 times 23. I can break it down: 17 * 20 = 340, and 17 * 3 = 51. Then add them: 340 + 51 = 391. Alternatively, 23 * 17: 23*10=230, 23*7=161, 230+161=391. Same result. So the answer is 391. The user wants a brief answer, so just state the number. Answer: 391 [non-thinking] Reasoning: None Answer: 391 [low-effort] Reasoning: Compute 17*23 = 17*20=340, plus 17*3=51 => 391. Answer: 391 ``` ### 3.2 Tool calling The `qwen3_coder` parser converts the model's tool markup into OpenAI-compatible structured calls. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city.", "parameters": { "type": "object", "properties": { "city": {"type": "string", "description": "The city name"}, }, "required": ["city"], }, }, }] response = client.chat.completions.create( model="ibm-granite/granite-4.2-3b", messages=[{"role": "user", "content": "What is the weather in Boston right now?"}], tools=tools, tool_choice="auto", ) choice = response.choices[0] message = choice.message print("Reasoning:", getattr(message, "reasoning_content", None)) print("Content:", message.content) for call in message.tool_calls or []: print("Tool:", call.function.name) print("Arguments:", call.function.arguments) print("Finish reason:", choice.finish_reason) ``` ```text Output theme={null} Reasoning: Okay, the user is asking for the weather in Boston right now. I need to use the available tool called get_weather. The tool requires the city parameter. Since the user specified Boston, I'll call get_weather with city set to Boston. Content: None Tool: get_weather Arguments: {"city": "Boston"} Finish reason: tool_calls ``` # K2 Horizon Source: https://docs.sglang.io/cookbook/autoregressive/IFM/K2-Horizon Deploy the IFM K2 Horizon family with SGLang — BF16 recipes for six dense and Mixture-of-Experts checkpoints on NVIDIA H200 GPUs. ## Deployment Install from a source checkout that includes the K2 Horizon runtime support in [PR #37654](https://github.com/sgl-project/sglang/pull/37654) before using the generated commands: ```bash Command theme={null} pip install --upgrade pip pip install -e "python" ``` For general source-install guidance, see [Install → Method 2: From source](/docs/get-started/install#method-2-from-source). Choose a model size to generate its validated single-node H200 launch command. The serving settings in all six base recipes completed two independent H200 benchmark launches at the pinned model revisions using the K2 Horizon runtime support in [PR #37654](https://github.com/sgl-project/sglang/pull/37654). Each speed and accuracy value in the benchmark cards is the arithmetic mean of two independent server launches. ## Playground The base recipes enable the `k2_horizon` reasoning parser. Use the Playground to configure the `k2_horizon` tool-call parser, TP and MoE expert-parallel overrides, NGRAM speculative decoding, PD disaggregation, and HiCache. A separate validation pass functionally exercised the tool-call parser on all six checkpoints; every other Playground override remains unverified. ## 1. Model introduction **K2 Horizon** is an IFM model family spanning six checkpoints from **0.9B** to **375B** parameters. The family includes dense models, Mixture-of-Experts models, and the MoVA-36B-A4B checkpoint's routed Mixture-of-Value-Attention path. This page provides native SGLang serving recipes for the family.
Checkpoint Architecture Context length H200 recipe
0.9B Dense 131,072 BF16 · TP1
3.7B Dense 524,288 BF16 · TP1
7B Dense 524,288 BF16 · TP1
32B Dense 524,288 BF16 · TP2
MoVA-36B-A4B MoE + MoVA 524,288 BF16 · TP2
375B-A23B MoE 524,288 BF16 · TP8
All six checkpoints use the Apache License 2.0. **Resources:** [K2 Horizon collection](https://huggingface.co/collections/IFM/k2-horizon) · [K2 Horizon 375B-A23B model card](https://huggingface.co/IFM/K2-Horizon-375B-A23B). ## 2. Configuration tips * **Hardware and precision:** The current matrix covers NVIDIA H200 GPUs and BF16 checkpoints only. * **Parallelism:** Use TP1 for 0.9B, 3.7B, and 7B; TP2 for 32B and MoVA-36B-A4B; and TP8 for 375B-A23B. Each recipe stays within one node. * **Attention:** The verified base recipes select FlashAttention-3 with `--attention-backend fa3`. * **Revisions:** The recipes pin revisions `6fbc54f7e4dc9d9cd671351f51498a1958dd2a01` (0.9B), `fe504ef19c7efa3120a9352bcfa180aa133946de` (3.7B), `85d46bbaf6ecd844a8ef61991f6e492d4faf4179` (7B), `eff3d24a3a40360c6fd1a1dd5723ce6329d93484` (32B), `e5c131d4d0ae196f5041284a0f9262d713083901` (MoVA-36B-A4B), and `70d7c58c1f8a2fc523c861c2cf7d2d339d7a8a0c` (375B-A23B). * **Parsers:** The base recipes enable reasoning with `--reasoning-parser k2_horizon`. Enable `--tool-call-parser k2_horizon` from the **Parsers** card in the [Playground](#playground) when you need tool calling. * **MoVA-36B-A4B router provenance:** The MoVA-36B-A4B checkpoint's source xLLM router used two GEMM partitions. `--json-model-override-args '{"xllm_source_router_gemm_partitions":2}'` preserves its BF16 partial-GEMM rounding and FP32 reduction order; SGLang does not infer this source topology from runtime TP. * **375B-A23B loading:** The 375B-A23B recipe disables multithreaded weight loading with `--model-loader-extra-config '{"enable_multithread_load":false}'`. * **Playground features:** TP, EP, NGRAM, PD disaggregation, and HiCache are optional, unverified command overrides. The tool-call parser was functionally exercised separately. * **Deliberate omissions:** Context parallelism is unsupported or unvalidated for xLLM. DP-attention and alternate MoE backend selectors are not exposed until they are validated. * **Validation:** The serving configuration for all six recipes completed native speed and full GSM8K evaluation on H200 using the K2 Horizon runtime support in [PR #37654](https://github.com/sgl-project/sglang/pull/37654). * **Benchmark provenance:** The results below were collected at revisions `9b9ec1f7e17f62ed218df542687a144116219d84` (0.9B), `c177771836a4c460743c00002c22483f6f18d1eb` (3.7B), `69ada542b68fe13d767479db2ab9421baff88681` (7B), `e1fd0277713e4eefcd3416348fd6fedacf7f2392` (32B), `16d20c739c687c08423422d1a2fbba6c529014cd` (MoVA-36B-A4B), and `12812264242a14dce44aa7ae27f931ff4584bcbf` (375B-A23B). The 375B-A23B benchmark launched a directly mounted checkpoint; the other five launches used materialized Hugging Face snapshots. ## 3. Benchmark results Speed was measured with native `sglang.benchmark.serving` using fixed random token IDs: 8,192 input tokens, 1,024 output tokens, 64 warmup requests, a cache flush, and seed `20260901`. The latency run used 32 prompts at concurrency 1. The throughput run used 256 prompts at concurrency 64 for TP1, 32 for TP2, and 8 for TP8. TTFT and TPOT are medians within each launch; the benchmark cards show the arithmetic mean across the two launches. Throughput per GPU is `(total input tokens + total output tokens) / duration / TP`. GSM8K used the full 1,319-example test split with `sgl-eval`, 32 client threads, a 32,768-token output limit, temperature 0, top-p 0.95, seed 0, and high reasoning effort. The mean and both independent server-launch scores are shown below. All requests completed without errors. Responses that reached the output limit remain in the score: truncation was 6.90% in both 0.9B launches, 4.09% and 4.70% for 3.7B, and at most 1.14% for the other checkpoints. | Checkpoint | Launch 1 | Launch 2 | Mean | | - | -: | -: | -: | | 0.9B | 85.22% | 85.29% | 85.25% | | 3.7B | 92.12% | 91.89% | 92.00% | | 7B | 95.30% | 94.47% | 94.88% | | 32B | 95.75% | 96.21% | 95.98% | | MoVA-36B-A4B | 95.45% | 94.84% | 95.15% | | 375B-A23B | 95.53% | 95.60% | 95.56% | ## 4. Advanced usage ### 4.1 Reasoning The base recipes include `--reasoning-parser k2_horizon`. The parser returns thinking in `reasoning_content` and the final answer in `content`. The example output below was captured with the 3.7B checkpoint and both parser flags enabled. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="IFM/K2-Horizon-3.7B", messages=[{"role": "user", "content": "Hello"}], ) message = response.choices[0].message reasoning = getattr(message, "reasoning_content", None) if reasoning is None: reasoning = (message.model_extra or {}).get("reasoning_content") print("Reasoning:", reasoning) print("Answer:", message.content) ``` ```text Output theme={null} Reasoning: We have a user who just said "Hello". We need to respond. As an AI, we should be friendly, perhaps ask how can we help. If says "Hello" alone, respond with greeting and offer assistance. Potentially ask what they need. There's no special instruction; just typical. But maybe we must think about the context: The user gave just a greeting; we should respond with a greeting and ask how we can help. Could also ask if they want anything specific. We should be concise but friendly: "Hello! How can I assist you today?" Provide some content. But maybe the user expects something else. But in this scenario, we just reply with a greeting. Thus answer: "Hello! What can I do for you?" Etc. Check any style guidelines: We want to keep it short or perhaps ask follow-up. The user didn't ask any question, just saying hello. Could also respond: "Hello! ..." Thus arguably the answer: "Hi there! How can I help you today?" Let's do that. But also maybe we could ask if they need help with something specific. That's appropriate. Thus the final answer: "Hello! How can I assist you today?" Possibly add friendly banter. We'll respond. Answer: Hello! 👋 How can I help you today? ``` ### 4.2 Tool calling Enable **Tool Call Parser** in the **Parsers** card of the [Playground](#playground) to add `--tool-call-parser k2_horizon`. The parser exposes K2 tool output as OpenAI-compatible `message.tool_calls`. The example output below was captured with the 32B checkpoint and both parser flags enabled. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Return weather for a city.", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, } ] response = client.chat.completions.create( model="IFM/K2-Horizon-32B", messages=[{"role": "user", "content": "Call get_weather for Paris."}], tools=tools, tool_choice={"type": "function", "function": {"name": "get_weather"}}, max_tokens=1024, ) message = response.choices[0].message reasoning = getattr(message, "reasoning_content", None) if reasoning is None: reasoning = (message.model_extra or {}).get("reasoning_content") print("Reasoning:", reasoning) print("Content:", message.content or "") for tool_call in message.tool_calls or []: print(f"Tool call: {tool_call.function.name}({tool_call.function.arguments})") ``` ```text Output theme={null} Reasoning: Okay, the user wants me to call the get_weather function for Paris. Let me check the tools available. There's a function called get_weather that takes a city parameter. The required parameter is city, and it's a string. So I need to make sure to pass "Paris" as the city. I don't see any other parameters needed. The user didn't mention any other details, so I'll just use the city name. Let me structure the tool call correctly. The function name is get_weather, and the arguments should be a JSON object with "city": "Paris". I need to make sure the JSON is properly formatted. Alright, that should do it. Content: Tool call: get_weather({"city": "Paris"}) ``` ### 4.3 HiCache For large checkpoints or repeated long-context prefixes, enable **HiCache** in the [Playground](#playground), then choose a storage backend and write policy. Treat the generated override as unverified until it is exercised on your storage stack. # IQuest-Q1 Source: https://docs.sglang.io/cookbook/autoregressive/IQuestLab/IQuest-Q1 Deploy IQuest-Q1 with SGLang — BF16 recipes for the 320B Mixture-of-Experts model on 8× NVIDIA H200, with optional MTP speculative decoding. ## Deployment IQuest-Q1 needs an SGLang build that includes IQuest-Q1 support. The nightly image carries it: ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` For other methods, see the [official SGLang installation guide](../../../docs/get-started/install). Download the checkpoint first; the MTP draft lives in its `mtp/` subdirectory: ```bash Command theme={null} hf download IQuestLab/IQuest-Q1 --local-dir /model/IQuest-Q1 ``` Pick a recipe to generate the launch command. **Low-Latency** enables MTP speculative decoding; **High-Throughput** serves the target model alone. ## 1. Model Introduction **IQuest-Q1** is a Mixture-of-Experts language model from IQuestLab with about 320B total parameters and 15B active per token. It has 88 layers that interleave three sliding-window attention layers (4,096-token window) with one full-attention layer, uses learned attention sinks, and routes each token to 8 of 256 experts. It ships with a single-layer MTP draft that SGLang applies recursively for speculative decoding. **Resources:** [HuggingFace](https://huggingface.co/IQuestLab/IQuest-Q1). ## 2. Configuration Tips * **Hardware.** IQuest-Q1 requires the FA3 attention backend for both the target and the MTP draft, because the attention sink is applied from the FA3 softmax LSE. SGLang builds FA3 for Hopper, so Blackwell GPUs are not supported. The BF16 weights take about 640 GB, which fits 8× H200 with tensor parallelism 8. * **Reasoning.** Thinking is on by default. Pass `"chat_template_kwargs": {"thinking": false}` to turn it off for a request. The `iquest_q1` reasoning parser returns the thinking in `reasoning_content`. * **Tool calling.** The `iquest_q1` tool-call parser supports `tool_choice` values `auto`, `required`, and a named function. * **MTP depth.** The single draft layer runs 5 sequential steps, one token each, and each round verifies 6 tokens. Keep `--speculative-num-draft-tokens` equal to `--speculative-num-steps` + 1. # LLaDA 2.1 Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/LLaDA-2.1 ## 1. Model Introduction [LLaDA 2.1](https://github.com/inclusionAI/LLaDA2.X) is a series of large-scale discrete diffusion language models (dLLMs) developed by the InclusionAI team at Ant Group. Unlike traditional autoregressive models that generate text left-to-right one token at a time, LLaDA 2.1 uses a diffusion-based approach — drafting tokens in parallel and refining them through iterative denoising, enabling self-correction during generation. **Key Features:** * **Token Editing (T2T + M2T)**: Combines Mask-to-Token (M2T) and Token-to-Token (T2T) editing, allowing the model to not only unmask tokens but also revise already-generated tokens mid-flight * **Dual Decoding Modes**: Speed Mode (S) for maximum throughput with T2T refinement, and Quality Mode (Q) for conservative thresholds and higher benchmark scores * **MoE Architecture**: Both variants use Mixture-of-Experts architecture for efficient scaling * **First Large-Scale RL for dLLMs**: Implements the first reinforcement learning framework specifically designed for diffusion language models, improving reasoning and instruction-following * **Lightning-Fast Decoding**: Up to 892 tokens/s on HumanEval+ for the 100B model **Available Models:**
Model Parameters Architecture Context Length HuggingFace
**LLaDA2.1-mini** 16B MoE (20 layers, 16 attention heads) 32,768 tokens [inclusionAI/LLaDA2.1-mini](https://huggingface.co/inclusionAI/LLaDA2.1-mini)
**LLaDA2.1-flash** 100B MoE 32,768 tokens [inclusionAI/LLaDA2.1-flash](https://huggingface.co/inclusionAI/LLaDA2.1-flash)
**License:** Apache 2.0. Please refer to the [official LLaDA2.X repository](https://github.com/inclusionAI/LLaDA2.X) for details. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and decoding mode. SGLang supports serving LLaDA-2.1 on NVIDIA H100, H200, B200, and AMD MI300X, MI325X, MI355X GPUs. ### 3.2 Configuration Tips **dLLM-Specific Parameters:**
Parameter Description Recommended Value
`--dllm-algorithm` Diffusion decoding algorithm `JointThreshold`
`--trust-remote-code` Required for LLaDA model loading Always enabled
`--mem-fraction-static` Static memory fraction for KV cache `0.8`
`--max-running-requests` Maximum concurrent requests `1` (for best quality)
`--attention-backend` Attention computation backend `flashinfer`
**Decoding Mode Comparison:**
Mode Threshold Speed Quality Best For
**Quality Mode (Q)** Conservative Moderate Higher benchmark scores Accuracy-critical tasks
**Speed Mode (S)** Aggressive Very fast, relies on T2T editing Slightly lower Throughput-critical tasks
**Hardware Requirements:** * **LLaDA2.1-mini (16B)**: \~47 GB VRAM, runs on a single GPU (TP=1) * **LLaDA2.1-flash (100B)**: Requires multi-GPU setup (TP=4 on H100/H200, TP=2 on B200) ## 4. Model Invocation ### 4.1 Deployment Start the server using the command generated above, for example: ```shell Command theme={null} python -m sglang.launch_server \ --model-path inclusionAI/LLaDA2.1-mini \ --dllm-algorithm JointThreshold \ --tp 1 \ --trust-remote-code \ --mem-fraction-static 0.8 \ --max-running-requests 1 \ --attention-backend flashinfer \ --host 0.0.0.0 \ --port 8000 ``` ### 4.2 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) **Simple Completion Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="inclusionAI/LLaDA2.1-mini", messages=[ {"role": "user", "content": "Explain what a diffusion language model is in simple terms."} ], max_tokens=1024 ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} Sure! Let's break it down in simple terms. A **diffusion language model** is a type of artificial intelligence that learns to generate text—like sentences, stories, or emails—by studying a lot of written text. Here’s how it works, using a simple real-life analogy: Imagine you have a big book full of stories. A diffusion language model is trying to learn how to write a new story. Instead of being told the rules, it starts by looking at all the words in the book and trying to understand how words usually go together. Now, think of the process like this: 1. **Start with random noise**: The model begins with a completely random set of words (like a scribble on paper). 2. ** ** "clean up" the noise**: It gradually "denoises" the noise by turning it into meaningful text, word by word, based on what it learned learned from the book. 3. **Learn from patterns**: As it does this, it learns patterns—like how words often follow each other, or how sentences start. 4. **Generate new text**: Once it’s learned the patterns, it can create new, coherent sentences or stories by starting from a and and building it up word by word. So, the "diffusion" part comes from the idea of going from random noise to clear, meaningful text—like turning a scribble into a full story. In short: A diffusion language model is an AI that learns to write text by reading lots of books and gradually turning random noise into coherent, meaningful sentences based on what it learned. ``` ### 4.3 Advanced Usage #### 4.3.1 Streaming ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="inclusionAI/LLaDA2.1-mini", messages=[ {"role": "user", "content": "Write a Python function to compute the Fibonacci sequence."} ], max_tokens=2048, stream=True ) for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ````text Output theme={null} Here are several ways to implement the Fibonacci sequence in Python: ## 1. Recursive Approach (Simple but Inefficient) ```python def fibonacci_recursive(n): """ Compute the nth Fibonacci number using recursion. Args: n (int): The position in the Fibonacci sequence (0-indexed) Returns: int: The nth Fibonacci number Raises: ValueError: If n is negative """ if n < 0: raise ValueError("n must be non-negative") if n <= 1: return n return fibonacci_recursive(n - 1) + fibonacci_recursive(n - 2) # Example usage print(fibonacci_recursive(10)) # Output: 55 ``` ## 2. Iterative Approach (Efficient) ... ```` #### 4.3.2 Code Generation ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="inclusionAI/LLaDA2.1-mini", messages=[ {"role": "user", "content": "Write a Python function that checks if a string is a palindrome. Include docstring and test cases."} ], max_tokens=2048 ) print(response.choices[0].message.content) ``` **Output Example:** ````text Output theme={null} ```python def is_palindrome(s): """ Check if a string is a palindrome. A palindrome is a word, phrase, or sequence that reads the same backward as forward. This function ignores case, spaces, punctuation, and non characters characters. Args: s (str): The string to check Returns: bool: True if the string is a palindrome, False otherwise Examples: >>> is_palindrome("racecar") True >>> is_palindrome("A man a plan a canal Panama") True >>> is_palindrome("race a car") False >>> is_palindrome("") True >>> is_palindrome("a") True """ # Remove non-alphanumeric characters and convert to lowercase cleaned = ''.join(char.lower() for char in s if char.isalnum()) # Check if the cleaned string reads the same forwards and backwards return cleaned == cleaned[::-1] # Test cases def test_is_palindrome(): """Test the is_palindrome function with various inputs.""" # Test basic palindromes assert is_palindrome("racecar") == True assert is_palindrome("level") == True assert is_palindrome("madam") == True assert is_palindrome("radar") == True # Test palindromes with spaces and punctuation assert is_palindrome("A man a plan a canal Panama") == True assert is_palindrome("race a car") == False assert is_palindrome("Was it a car or a cat I saw?") == True assert is_palindrome("Madam, I'm Adam") == True # Test edge cases assert is_palindrome("") == True assert is_palindrome("a") == True assert is_palindrome("A") == True assert is_palindrome("Aa") == True # Test non-palindromes assert is_palindrome("hello") == False assert is_palindrome("world") == False assert is_palindrome("python") == False # Test single characters assert is_palindrome("1") == True assert is_palindrome("1") == True print("All tests passed!") # Run the tests if __name__ == "__main__": # Example usage print("Testing isalindrome function:") print(f"'racecar' {is_palindrome('racecar')}") print(f"'A man a plan a canal Panama': {is_palindrome('A man a plan a canal Panama')}") print(f"'race a car': {is_palindrome('race a car')}") print(f"'hello': {is_palindrome('hello')}") # Run tests test_is_palindrome() ``` This implementation includes: 1. **Comprehensive function** `is_palindrome()` that: - Ignores case by converting to lowercase - Removes all non-alphanumeric characters (spaces, punctuation, etc.) - Uses string slicing (`[::-1]`) to reverse the string 2. **Detailed docstring** explaining: - What the function does - How it works - Return value - Examples of usage 3. **Extensive test cases** covering: - Basic palindromes - Palindromes with spaces and punctuation - Edge cases (empty string, single character) - Non-palindromes - Mixed case scenarios 4. **Test function** that uses assertions to verify the function works correctly The function efficiently handles real-world palindrome checking by ignoring case, spaces, and punctuation, making it suitable for phrases like "A man a plan a canal Panama". ```` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 (4x) * SGLang Version: 0.5.8+ #### 5.1.1 LLaDA2.1-mini **Model Deployment:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path inclusionAI/LLaDA2.1-mini \ --dllm-algorithm JointThreshold \ --tp 1 \ --trust-remote-code \ --mem-fraction-static 0.8 \ --max-running-requests 1 \ --attention-backend flashinfer ``` * Latency Benchmark ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model inclusionAI/LLaDA2.1-mini \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Latency Result**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 9.90 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 3433 Request throughput (req/s): 1.01 Input token throughput (tok/s): 616.26 Output token throughput (tok/s): 426.26 Peak output token throughput (tok/s): 1010.00 Peak concurrent requests: 3 Total token throughput (tok/s): 1042.53 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 988.87 Median E2E Latency (ms): 655.27 P90 E2E Latency (ms): 1952.50 P99 E2E Latency (ms): 2932.19 ---------------Time to First Token---------------- Mean TTFT (ms): 152.74 Median TTFT (ms): 150.37 P99 TTFT (ms): 229.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 2.16 Median TPOT (ms): 2.08 P99 TPOT (ms): 3.72 ---------------Inter-Token Latency---------------- Mean ITL (ms): 2.10 Median ITL (ms): 1.99 P95 ITL (ms): 4.03 P99 ITL (ms): 6.34 Max ITL (ms): 26.59 ================================================== ``` * Throughput Benchmark ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model inclusionAI/LLaDA2.1-mini \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * **Throughput Result**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 467.74 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 189717 Request throughput (req/s): 1.07 Input token throughput (tok/s): 534.12 Output token throughput (tok/s): 540.17 Peak output token throughput (tok/s): 1753.00 Peak concurrent requests: 105 Total token throughput (tok/s): 1074.30 Concurrency: 90.77 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 84912.27 Median E2E Latency (ms): 86564.26 P90 E2E Latency (ms): 110567.26 P99 E2E Latency (ms): 114303.38 ---------------Time to First Token---------------- Mean TTFT (ms): 83920.39 Median TTFT (ms): 85669.54 P99 TTFT (ms): 112969.91 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 2.67 Median TPOT (ms): 1.65 P99 TPOT (ms): 4.43 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.69 Median ITL (ms): 1.46 P95 ITL (ms): 3.96 P99 ITL (ms): 4.84 Max ITL (ms): 92.08 ================================================== ``` #### 5.1.2 LLaDA2.1-flash **Model Deployment:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path inclusionAI/LLaDA2.1-flash \ --dllm-algorithm JointThreshold \ --tp 4 \ --trust-remote-code \ --mem-fraction-static 0.8 \ --max-running-requests 1 \ --attention-backend flashinfer ``` * Latency Benchmark ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model inclusionAI/LLaDA2.1-flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Latency Result**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 14.46 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 3276 Request throughput (req/s): 0.69 Input token throughput (tok/s): 421.79 Output token throughput (tok/s): 291.75 Peak output token throughput (tok/s): 676.00 Peak concurrent requests: 3 Total token throughput (tok/s): 713.53 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1445.16 Median E2E Latency (ms): 968.06 P90 E2E Latency (ms): 3101.86 P99 E2E Latency (ms): 4208.49 ---------------Time to First Token---------------- Mean TTFT (ms): 231.63 Median TTFT (ms): 242.67 P99 TTFT (ms): 341.33 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 3.04 Median TPOT (ms): 2.79 P99 TPOT (ms): 5.33 ---------------Inter-Token Latency---------------- Mean ITL (ms): 3.05 Median ITL (ms): 2.41 P95 ITL (ms): 7.25 P99 ITL (ms): 8.27 Max ITL (ms): 29.27 ================================================== ``` * Throughput Benchmark ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model inclusionAI/LLaDA2.1-flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * **Throughput Result**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 671.85 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 177961 Request throughput (req/s): 0.74 Input token throughput (tok/s): 371.85 Output token throughput (tok/s): 376.07 Peak output token throughput (tok/s): 1521.00 Peak concurrent requests: 103 Total token throughput (tok/s): 747.92 Concurrency: 91.28 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 122658.36 Median E2E Latency (ms): 125265.55 P90 E2E Latency (ms): 159554.07 P99 E2E Latency (ms): 165174.88 ---------------Time to First Token---------------- Mean TTFT (ms): 121009.17 Median TTFT (ms): 124437.80 P99 TTFT (ms): 163579.29 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.73 Median TPOT (ms): 2.16 P99 TPOT (ms): 7.13 ---------------Inter-Token Latency---------------- Mean ITL (ms): 2.38 Median ITL (ms): 1.40 P95 ITL (ms): 6.89 P99 ITL (ms): 8.60 Max ITL (ms): 176.78 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:8000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. **Results:** ```text Output theme={null} Accuracy: 0.895 Invalid: 0.000 Latency: 100.552 s Output throughput: 262.094 token/s ``` # Ling-2.5-1T Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-2.5-1T ## 1. Model Introduction [Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T) is the latest flagship instant model in the Ling family. Thinking models raise the ceiling of intelligence, while instant models expand its reach by balancing efficiency and performance—making AGI not only more powerful, but also more accessible. Ling-2.5-1T delivers comprehensive upgrades across model architecture, token efficiency, and preference alignment, designed to bring universally accessible AI to a new level of quality. **Key Features:** * **Trillion-Scale Model**: 1T total parameters with 63B active parameters (up from 51B in the previous generation). Pre-training corpus expanded from 20T to 29T tokens. Leveraging an efficient hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention), the model delivers exceptionally high throughput while processing context lengths of up to 1M tokens. * **Token Efficiency**: By introducing a composite reward mechanism combining "Correctness" and "Process Redundancy", Ling-2.5-1T further pushes the frontier of efficiency-performance balance in instant models. At comparable token efficiency levels, Ling-2.5-1T's reasoning capabilities significantly outperform its predecessor, approaching the level of frontier "thinking models" that typically consume \~4x the output tokens. * **Preference Alignment**: Through refined alignment strategies—such as bidirectional RL feedback and Agent-based instruction constraint verification—Ling-2.5-1T achieves substantial improvements over the previous generation in preference alignment tasks, including creative writing and instruction following. * **Agentic Capabilities**: Trained with Agentic RL in large-scale high-fidelity interactive environments, Ling-2.5-1T is compatible with mainstream agent platforms such as Claude Code, OpenCode, and OpenClaw. It achieves leading open-source performance on the general tool-calling benchmark, BFCL-V4. * **Context Length**: 256K -> 1M (YaRN) **Available Models:** * **BF16**: [inclusionAI/Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T) **License:** MIT ## 2. SGLang Installation Ling-2.5-1T runs on the standard SGLang Docker image: ```bash Command theme={null} # NVIDIA (H200 / B200 / GB200 / GB300) docker pull lmsysorg/sglang:latest ``` For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). Ling-2.5-1T is also supported via the **nightly PyPI builds**. See the [SGLang Installation (PyPI)](../../../docs/get-started/install) guide for setup instructions. ## 3. Model Deployment Ling-2.5-1T is a trillion-parameter BF16 model that requires multi-node deployment (at least 2 nodes). Use the configuration selector below to generate the deployment command for your hardware platform. ### Configuration Tips * The `--trust-remote-code` flag is required for this model due to custom modeling code. * `--tp-size` can be set to a maximum of 8 for this model. If you have more GPUs available, increase `--pp-size` to scale across additional nodes. * Adding `--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}'` enables faster model loading. * On H200/GB200/GB300 with 2-node deployment, `--mem-frac 0.95` is required to avoid OOM since the model occupies most of the GPU memory. For better throughput, consider 4-node deployment (ref [model card](https://huggingface.co/inclusionAI/Ling-2.5-1T#run-inference) for more details). ## 4. Model Invocation ### 4.1 Basic Usage For example, launch the server on 2 H200 nodes: ```bash Command theme={null} export MASTER_IP=10.10.0.1 # The IP of Node 0 export PORT=30000 export DIST_PORT=50000 # Node 0: python3 -m sglang.launch_server \ --model-path inclusionAI/Ling-2.5-1T \ --trust-remote-code \ --tp-size 8 \ --pp-size 2 \ --nnodes 2 \ --node-rank 0 \ --host 0.0.0.0 \ --port ${PORT} \ --dist-init-addr ${MASTER_IP}:${DIST_PORT} \ --tool-call-parser qwen \ --model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \ --mem-frac 0.95 # Node 1: python3 -m sglang.launch_server \ --model-path inclusionAI/Ling-2.5-1T \ --trust-remote-code \ --tp-size 8 \ --pp-size 2 \ --nnodes 2 \ --node-rank 1 \ --dist-init-addr ${MASTER_IP}:${DIST_PORT} \ --tool-call-parser qwen \ --model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \ --mem-frac 0.95 ``` Once the server is running, send requests to the master node: ```bash Command theme={null} curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}' ``` Output: ```json Config theme={null} { "id": "e82af153da844ee6aed7a27a3187f2f4", "object": "chat.completion", "created": 1771216764, "model": "auto", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "The capital of France is **Paris**.\n\n**Additional details:**\n* It is the largest city in France.\n* It is located in the north-central part of the country along the Seine River.\n* Paris is often referred to as \"The City of Light\" (*La Ville Lumière*).", "reasoning_content": null, "tool_calls": null }, "logprobs": null, "finish_reason": "stop", "matched_stop": 156895 } ], "usage": { "prompt_tokens": 25, "total_tokens": 93, "completion_tokens": 68, "prompt_tokens_details": null, "reasoning_tokens": 0 } } ``` For more API usage examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Tool Calling Example ```bash Command theme={null} curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "inclusionAI/Ling-2.5-1T", "messages": [{"role": "user", "content": "Search for the latest news about AI"}], "tools": [{ "type": "function", "function": { "name": "search", "description": "Search for information on the internet", "parameters": { "type": "object", "properties": { "query": {"type": "string", "description": "The search query"} }, "required": ["query"] } } }], "tool_choice": "auto" }' ``` Output: ```json Config theme={null} { "id": "b968e45c7d414f7482c8ffc0f9c6b688", "object": "chat.completion", "created": 1771216520, "model": "inclusionAI/Ling-2.5-1T", "choices": [ { "index": 0, "message": { "role": "assistant", "content": null, "reasoning_content": null, "tool_calls": [ { "id": "call_e75f711d8ad840ed9d382c9e", "index": 0, "type": "function", "function": { "name": "search", "arguments": "{\"query\": \"latest news about AI\"}" } } ] }, "logprobs": null, "finish_reason": "tool_calls", "matched_stop": null } ], "usage": { "prompt_tokens": 173, "total_tokens": 196, "completion_tokens": 23, "prompt_tokens_details": null, "reasoning_tokens": 0 } } ``` ## 5. Benchmark ### GSM8K * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. * Test Result ```text Output theme={null} Accuracy: 0.960 Invalid: 0.000 Latency: 45.410 s Output throughput: 560.642 token/s ``` # Ling-2.6 Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-2.6 ## 1. Model Introduction The **Ling-2.6** family from inclusionAI is the next iteration of the Ling instant-model series. Continuing the architectural direction set by Ling-2.5, Ling-2.6 doubles down on **inference efficiency**, **token efficiency**, and **agent performance** — staying competitive with frontier instant models while being faster, leaner, and better suited for production agent workloads. **Key Features:** * **Hybrid Linear Attention**: A `1:7 MLA + Lightning Linear` hybrid built on top of a highly sparse MoE backbone. Compared with same-class SOTA models, Ling-2.6-flash shows up to \~4× higher prefill and decode throughput in long-context scenarios; Ling-2.6-1T is shipped in FP8 so it fits a single GB300 node with `--tp 4`. * **Token Efficiency**: Trained with explicit token-efficiency objectives. On the full Artificial Analysis suite, Ling-2.6-flash uses only \~15M output tokens while remaining competitive — a meaningfully stronger intelligence-per-token profile than long-reasoning peers. * **Agentic Capabilities**: Refined for tool use, multi-step planning, and long-horizon execution. Reaches SOTA-class results on **BFCL-V4**, **TAU2-bench**, **SWE-bench Verified**, **Claw-Eval**, and **PinchBench**, and is validated against Claude Code, Kilo Code, Qwen Code, Hermes Agent, and OpenClaw. * **Long Context**: Native 128K, extendable to **256K (Ling-2.6-flash)** and **256K → 1M (Ling-2.6-1T via YaRN)**. **Available Models:** * **BF16**: [inclusionAI/Ling-2.6-flash](https://huggingface.co/inclusionAI/Ling-2.6-flash) — 104B total / 7.4B active * **FP8 (E4M3)**: [inclusionAI/Ling-2.6-1T](https://huggingface.co/inclusionAI/Ling-2.6-1T) — \~1T total **License:** MIT ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment ### 3.1 Ling-2.6-flash Ling-2.6-flash is a 104B/7.4B-active MoE that runs comfortably on a single 4-GPU node. Use the selector below to generate the launch command for your hardware. #### Configuration Tips * `--trust-remote-code` is required (custom `BailingMoeV2_5ForCausalLM` modeling code). * `--tp-size 4` is the reference layout. On 4× H20-3e the model reaches \~340 tokens/s decode at TP=4, batch 32. * Native context is 128K. Enable YaRN (`--json-model-override-args '{"rope_scaling": {"rope_type": "yarn", "factor": 2.0, ...}}'`) to extend to 256K — the snippet does this for you. * `--tool-call-parser qwen25` matches the model's `...` schema. * The recommended baseline does **not** include `--reasoning-parser qwen3`. Ling-2.6 is a controllable-reasoning model whose chat template defaults to `detailed thinking off`; the SGLang `qwen3` reasoning parser, in contrast, assumes default-thinking semantics and would mis-route normal output into `reasoning_content`. Only enable it if you specifically want `...` blocks split out — see [§4.3 Thinking Mode](#4-3-thinking-mode). * **MTP (multi-token prediction)** is supported. Add `--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --mamba-radix-cache-strategy extra_buffer` to enable it — see the [model card](https://huggingface.co/inclusionAI/Ling-2.6-flash#run-inference) for the full example. ### 3.2 Ling-2.6-1T Ling-2.6-1T ships in **FP8 (E4M3)**, so unlike Ling-2.5-1T it fits a **single GB300 node with `--tp 4`**. On smaller GPUs (H200/B200), a 2-node deployment with `--pp-size 2` is required. #### Configuration Tips * `--trust-remote-code` is required for the custom modeling code. * `--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}'` significantly speeds up the multi-shard FP8 weight load (26 safetensors shards + an MTP layer). * Use `--tool-call-parser qwen` for tool calling. * The recommended baseline does **not** include `--reasoning-parser qwen3`. Ling-2.6's chat template defaults to `detailed thinking off`, while SGLang's `qwen3` reasoning parser assumes default-thinking semantics — combining the two requires a per-request workaround for tool calls (see [§4.3 Thinking Mode](#4-3-thinking-mode)). Only enable `--reasoning-parser qwen3` if you specifically want `...` blocks split into `reasoning_content`. * For 2-node deployments, set `MASTER_IP`, `PORT`, and `DIST_PORT` consistently across both nodes. ## 4. Model Invocation For example, launch a Ling-2.6-1T server on a single GB300 node: ```bash Command theme={null} sglang serve \ --model-path inclusionAI/Ling-2.6-1T \ --tp-size 4 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 \ --tool-call-parser qwen \ --model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}' ``` ### 4.1 Basic Usage ```bash Command theme={null} curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}' ``` Output: ```json Config theme={null} { "id": "...", "object": "chat.completion", "model": "auto", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "The capital of France is **Paris**.", "reasoning_content": null, "tool_calls": null }, "finish_reason": "stop" } ] } ``` ### 4.2 Tool Calling Example ```bash Command theme={null} curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [{"role": "user", "content": "Search for the latest news about AI"}], "tools": [{ "type": "function", "function": { "name": "search", "description": "Search for information on the internet", "parameters": { "type": "object", "properties": { "query": {"type": "string", "description": "The search query"} }, "required": ["query"] } } }], "tool_choice": "auto" }' ``` Output: ```json Config theme={null} { "choices": [ { "message": { "role": "assistant", "content": null, "tool_calls": [ { "id": "call_...", "type": "function", "function": { "name": "search", "arguments": "{\"query\": \"latest news about AI\"}" } } ] }, "finish_reason": "tool_calls" } ] } ``` ### 4.3 Thinking Mode Both Ling-2.6-flash and Ling-2.6-1T are **controllable-reasoning** models. Their chat template uses textual directives in the system message — `detailed thinking on` or `detailed thinking off` — to toggle thinking. The template **defaults to `detailed thinking off`** when neither phrase is present, and it does **not** read the Qwen3-style `enable_thinking` template variable. #### Enabling thinking Include `detailed thinking on` in the first system message: ```bash Command theme={null} curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [ {"role": "system", "content": "detailed thinking on"}, {"role": "user", "content": "If a box has 12 red balls and 8 blue balls, then 5 red balls are removed, how many balls remain?"} ] }' ``` If you already have a system prompt, append the directive on its own line: ```json theme={null} {"role": "system", "content": "You are a helpful assistant.\ndetailed thinking on"} ``` When thinking is on, the model emits `...` blocks before its final answer. To get those split into `message.reasoning_content` automatically, also launch the server with `--reasoning-parser qwen3`. #### Caveat: `--reasoning-parser qwen3` + tool calling The SGLang `qwen3` reasoning parser was written for Qwen3, where models are **default-thinking** and clients opt out via `chat_template_kwargs.enable_thinking=false`. Ling-2.6 is the opposite — default-non-thinking, with toggling done in the system message. As a result, when the server is launched with **both** `--tool-call-parser qwen` and `--reasoning-parser qwen3`, every tool-call request must include `chat_template_kwargs.enable_thinking=false`, otherwise the parser routes the `...` block into `reasoning_content` instead of `message.tool_calls`: ```bash Command theme={null} curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [{"role": "user", "content": "Search for the latest news about AI"}], "tools": [...], "tool_choice": "auto", "chat_template_kwargs": {"enable_thinking": false} }' ``` `enable_thinking` here is consumed by the SGLang reasoning parser, **not** by the chat template — Ling-2.6's template ignores it. For the simplest configuration, just omit `--reasoning-parser qwen3` and toggle thinking via the system message. For more API examples, see the [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request). ## 5. Benchmark ### GSM8K (Ling-2.6-1T, GB300 × 4) Reference run on a single GB300 node with `--tp 4`: ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. ```text Output theme={null} Accuracy: 0.9621 (1269 / 1319) ``` For Ling-2.6-flash, see the official numbers on the [model card](https://huggingface.co/inclusionAI/Ling-2.6-flash) (BFCL-V4, TAU2-bench, SWE-bench Verified, Claw-Eval, PinchBench, Artificial Analysis). # Ling-3.0-flash Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.0-flash Deploy Ling-3.0-flash with SGLang — a 124B hybrid KDA + MLA MoE in BF16, FP8, INT4, or MXFP4 on Hopper and Blackwell GPUs. ## Deployment ```bash Command theme={null} docker pull lmsysorg/sglang:dev-Ling-3.0-flash ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + recipe to generate the launch command. Three serving strategies are covered: * **Low-Latency** — fastest reply for a single user. Pick for chat. These recipes run speculative decoding: pick **NEXTN** (built-in MTP layer, no extra checkpoint) or **DSPARK** (external draft model) in the **Spec Decode** selector of the Deploy panel. * **High-Throughput** — most tokens per second across many users. Best for batch jobs. These recipes turn speculative decoding off, since at saturation the draft/verify overhead outweighs the speedup. * **HiCache + Mooncake** — writes reusable prefixes to Mooncake L3 storage. Start the Mooncake services in §3.3 before launching the generated server command. ## Playground The Playground is where you experiment with **SGLang features beyond the documented matrix**. The Deploy panel above only emits the curated recipe combinations on this page; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction Ling-3.0-flash is a hybrid-attention Mixture-of-Experts (MoE) language model from the BailingMoeV3 family. It interleaves Kimi Delta Attention (KDA) linear-attention layers with gated Multi-head Latent Attention (MLA) full-attention layers, on top of a fine-grained MoE feed-forward network. This keeps per-token inference cost close to a small model — **124B total parameters with only 5.1B active** — while retaining large-model capacity. It is a hybrid-reasoning model with thinking enabled by default, and it supports structured tool calling. Native context length is 256K. **Available Models:** * **BF16**: [inclusionAI/Ling-3.0-flash](https://huggingface.co/inclusionAI/Ling-3.0-flash) — 124B total / 5.1B active * **FP8** (blockwise E4M3): [inclusionAI/Ling-3.0-flash-fp8](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp8) * **INT4** (compressed-tensors W4A16): [inclusionAI/Ling-3.0-flash-int4](https://huggingface.co/inclusionAI/Ling-3.0-flash-int4) * **MXFP4**: [inclusionAI/Ling-3.0-flash-fp4](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp4) * **DSPARK draft**: [inclusionAI/Ling-3.0-flash-dspark](https://huggingface.co/inclusionAI/Ling-3.0-flash-dspark) — 5-layer DSpark draft checkpoint for DSPARK speculative decoding **License:** MIT **Resources:** [HuggingFace](https://huggingface.co/inclusionAI/Ling-3.0-flash). ## 2. Configuration Tips * BF16 tensor parallelism follows the GPU: `--tp 4` on 141 GB-class cards (H20-3e, H200) and 4-GPU Blackwell nodes (B200, GB300); `--tp 8` on 80 GB cards (H100, H800). * The FP8 recipes pair `--tp` with a matching `--ep-size` (`--tp 4 --ep-size 4` on 4-GPU nodes, `--tp 8 --ep-size 8` on H100/H800). The checkpoint uses blockwise (128×128) E4M3 expert weights, so a pure tensor-parallel shard must satisfy `(768 / TP) % 128 == 0` — only TP2 qualifies; expert parallelism splits experts whole instead of by column, which lifts that restriction and uses the full node. SGLang detects the quantization format from the checkpoint's `quantization_config`, so no explicit quantization flag is needed. * INT4 uses compressed-tensors W4A16 experts. With the default MoE runner, SGLang selects Marlin on H200 and the graph-compatible Triton implementation on Blackwell; do not force a backend in the launch command. * MXFP4 uses the native FlashInfer runner on H200 and B200. The checkpoint mixes MXFP4 routed experts with block-FP8 dense and shared projections; on B200, `--fp8-gemm-backend triton` avoids an unsupported FlashInfer FP8 tactic while the routed experts remain native MXFP4. * `--reasoning-parser ling3` and `--tool-call-parser ling3` enable Ling-3.0-specific reasoning and structured tool-call parsing; toggle them in the **Parsers** card of the [Playground](#playground). * Both the chat template and the `ling3` reasoning parser default to thinking on. A single request can turn it off with `"chat_template_kwargs": {"enable_thinking": false}` (see §3.1). * The BF16/FP8 recipes use `--mem-fraction-static 0.8`; INT4/MXFP4 use `0.85`. These values reserve the headroom used by the validated graph-enabled runs. * The checkpoint ships a built-in MTP layer (`num_nextn_predict_layers: 1`); enable it with `--speculative-algorithm NEXTN` — no separate draft model is needed. The Low-Latency recipes default to it via the **Spec Decode** selector; toggle it in the **Speculative Decoding** card of the [Playground](#playground). * DSPARK is the alternative speculative path: it drafts with the external [inclusionAI/Ling-3.0-flash-dspark](https://huggingface.co/inclusionAI/Ling-3.0-flash-dspark) checkpoint instead of the built-in MTP layer. Pick **DSPARK** in the Deploy panel's **Spec Decode** selector (also available in the [Playground](#playground)); it composes `--speculative-algorithm DSPARK`, the draft path, and `--enable-linear-replayssm-spec`, the KDA verify path that folds per-draft intermediate states onto a fixed ring instead of snapshotting full states. The draft block size auto-infers from the checkpoint; tune it with `--speculative-dspark-block-size`. The draft's block size of 8 makes the verify window 9 tokens, and the KDA ReplaySSM ring must be a power of two at least twice the window — the recipe therefore pins `--linear-replayssm-cache-len 32`, since the 16 default fails startup validation. MXFP4 requires `--moe-runner-backend flashinfer_mxfp4`; the mixed checkpoint does not auto-select it. * Native context is 256K; SGLang reads it from the checkpoint's `max_position_embeddings`, so no `--context-length` flag is needed. * The **HiCache** card in the [Playground](#playground) exposes the validated Mooncake L3 path. It adds the hybrid-KDA scheduler and prefix-key settings together; see §3.3 for the required services. ## 3. Advanced Usage ### 3.1 Reasoning Ling-3.0-flash thinks by default. With `--reasoning-parser ling3` (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)), the chain-of-thought is returned in `message.reasoning_content` and the final answer in `message.content`: ```bash Command theme={null} curl -s http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "inclusionAI/Ling-3.0-flash", "messages": [{"role": "user", "content": "What is 15% of 240?"}] }' ``` ```json Output theme={null} { "choices": [ { "message": { "role": "assistant", "content": "15% of 240 = **36**", "reasoning_content": "The user is asking a simple percentage calculation: 15% of 240. This is straightforward: 0.15 × 240 = 36.", "tool_calls": null }, "finish_reason": "stop" } ] } ``` Thinking is controlled by the chat template's `enable_thinking` kwarg and is on by default. Disable it per request with `"chat_template_kwargs": {"enable_thinking": false}`. ### 3.2 Tool Calling With `--tool-call-parser ling3` (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)), structured calls are parsed into `message.tool_calls` and `finish_reason` is `tool_calls`: ```bash Command theme={null} curl -s http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "inclusionAI/Ling-3.0-flash", "messages": [{"role": "user", "content": "Search for the latest news about AI"}], "tools": [{ "type": "function", "function": { "name": "search", "description": "Search for information on the internet", "parameters": { "type": "object", "properties": { "query": {"type": "string", "description": "The search query"} }, "required": ["query"] } } }], "tool_choice": "auto" }' ``` ```json Output theme={null} { "choices": [ { "message": { "role": "assistant", "content": "", "reasoning_content": "The user wants me to search for the latest news about AI. I'll use the search tool with a query about the latest AI news.", "tool_calls": [ { "id": "call_0822dd418aa34254aa5b19e1", "index": 0, "type": "function", "function": { "name": "search", "arguments": "{\"query\": \"latest news about AI 2025\"}" } } ] }, "finish_reason": "tool_calls" } ] } ``` For more API examples, see the [SGLang Basic Usage Guide](/docs/basic_usage/send_request). ### 3.3 HiCache with Mooncake The HiCache recipes use Mooncake as L3 prefix storage. Start the metadata server, master, and storage client before you launch SGLang. The following command runs all three services from the same image in a separate container: ```bash Command theme={null} docker run --rm --network host --ipc=host \ lmsysorg/sglang:dev-Ling-3.0-flash \ bash -lc ' python3 -m mooncake.http_metadata_server --port 8290 & mooncake_master --port 50171 --metrics_port 9024 & exec mooncake_client \ --host=127.0.0.1 \ --port=50172 \ --master_server_address=127.0.0.1:50171 \ --metadata_server=http://127.0.0.1:8290/metadata \ --protocol=tcp \ --device_names= \ --global_segment_size=4294967296 \ --enable_http_server=true \ --http_port=8291 ' ``` Then select **HiCache + Mooncake** in Deployment, or enable **HiCache** in the Playground. Docker commands use host networking so the SGLang container can reach these localhost services. You can change the master and metadata endpoints in the **Env** dialog. This validated setup uses TCP, so `MOONCAKE_DEVICE=` and the client's `--device_names=` are intentionally empty. Set both to your actual device list only when you configure an RDMA deployment. With the default `chunked_prefill_size` of 8192, a cold request writes through only when its uncached extend length fits in one chunk. A longer cold first request skips write-through for that influx; a repeat with the same prefix can hit the device radix cache and proceed normally. For this hybrid KDA model, use the Mooncake L3 recipe shown here. Host-memory L2 eviction is not exposed because the KDA cache path is not currently compatible with it. Track the limitation in [issue #33713](https://github.com/sgl-project/sglang/issues/33713). For storage sizing and backend details, see [HiCache best practices](/docs/advanced_features/hicache_best_practices). # Ling-3.0-flash-VL Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.0-flash-VL Deploy Ling-3.0-flash-VL with SGLang — a ~125B hybrid KDA + MLA MoE vision-language model (5.1B active) with image and video input, thinking mode, and 128K context on Blackwell and Hopper GPUs. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. Ling-3.0-flash-VL support requires sglang with sgl-project/sglang#38526 (or newer); until it merges, install directly from the PR head: ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install "sglang @ git+https://github.com/sgl-project/sglang.git@refs/pull/38526/head#subdirectory=python" uv pip install torchcodec==0.16.0 # optional, preferred video-decode backend ``` Video input needs `decord` or `torchcodec` in the environment; audio is not supported. Downloading the checkpoint requires `HF_TOKEN`. ```bash Command theme={null} docker pull lmsysorg/sglang:dev-Ling-3.0-flash-VL ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware and quantization to generate the launch command. The checkpoint ships in BF16 with FP8, INT4, and FP4 variants, and each combination carries one **Balanced** recipe: BF16 uses `--tp 4` on 288 GB-class (GB300, B300) and 141 GB-class (B200, H200) GPUs and `--tp 8` on H100 (sized so the \~250 GB of BF16 weights fit with KV-cache headroom); FP8 uses `--tp 1` on GB300/B300/B200 and `--tp 2` on H200/H100; INT4 and FP4 use `--tp 1` everywhere. DGX Spark offers INT4 and FP4 only. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits the recipes on this page; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction Ling-3.0-flash-VL is the vision-language member of the BailingMoeV3 family. Its language backbone is the [Ling-3.0-flash](/cookbook/autoregressive/InclusionAI/Ling-3.0-flash) hybrid-attention Mixture-of-Experts (MoE) model: 42 layers that interleave Kimi Delta Attention (KDA) linear-attention layers with gated Multi-head Latent Attention (MLA) full-attention layers, on top of a fine-grained MoE feed-forward network with 512 routed experts (8 active per token) plus a shared expert. A 27-layer vision encoder (0.4B parameters, 16-pixel patches with 2×2 spatial merging and paired frames for video) feeds image and video tokens into that backbone through a small MLP projector. In total the checkpoint holds **\~125B parameters with \~5.1B active per token**, so per-token inference cost stays close to a small model. It is a thinking model: the chat template turns chain-of-thought on by default and exposes an `enable_thinking` switch. It supports structured tool calling. Inputs are text, images, and video; audio is not supported. Native context length is 128K tokens. Unlike Ling-3.0-flash, the VL checkpoint ships no built-in MTP draft layer, so there is no speculative-decoding recipe. **Available Models:** * **BF16**: [inclusionAI/Ling-3.0-flash-VL](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL) — \~125B total / \~5.1B active * **FP8**: [inclusionAI/Ling-3.0-flash-VL-FP8](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-FP8) — 128×128 block quantization * **INT4 (GPTQ)**: [inclusionAI/Ling-3.0-flash-VL-int4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-int4) — compressed-tensors, group size 32 * **FP4 (MXFP4)**: [inclusionAI/Ling-3.0-flash-VL-fp4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-fp4) — e2m1 experts + FP8 e4m3 activations, Blackwell only **License:** MIT **Recommended generation:** `temperature=1.0`, `top_p=0.95`, `top_k=20` (from the checkpoint's `generation_config.json`; SGLang applies these defaults, so the samples below do not set them). **Resources:** [HuggingFace](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL). ## 2. Configuration Tips * `--trust-remote-code` is required. The checkpoint declares its config and multimodal processor through `auto_map`, and SGLang's native implementation loads the image/video preprocessing and the chat template from those files. Serving from the Hub repo or from a local snapshot needs no extra environment setup. * BF16 weights take \~250 GB. The recipes use `--tp 4` on 288 GB-class (GB300, B300) and 141 GB-class (B200, H200) GPUs and `--tp 8` on H100, with the default memory pool. The 4×GB300 and 4×H200 cells are validated end to end; the rest are sized by the same rule and carry the unverified badge. Adjust TP in the **Attention Parallelism** card of the [Playground](#playground). * HiCache is supported: `--enable-hierarchical-cache` attaches the hierarchical cache to the hybrid SSM state (verified with CPU offload; L3 storage backends such as Mooncake are untested). On shared hosts, set an explicit `--hicache-size ` — the default ratio can request more host RAM than is free. * Vision token budget: the encoder emits one token per 32×32-pixel block after merging, so a 1280×720 image costs about 880 tokens and the processor's `max_pixels` of 4,194,304 caps a single image at roughly 4,100 tokens. Video frames are paired before patching; the sample clip in §3.2 costs 4,224 tokens. Size `--chunked-prefill-size` and `--max-running-requests` with these counts in mind for image-heavy traffic. * Parsers: the generated recipes include `--reasoning-parser auto --tool-call-parser auto` by default. Both resolve to `ling3` from the chat template (the server logs `Auto-detected --reasoning-parser as ling3 from chat template` and the same for the tool-call parser), so no model-specific parser name is needed; the explicit form `--reasoning-parser ling3 --tool-call-parser ling3` is equivalent. With the parsers on, reasoning is returned in `message.reasoning_content` and structured tool calls in `message.tool_calls`; without them, `message.content` carries the reasoning followed by `` and the answer (the opening `` tag sits in the prompt and is not echoed). * Thinking is on by default (`enable_thinking` defaults to true in the chat template, and the `ling3` reasoning parser follows that default). Turn it off per request with `"chat_template_kwargs": {"enable_thinking": false}` (§3.3). * Native context is 128K, and the generated recipes default to the 256K YaRN variant (`--context-length 262144` plus the `rope_scaling` override and `SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1`), matching the model card. For the native 128K window, drop all three from the command. * Audio content parts are rejected before preprocessing; send images as `image_url` and clips as `video_url` content parts (§3). * The FP8 variant (`inclusionAI/Ling-3.0-flash-VL-FP8`) uses 128×128 block quantization, so plain TP must keep `768 ÷ tp` divisible by 128 (TP 1, 2, 3, or 6 — TP=4 fails to load). FP8 recipes default to a single GPU where the \~126 GB weights fit (`--tp 1` on 288 GB-class and B200, verified on GB300) or `--tp 2` on tighter cards. As a manual alternative on four GPUs, `--tp 4 --ep 4` keeps experts whole per rank (measured on 4×GB300). The BF16 checkpoint can also be quantized online with `--quantization fp8`; measured accuracy and speed for both paths are in the benchmark cards below. * The INT4 variant (`inclusionAI/Ling-3.0-flash-VL-int4`) is a GPTQ 4-bit export (compressed-tensors, group size 32). It runs on all listed hardware with single-GPU (`--tp 1`) recipes; GB300, H200, and DGX Spark cells are verified, the rest carry the unverified badge. Shared-expert fusion is auto-disabled for the INT4 and FP4 checkpoints by the current dev image and by sglang main; older images need `--disable-shared-experts-fusion` explicitly. * The FP4 variant (`inclusionAI/Ling-3.0-flash-VL-fp4`) is an MXFP4 export (e2m1 routed-expert weights with e8m0 scales, group 32, plus FP8 e4m3 dynamic activations) and runs on Hopper (SM90 CUTLASS W4A16) and Blackwell (SM100/SM120): H200, GB300, B300, B200, and DGX Spark. The `flashinfer_mxfp4` MoE backend is selected automatically on the `lmsysorg/sglang:dev-Ling-3.0-flash-VL` image and on sglang main; on builds that already include the VL integration but predate the auto-selection, add `--moe-runner-backend flashinfer_mxfp4` explicitly. GB300, H200, and DGX Spark cells are verified, the rest carry the unverified badge. ## 3. Advanced Usage The examples below were run against a server launched with the recipe above plus `--reasoning-parser auto --tool-call-parser auto`, which resolves both parsers to `ling3` (§2). Reasoning therefore arrives in `message.reasoning_content` and the answer in `message.content`. ### 3.1 Image Understanding Send images as OpenAI-style `image_url` content parts: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="inclusionAI/Ling-3.0-flash-VL", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image in one sentence."}, ], }], max_tokens=2048, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) ``` ```text Output theme={null} Reasoning: The user wants a one-sentence description of this image. Let me look at the image carefully. The image shows a man wearing a yellow shirt (with some kind of text on it, probably "Sudsy" or similar) standing behind a yellow taxi SUV, ironing clothes on a portable ironing board set up at the rear of the vehicle. On the left side, another yellow taxi (sedan) is passing by on the street. The background is an urban streetscape with buildings lined up. I need to write this in one sentence. For example: "A man wearing a yellow shirt irons clothes on a portable ironing board attached to the back of a yellow taxi on a city street, while another taxi passes by." Let me make it concise yet descriptive. One sentence. "On a city street, a man in a yellow sweatshirt irons clothes on a portable ironing board mounted on the back of a yellow taxi SUV, while another yellow taxi drives past." This works. I'll return only the answer. Content: A man in a yellow shirt irons clothes on a portable ironing board set up at the back of a yellow taxi on a city street, while another yellow cab drives past. ``` ### 3.2 Video Understanding Clips go in as `video_url` content parts. The processor samples and pairs frames itself; no client-side frame extraction is needed: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY", timeout=900) resp = client.chat.completions.create( model="inclusionAI/Ling-3.0-flash-VL", messages=[{ "role": "user", "content": [ {"type": "video_url", "video_url": {"url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4"}}, {"type": "text", "text": "What is happening in this video? Answer in two sentences."}, ], }], max_tokens=2048, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) ``` ```text Output theme={null} Reasoning: The user wants me to describe what is happening in this video in two sentences. Looking at the video, I can see: - A man on stage (appears to be Steve Jobs) presenting something - He's wearing a black shirt and jeans - There's a large screen behind him showing a close-up of a hand reaching toward a pocket on blue jeans - Text appears on screen asking "Anyone know what this pocket is for?" - He pulls out a small white device (iPod Nano) from his pocket - The text says "I've always wondered that" and "well now we know" and "is the new iPod Nano" - He holds up the white iPod Nano device This is clearly Steve Jobs presenting the new iPod Nano at an Apple event. The small white device is the iPod Nano. I need to summarize this in exactly two sentences. Content: A presenter on stage pulls a small white device from his pocket while a large screen behind him displays a close-up image of a hand reaching toward a jeans pocket. He holds the device up to reveal it is the new iPod Nano, answering the on-screen text that questioned what the small pocket was designed for. ``` ### 3.3 Thinking Mode Thinking is on by default. Disable it for a single request through the chat template's `enable_thinking` kwarg; the model then answers directly and `reasoning_content` stays empty: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="inclusionAI/Ling-3.0-flash-VL", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image in one sentence."}, ], }], max_tokens=512, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) ``` ```text Output theme={null} Reasoning: None Content: A man in a yellow sweatshirt irons a blue shirt on a folding board attached to the back of a parked yellow taxi SUV, while another yellow cab drives past on a city street. ``` ### 3.4 Tool Calling With the tool-call parser on, structured calls are parsed into `message.tool_calls` and `finish_reason` is `tool_calls`. Send the tool result back as a `tool` message to get the final answer; on this thinking model the follow-up turn may put text in `reasoning_content` as well as `content`, so print both: ```python Example theme={null} import json from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string", "description": "City name"}}, "required": ["city"], }, }, }] messages = [{"role": "user", "content": "What's the weather like in Paris today?"}] resp = client.chat.completions.create( model="inclusionAI/Ling-3.0-flash-VL", messages=messages, tools=tools, tool_choice="auto", max_tokens=2048, ) msg = resp.choices[0].message print("finish_reason:", resp.choices[0].finish_reason) print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) for call in msg.tool_calls: print("Tool call:", call.function.name, call.function.arguments) messages.append({ "role": "assistant", "content": msg.content or "", "tool_calls": [{"id": c.id, "type": "function", "function": {"name": c.function.name, "arguments": c.function.arguments}} for c in msg.tool_calls], }) messages.append({ "role": "tool", "tool_call_id": msg.tool_calls[0].id, "content": json.dumps({"city": "Paris", "temperature_c": 18, "condition": "light rain"}), }) resp = client.chat.completions.create( model="inclusionAI/Ling-3.0-flash-VL", messages=messages, tools=tools, max_tokens=2048, ) msg = resp.choices[0].message print("finish_reason:", resp.choices[0].finish_reason) print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) ``` ```text Output theme={null} finish_reason: tool_calls Reasoning: The user wants to know the current weather in Paris. I'll call the get_weather function with the city "Paris". Content: Let me check the weather in Paris for you right away! Tool call: get_weather {"city": "Paris"} finish_reason: stop Reasoning: None Content: Here's the current weather in Paris: - 🌡️ **Temperature:** 18°C - 🌧️ **Condition:** Light rain It looks like a mild day with some light rain. You might want to bring an umbrella if you're heading out! ☂️ ``` For more API examples, see the [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) and the [Tool Parser guide](../../../docs/advanced_features/tool_parser). # Ling-3.0-tiny Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.0-tiny Deploy Ling-3.0-tiny with SGLang — a compact ~7.9B total / ~1.2B active hybrid KDA + MLA MoE in BF16, FP8, or INT4, with thinking mode and tool calling. ## Deployment ```bash Command theme={null} docker pull lmsysorg/sglang:dev-Ling-3.0-tiny ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + recipe to generate the launch command. One serving strategy is covered: * **High-Throughput** — most tokens per second across many users. Best for batch jobs. Ling-3.0-tiny ships no built-in MTP draft layer (`num_nextn_predict_layers: 0`), so there is no NEXTN speculative-decoding recipe. ## Playground The Playground is where you experiment with **SGLang features beyond the documented matrix**. The Deploy panel above only emits the curated recipe combinations on this page; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction Ling-3.0-tiny is a compact hybrid-attention Mixture-of-Experts (MoE) language model from the BailingMoeV3 family — the small variant of [Ling-3.0-flash](/cookbook/autoregressive/InclusionAI/Ling-3.0-flash). It interleaves Kimi Delta Attention (KDA) linear-attention layers with gated Multi-head Latent Attention (MLA) full-attention layers on top of a fine-grained MoE feed-forward network, keeping per-token inference cost near a \~1B dense model — **\~7.9B total parameters with \~1.2B active** — while retaining large-model capacity. It is a thinking model with chain-of-thought enabled by default, and it supports structured tool calling. Native context length is 128K. Unlike Ling-3.0-flash, it ships **no built-in MTP draft layer**, so it does not use NEXTN speculative decoding. **Available Models:** * **BF16**: [inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny) — \~7.9B total / \~1.2B active * **FP8** (blockwise E4M3): [inclusionAI/Ling-3.0-tiny-fp8](https://huggingface.co/inclusionAI/Ling-3.0-tiny-fp8) * **INT4** (compressed-tensors W4A16): [inclusionAI/Ling-3.0-tiny-int4](https://huggingface.co/inclusionAI/Ling-3.0-tiny-int4) **License:** MIT **Resources:** [HuggingFace](https://huggingface.co/inclusionAI/Ling-3.0-tiny). ## 2. Configuration Tips * At \~7.9B total / 15.8 GB in BF16 (\~7.9 GB in FP8 and \~5.8 GB in INT4), a single GPU is plenty on every supported card. Tensor parallelism is only useful to raise aggregate KV-cache capacity for many long-context concurrent requests — add `--tp 2`/`--tp 4` to a multi-GPU serve directly. * Use the dedicated `lmsysorg/sglang:dev-Ling-3.0-tiny` runtime image; it includes the compressed-tensors Hopper and Blackwell backends that INT4 needs. * The FP8 checkpoint uses blockwise (128×128) E4M3 weights with dynamic activations, quantized from the BF16 model with attention projections, the dense MoE gate, and the lm\_head left in higher precision. SGLang detects the format from the checkpoint's `quantization_config`, so no explicit quantization flag is needed, and the same single-GPU recipe serves it. * The INT4 checkpoint uses symmetric group-32 W4A16 routed experts. SGLang selects Marlin on Hopper and Triton WNA16 on Blackwell automatically; no explicit quantization or MoE backend flag is needed. * Unlike Ling-3.0-flash (which pairs `--reasoning-parser ling3` / `--tool-call-parser ling3`), Ling-3.0-tiny uses `--reasoning-parser deepseek-r1` and `--tool-call-parser glm45` (its auto-detected template pairing) — the template wraps tool calls in `` blocks and emits an inline `...` chain-of-thought. Toggle them in the **Parsers** card of the [Playground](#playground). * Only `--model-path`, `--host`, and `--port` are needed. SGLang auto-resolves the context length (native 128K from `max_position_embeddings`), the attention backend, and `--mem-fraction-static` from the GPU and the CUDA-graph runtime, so the recipes leave them unset. * The chat template defaults to thinking on. Turn it off per request with `"chat_template_kwargs": {"enable_thinking": false}` for direct answers without the `...` block. * Ling-3.0-tiny ships no built-in MTP draft layer (`num_nextn_predict_layers: 0`), so `--speculative-algorithm NEXTN` is not applicable. ## 3. Advanced Usage ### 3.1 Reasoning With `--reasoning-parser deepseek-r1` (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)), the chain-of-thought is returned in `message.reasoning_content` and the final answer in `message.content`: ```bash Command theme={null} curl -s http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "inclusionAI/Ling-3.0-tiny", "messages": [{"role": "user", "content": "What is 15% of 240?"}] }' ``` ```json Output theme={null} { "choices": [ { "message": { "role": "assistant", "content": "15% of 240 is **36**.\n\n**Calculation:** 0.15 × 240 = 36", "reasoning_content": "The user is asking for 15% of 240. This is a simple percentage calculation.\n\n15% of 240 = 0.15 × 240 = 36\n\nLet me verify: 0.15 × 240 = 0.15 × 200 + 0.15 × 40 = 30 + 6 = 36. Yes, that's correct.", "tool_calls": null }, "finish_reason": "stop" } ] } ``` Thinking is controlled by the chat template's `enable_thinking` kwarg and is on by default. Disable it per request with `"chat_template_kwargs": {"enable_thinking": false}`. ### 3.2 Tool Calling With `--tool-call-parser glm45` (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)), structured calls are parsed into `message.tool_calls` and `finish_reason` is `tool_calls`: ```bash Command theme={null} curl -s http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "inclusionAI/Ling-3.0-tiny", "messages": [{"role": "user", "content": "Search for the latest news about AI"}], "tools": [{ "type": "function", "function": { "name": "search", "description": "Search for information on the internet", "parameters": { "type": "object", "properties": { "query": {"type": "string", "description": "The search query"} }, "required": ["query"] } } }], "tool_choice": "auto" }' ``` ```json Output theme={null} { "choices": [ { "message": { "role": "assistant", "content": "Let me search for the latest news about AI for you.", "reasoning_content": "The user wants me to search for the latest news about AI. I'll use the search tool to find recent AI news.", "tool_calls": [ { "id": "call_79b73a89696d4544ac6dd724", "index": 0, "type": "function", "function": { "name": "search", "arguments": "{\"query\": \"latest AI news 2025\"}" } } ] }, "finish_reason": "tool_calls" } ] } ``` For more API examples, see the [SGLang Basic Usage Guide](/docs/basic_usage/send_request). # Ring-2.5-1T Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ring-2.5-1T ## 1. Model Introduction [Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T) is the world's first open-source trillion-parameter reasoning model based on hybrid linear attention architecture, developed by InclusionAI. Building on Ring-1T, Ring-2.5-1T demonstrates substantial improvements in generation efficiency, reasoning depth, and long-horizon task execution capabilities. **Key Features:** * **Trillion-Scale Model**: \~1T total parameters with 63B activation parameters using a hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention) * **Generation Efficiency**: Reduces memory access overhead by over 10x and increases generation throughput by more than 3x for sequences exceeding 32K tokens * **Deep Reasoning**: Achieves gold medal level for both IMO 2025 and CMO 2025, with dense rewards for rigorous reasoning process feedback * **Long-horizon Task Execution**: Enhanced autonomous execution capability through large-scale fully-async agentic RL training * **Tool Calling**: Supports function calling with XML-style tool call format * **Context Length**: 128K -> 256K (YaRN) **Available Models:** * **FP8 (8-bit quantized)**: [inclusionAI/Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T) **License:** MIT ## 2. SGLang Installation Ring-2.5-1T runs on the standard SGLang Docker image: ```bash Command theme={null} # NVIDIA (H200 / B200 / GB200 / GB300) docker pull lmsysorg/sglang:latest # For MI300X/325X docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x # For MI355X docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x ``` For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform. ### 3.2 Configuration Tips * The `--trust-remote-code` flag is required for this model due to custom modeling code. * The model uses FP8 quantization (compressed-tensors format). ## 4. Model Invocation Deploy Ring-2.5-1T with the following command (on H200, all features enabled): ```shell Command theme={null} sglang serve \ --model-path inclusionAI/Ring-2.5-1T \ --tp 8 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser To enable reasoning output separation, add `--reasoning-parser deepseek-r1` when launching the server. The thinking process is returned via `reasoning_content` in the streaming response. ```shell Command theme={null} sglang serve \ --model-path inclusionAI/Ring-2.5-1T \ --tp 8 \ --trust-remote-code \ --reasoning-parser deepseek-r1 \ --host 0.0.0.0 \ --port 30000 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="inclusionAI/Ring-2.5-1T", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True ) for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: print(delta.reasoning_content, end="", flush=True) if delta.content: print(delta.content, end="", flush=True) print() ```
Output Example ```text Output theme={null} We are asked: "Solve this problem step by step: What is 15% of 240?" This is a straightforward percentage calculation. We need to show step-by-step solution. We can compute 15% of 240 as (15/100)*240 = 0.15 * 240 = 36. But we need to present step by step. Also ensure it's clear. We could also break down: 10% of 240 = 24, then 5% = 12, so 15% = 36. But any method is fine. We'll produce a solution with explanation: "To find 15% of 240, multiply 240 by 0.15 (or 15/100)." We'll show: 15% = 15/100 = 0.15 Then 0.15 × 240 = 36. Alternatively: (15/100) × 240 = (15 × 240) / 100 = 3600/100 = 36. Finally, answer: 36. We can also illustrate stepwise: "First, convert the percentage to a decimal: 15% = 0.15. Then multiply by the number: 0.15 × 240 = 36." We'll present as a final answer: \boxed{36}. However, we need to provide step-by-step solution as per instructions. We'll write a full explanation. We can also use the fraction method: 15% of 240 = (15/100)*240 = (15*240)/100 = 3600/100 = 36. Alr. I think that's it. **Step 1:** Write 15% as a fraction or decimal. \[ 15\% = \frac{15}{100} = 0.15\] **Step 2:** Multiply the number (240) by this fraction/decimal. \[ 240 \times 0.15 = 36\] Alternatively, using the fraction: \[ \frac{15}{100} \times 240 = \frac{15 \times 240}{100} = \frac{3600}{100} = 36\] **Conclusion:** 15% of 240 is 36. \[ \boxed{36} \] ```
#### 4.2.2 Tool Calling To enable tool calling, add `--tool-call-parser qwen` when launching the server. ```shell Command theme={null} sglang serve \ --model-path inclusionAI/Ring-2.5-1T \ --tp 8 \ --trust-remote-code \ --tool-call-parser qwen \ --host 0.0.0.0 \ --port 30000 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" } }, "required": ["location"] } } } ] response = client.chat.completions.create( model="inclusionAI/Ring-2.5-1T", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools ) print(response.choices[0].message.tool_calls) ``` **Output Example:** ```text Output theme={null} [ChatCompletionMessageFunctionToolCall(id='call_770360e31d194ed79d32cd8c', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)] ``` ## 5. Benchmark ### GSM8K * Deployment Command ```bash Command theme={null} sglang serve \ --model-path inclusionAI/Ring-2.5-1T \ --tp-size 8 \ --trust-remote-code ``` * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 \ --temperature 1.2 \ --top-p 0.8 \ --max-tokens 32768 \ --chat-template-kwarg enable_thinking=true ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. * Test Result ```text Output theme={null} Accuracy: 0.955 Invalid: 0.010 Latency: 615.833 s Output throughput: 412.360 token/s ``` # Ring-2.6-1T Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ring-2.6-1T ## 1. Model Introduction [Ring-2.6-1T](https://huggingface.co/inclusionAI/Ring-2.6-1T) is InclusionAI's trillion-parameter flagship reasoning model for real-world complex task execution. It targets agent workflows, engineering development, scientific research analysis, enterprise automation, and other long-horizon settings where the model must plan, use tools, recover from intermediate errors, and keep context across multiple steps. **Key Features:** * **Trillion-Scale Reasoning Model**: `BailingMoeV2_5ForCausalLM` with a `bailing_hybrid` architecture, 80 hidden layers, 256 routed experts, 8 selected experts per token, and FP8 compressed-tensors weights. * **Agent Execution**: Designed for multi-step task decomposition, tool collaboration, context continuation, and long-horizon execution. The model card reports 87.60 on PinchBench, 63.82 on ClawEval, and 95.32 on Tau2-Bench Telecom for the `high` setting. * **Reasoning Effort**: The model card describes `high` and `xhigh` reasoning-effort modes. In SGLang's OpenAI-compatible chat API, use top-level `reasoning_effort: "high"` for production agent workflows. To request the model-card `xhigh` prompt path, pass it through `chat_template_kwargs.reasoning_effort`. * **Hybrid Attention**: Uses the Bailing hybrid stack with MLA plus Lightning linear attention kernels in SGLang. * **Context Length**: Native 128K in the released config. Configure YaRN separately if you need a 256K deployment. **Available Models:** * **FP8 (E4M3 compressed-tensors)**: [inclusionAI/Ring-2.6-1T](https://huggingface.co/inclusionAI/Ring-2.6-1T) **License:** MIT ## 2. SGLang Installation Ring-2.6-1T requires recent SGLang builds with Bailing hybrid model support. Start with the latest SGLang Docker image when validating this cookbook: ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment Use the selector below to generate a single-node command for the tested hardware targets. ### Configuration Tips * `--trust-remote-code` is required for the model's custom Bailing hybrid implementation. * Use `--tp-size 4` on a single 4-GPU GB300 node. * Use `--tp-size 8` on a single 8-GPU B200 node. * Use `--tp-size 8` on a single 8-GPU H200 node. * Use `--mem-fraction-static 0.95` on GB300 x4. The model uses about 238.5GB/GPU after loading, so lower values can fail during KV-pool initialization. * Use `--mem-fraction-static 0.8` on B200 x8. * Use `--mem-fraction-static 0.95` on H200 x8. * `--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}'` is recommended because the model has 175 large safetensors shards. * Keep `--tool-call-parser glm` enabled by default for OpenAI-compatible tool calls. Ring's template emits XML `/` tool calls, which the `qwen` parser does not convert into `message.tool_calls`. * Keep `--reasoning-parser deepseek-r1` enabled by default so `...` content is split into `message.reasoning_content`. ## 4. Model Invocation ### 4.1 Basic Usage For example, launch the server on a single 4-GPU GB300 node: ```bash Command theme={null} export PORT=30000 sglang serve \ --model-path inclusionAI/Ring-2.6-1T \ --tp-size 4 \ --trust-remote-code \ --host 0.0.0.0 \ --port ${PORT} \ --mem-fraction-static 0.95 \ --model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}' \ --tool-call-parser glm \ --reasoning-parser deepseek-r1 ``` Send a basic chat request: ```bash Command theme={null} curl -s http://localhost:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}], "max_tokens": 128 }' ``` ### 4.2 Reasoning Effort Ring-2.6-1T exposes two reasoning-effort levels in the model card: `high` and `xhigh`. In SGLang's OpenAI-compatible chat API, start with top-level `reasoning_effort: "high"` for agent and production workflows: ```bash Command theme={null} curl -s http://localhost:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [{"role": "user", "content": "Solve: if 3x + 7 = 52, what is x?"}], "reasoning_effort": "high", "max_tokens": 512 }' ``` For the model-card `xhigh` path, pass the template value explicitly: ```bash Command theme={null} curl -s http://localhost:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [{"role": "user", "content": "Solve: if 3x + 7 = 52, what is x?"}], "chat_template_kwargs": {"reasoning_effort": "xhigh"}, "max_tokens": 512 }' ``` With the default deployment command, thinking text is separated into `message.reasoning_content` when the model emits `...` blocks. ### 4.3 Tool Calling Example ```bash Command theme={null} curl -s http://localhost:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [{"role": "user", "content": "What is the weather in Beijing?"}], "tools": [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"} }, "required": ["location"] } } }], "tool_choice": "auto", "max_tokens": 512 }' ``` For more API examples, see the [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request). ## 5. Benchmark ### 5.1 Speed Benchmark * Hardware: NVIDIA B200 GPU (8x), NVIDIA H200 GPU (8x), and NVIDIA GB300 GPU (4x) * Model: `inclusionAI/Ring-2.6-1T` * Docker image: `lmsysorg/sglang:latest` * SGLang version tested: `0.5.11` * Tensor Parallelism: 8 on B200 x8 and H200 x8, 4 on GB300 x4 Use the deployment command from [Section 3](#3-model-deployment), then confirm that the server is healthy before running benchmarks: ```bash Command theme={null} curl -s http://localhost:${PORT}/health curl -s http://localhost:${PORT}/v1/models ``` #### 5.1.1 Latency-Sensitive Benchmark * Test Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port ${PORT} \ --model inclusionAI/Ring-2.6-1T \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results (B200 x8): ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 207.18 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.05 Input token throughput (tok/s): 29.45 Output token throughput (tok/s): 20.37 Total token throughput (tok/s): 49.82 Mean E2E Latency (ms): 20715.16 Mean TTFT (ms): 187.86 Mean TPOT (ms): 44.65 Mean ITL (ms): 48.76 ================================================== ``` * Test Results (GB300 x4): ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 62.21 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.16 Input token throughput (tok/s): 98.07 Output token throughput (tok/s): 67.83 Total token throughput (tok/s): 165.91 Mean E2E Latency (ms): 6218.57 Mean TTFT (ms): 233.04 Mean TPOT (ms): 14.21 Mean ITL (ms): 14.22 ================================================== ``` * Test Results (H200 x8): ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 57.10 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.18 Input token throughput (tok/s): 106.85 Output token throughput (tok/s): 73.91 Total token throughput (tok/s): 180.76 Mean E2E Latency (ms): 5707.72 Mean TTFT (ms): 163.35 Mean TPOT (ms): 13.17 Mean ITL (ms): 13.17 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Test Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port ${PORT} \ --model inclusionAI/Ring-2.6-1T \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 100 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results (B200 x8): ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 100 Benchmark duration (s): 46.30 Total input tokens: 50561 Total generated tokens: 52444 Request throughput (req/s): 2.16 Input token throughput (tok/s): 1092.10 Output token throughput (tok/s): 1132.77 Total token throughput (tok/s): 2224.86 Mean E2E Latency (ms): 27581.74 Mean TTFT (ms): 1710.53 Mean TPOT (ms): 51.27 Mean ITL (ms): 49.43 ================================================== ``` * Test Results (GB300 x4): ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 100 Benchmark duration (s): 55.80 Total input tokens: 50561 Total generated tokens: 52444 Request throughput (req/s): 1.79 Input token throughput (tok/s): 906.10 Output token throughput (tok/s): 939.84 Total token throughput (tok/s): 1845.94 Mean E2E Latency (ms): 33736.85 Mean TTFT (ms): 2156.40 Mean TPOT (ms): 63.09 Mean ITL (ms): 60.33 ================================================== ``` * Test Results (H200 x8): ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 100 Benchmark duration (s): 44.51 Total input tokens: 50561 Total generated tokens: 52444 Request throughput (req/s): 2.25 Input token throughput (tok/s): 1135.88 Output token throughput (tok/s): 1178.18 Total token throughput (tok/s): 2314.06 Mean E2E Latency (ms): 27177.14 Mean TTFT (ms): 2173.08 Mean TPOT (ms): 51.11 Mean ITL (ms): 47.77 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * Benchmark Command: ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:${PORT}/v1 \ --model auto \ --num-examples 200 \ --num-threads 64 \ --max-tokens 2048 \ --reasoning-effort high ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * Test Results (B200 x8): ```text Output theme={null} Total latency: 100.378 s Score: 0.990 Output throughput: 627.401 token/s ``` * Test Results (GB300 x4): ```text Output theme={null} Total latency: 98.386 s Score: 0.990 Output throughput: 621.469 token/s ``` * Test Results (H200 x8): ```text Output theme={null} Total latency: 76.849 s Score: 0.990 Output throughput: 793.125 token/s ``` # Intern-S1 Source: https://docs.sglang.io/cookbook/autoregressive/InternLM/Intern-S1 ## 1. Model Introduction Intern-S1 includes the large **Intern-S1** MoE model and the smaller **Intern-S1-mini** dense model. The command generator below covers BF16 and FP8 serving on NVIDIA H100/H200/B200/B300 platforms. ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install), or install from source: ```bash Command theme={null} uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' ``` ## 3. Model Deployment ### 3.1 Basic Configuration ### 3.2 Configuration Tips * FP8 checkpoints use the matching BF16 checkpoint as tokenizer path. * B300 deployments use `--attention-backend flashinfer`. * Enable `--reasoning-parser interns1` and `--tool-call-parser interns1` when your workload needs structured reasoning or tool-call parsing. # Intern-S2-Mobius Source: https://docs.sglang.io/cookbook/autoregressive/InternLM/Intern-S2-Mobius Deploy the BF16 and FP8 Intern-S2-Mobius checkpoints with SGLang on NVIDIA H200 and B200. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` Intern-S2-Mobius was upstreamed in PR [#33691](https://github.com/sgl-project/sglang/pull/33691) (merged 2026-08-08) — it lives on `lmsysorg/sglang:dev` (nightly) until the next release cut. For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + recipe to generate the launch command. The two serving strategies cover the main operating points: * **Low-Latency** — MTP (NEXTN) speculative decoding on. Fastest reply for a single user. * **High-Throughput** — spec off, more tokens per second when many users share the server. Speed and accuracy numbers apply to the BF16 checkpoint. They were measured with `--random-range-ratio 1.0`, `--flush-cache`, on 2×H200 TP=2 against `main @ e0828ee3` + PR [#33691](https://github.com/sgl-project/sglang/pull/33691) head (since merged 2026-08-08 — `lmsysorg/sglang:dev` is the live equivalent). GSM8K is the full 1319-example test split; GPQA is Diamond 198 problems × 8 repeats (pass\@1 avg-of-8). Both ran with no server-side sampling override, so the checkpoint's `generation_config.json` defaults applied (temperature 1.0, top\_p 0.95, top\_k 20). The FP8 H200 recipes have launch, text, vision, and stop-reason verification but no benchmark numbers yet. All B200 recipes are inferred and unverified. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction **Intern-S2-Mobius** is InternLM's 35B scientific multimodal foundation model built on the **Mobius-v0** architecture (continually pre-trained from Qwen3.5-35B, then SFT and RL post-trained). Instead of binding knowledge storage and reasoning computation layer by layer as conventional Transformers do, Mobius organizes knowledge into a **globally shared Memory** that multiple **Reasoners** iteratively query against, yielding two native capabilities: * **Backward Residual Connection** — shallow and deep reasoning stages can reach knowledge across the model rather than relying only on forward layer-wise flow. * **Dynamic Latent Reasoning** — recurrent latent iteration refines hidden states before decoding, internalizing part of the deliberation process and shrinking visible chain-of-thought. The reported result is roughly a **4× end-to-end inference speedup** over the Qwen3.5-35B baseline while holding comparable scores on general reasoning benchmarks and improving on scientific tasks (Biology-Instructions, Mol-Instructions, MolecularIQ). On the serving side the model is a hybrid: 30 of 40 transformer layers use **GDN (Gated Delta Net) linear attention** (`kimi-linear`-family), with a **full-attention** layer every 4th layer (`full_attention_interval: 4` → **10 full-attention** layers), and the bottom of the stack is MoE-routed (2,560 routed experts × 512 intermediate, 8 active per token); a separate **MoE-256 / top-8 MTP (NEXTN) layer** feeds speculative decoding. It takes images via a vision tower and recognizes the standard `<|vision_start|>…<|vision_end|>` + `<|image_pad|>` markers. Context length is **262,144** tokens.
Variant Architecture Context License
Intern-S2-Mobius Mobius-v0 · GDN ×30 + full ×10 · MoE-2560 / top-8 · MTP · BF16 262,144 Apache-2.0
Intern-S2-Mobius-FP8 Mobius-v0 · GDN ×30 + full ×10 · MoE-2560 / top-8 · MTP · FP8 E4M3 262,144 Apache-2.0
**Recommended generation:** `temperature=0.8`, `top_p=1.0`, `top_k=50`, `min_p=0.0` — the values the [model card](https://huggingface.co/internlm/Intern-S2-Mobius) recommends. Both checkpoints ship `generation_config.json` with different values (`temperature=1.0`, `top_p=0.95`, `top_k=20`), and SGLang applies that file by default (`--sampling-defaults model`) — so send the recommended values explicitly per request if you want them. **Resources:** [HuggingFace (BF16)](https://huggingface.co/internlm/Intern-S2-Mobius) · [HuggingFace (FP8)](https://huggingface.co/internlm/Intern-S2-Mobius-FP8) · [GitHub (InternLM/Intern-S2-Mobius)](https://github.com/InternLM/Intern-S2-Mobius). ## 2. Configuration Tips * **Trust remote code is required.** Intern-S2-Mobius ships a custom `configuration_interns2_mobius.py` / `modeling_interns2_mobius.py` on its HF repo; every recipe adds `--trust-remote-code`. * **FP8 uses one H200.** The FP8 checkpoint stores E4M3 weights with 128×128 block scales and dynamic activation quantization. The verified H200 recipes use TP=1, `deep_gemm`, `--mem-fraction-static 0.6`, disabled prefill CUDA graphs, and decode graphs up to batch size 16. Higher memory fractions or the default graph range exhausted H200 memory during DeepGEMM warmup; the running MTP server used about 99 GB. * **Speculative decoding schedule.** The checkpoint ships one MTP layer. Enable MTP for the lowest latency (`--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`). We measured accept-length \~3.9/4 draft tokens at 8K-in / 1K-out, roughly tripling single-stream decode speed (median TPOT 9.79 ms → 3.13 ms at conc=1, 14.26 ms → 6.84 ms at conc=16) and roughly doubling mid-concurrency total throughput (9358 → 18029 tokens/s at conc=16, 21395 → 26033 tokens/s at conc=64). The high-throughput recipe stays spec-off because once you can batch wide, its saturation point is higher (34786 tokens/s at conc=256 vs the spec recipe's peak at conc=64). * **Mamba pool sizing.** GDN layers live in a separate Mamba state pool; the `--mamba-full-memory-ratio` (defaults to 0.9) controls the split between the 10 full-attention layers' KV pool and the 30 GDN layers' conv+SSM state pool. Default split comfortably handles conc=64 on a 2×H200 node; if you need higher concurrency than `--max-running-requests` allows for your workload, raise `--mamba-full-memory-ratio` slightly (each +1% mamba ratio costs full-attn KV). * **Vision input.** Images are accepted via the standard `image_url` chat message type. Vision tokens are counted into the prompt (`prompt_tokens_details.image_tokens` shows the count), and the model honors `<|vision_start|>` / `<|vision_end|>` boundaries exactly. * **B200 sizing.** B200 (192 GB HBM) fits the BF16 weights + KV + Mamba pool on a **single GPU** with `--tp 1`. The FP8 B200 cells inherit the verified H200 TP=1 recipe but remain unverified; treat them as a starting point until a Blackwell measurement is available. ## 3. Advanced Usage The outputs below are verbatim captures from a live server (sampling per the checkpoint's `generation_config.json`, temperature 1.0). Re-running the same request yields a semantically equivalent but textually different trace — treat them as representative, not deterministic. ### 3.1 Reasoning InternS2-Mobius is a hybrid-reasoning model — thinking traces start with "Thinking Process:" before the final answer. Enable the **`qwen3` reasoning parser** (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to split thinking into `message.reasoning_content` and the answer into `message.content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="internlm/Intern-S2-Mobius", messages=[{"role": "user", "content": "What is 15% of 240?"}], ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Reasoning: Thinking Process: 1. **Identify the core question:** The user is asking for 15% of 240. 2. **Determine the calculation method:** To find a percentage of a number, multiply the number by the percentage expressed as a decimal or fraction. * Percentage: 15% * Decimal: 0.15 * Fraction: 15/100 3. **Perform the calculation:** $240 \times 0.15$ * Method 1: $240 \times 0.10 = 24$ (10%) and $240 \times 0.05 = 12$ (5%). Then add them: $24 + 12 = 36$. * Method 2: $240 \times 15 = 3600$. Divide by 100 -> 36. 4. **Verify the result:** The calculation is correct. 5. **Formulate the answer:** State the final number clearly.cw Answer: 15% of 240 is **36**. Here is the math: $240 \times 0.15 = 36$ ``` ### 3.2 Tool Calling Enable the **`qwen3_coder` tool-call parser** (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Intern-S2-Mobius emits `………value……` — this is exactly the format `qwen3_coder` parses; without the parser the call is left as raw text in `content`. On this thinking-mode model the turn also fills `reasoning_content`, so print both fields. **Auto-resolution works out of the box.** Intern-S2-Mobius's chat template contains the ` ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, }, "required": ["location"], }, }, }] resp = client.chat.completions.create( model="internlm/Intern-S2-Mobius", messages=[{"role": "user", "content": "What is the weather in Beijing?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Reasoning: The user is asking for the weather in Beijing. I have access to a get_weather function that can get the current weather for a location. The function requires a "location" parameter which should be the city name. In this case, the user specified "Beijing", so I should use that as the location parameter. Tool calls: [ { "id": "call_545b5956b4c3457286261490", "index": 0, "type": "function", "function": {"name": "get_weather", "arguments": "{\"location\": \"Beijing\"}"} } ] finish_reason: tool_calls ``` ### 3.3 Vision Input Intern-S2-Mobius takes images via the OpenAI-compatible `image_url` content type. Vision input works with the same server the Deploy panel produces — no extra model-specific flags needed. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="internlm/Intern-S2-Mobius", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg"}}, {"type": "text", "text": "Describe this image in one sentence."}, ], }], ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Reasoning: The user wants a one-sentence description of the image. Key elements: Tiger, lying down, grass, looking at camera. Drafting: A tiger is lying in the green grass looking directly at the camera. Refining for flow and detail: A majestic tiger with striking orange and black stripes rests calmly on a bed of lush green grass, staring intently directly at the viewer. Answer: A majestic tiger with striking orange and black stripes rests calmly on a bed of lush green grass, staring intently directly at the viewer. ``` # Intern-S2-Preview Source: https://docs.sglang.io/cookbook/autoregressive/InternLM/Intern-S2-Preview ## 1. Model Introduction **Intern-S2-Preview** is an efficient 35B scientific multimodal foundation model. Beyond conventional parameter and data scaling, Intern-S2-Preview explores task scaling: increasing the difficulty, diversity, and coverage of scientific tasks to further unlock model capabilities. **Resources:** * HuggingFace: [internLM/Intern-S2-Preview](https://huggingface.co/internLM/Intern-S2-Preview) ## 2. SGLang Installation SGLang offers multiple installation methods. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Install SGLang from source or use an NVIDIA Docker image: ```bash Command theme={null} # Install from source uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' # Or use Docker for NVIDIA GPUs docker pull lmsysorg/sglang:latest ``` For how to actually launch a docker image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the [command generator](#3-model-deployment) below produces): ```bash Command theme={null} docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:latest \ sglang serve ``` ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the selector below to generate the deployment command for your hardware and parser configuration. ### 3.2 Configuration Tips * Use `tp>=2` for the NVIDIA deployment commands. * Use `--reasoning-parser qwen3` to separate reasoning content from final content in streaming responses. * Use `--tool-call-parser qwen3_coder` when serving tool-calling workloads. * Add `--mamba-radix-cache-strategy extra_buffer` with `--speculative-algo 'NEXTN'` to enable MTP. * If weight loading is slow, add `--model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 64}'`. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, see: * [Basic API Usage](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Vision Input Intern-S2-Preview supports image inputs. Here is an example with an image: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="internLM/Intern-S2-Preview", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg" }, }, { "type": "text", "text": "Describe this image in detail.", }, ], } ], max_tokens=2048, stream=True, ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, "reasoning_content") and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` #### 4.2.2 Reasoning Parser Enable streaming to read reasoning content separately from the final answer: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="internLM/Intern-S2-Preview", messages=[ {"role": "user", "content": "Solve this step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True, ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, "reasoning_content") and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` #### 4.2.3 Tool Calling Serve with `--tool-call-parser qwen3_coder` enabled, then send OpenAI-compatible tool requests: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name", } }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="internLM/Intern-S2-Preview", messages=[{"role": "user", "content": "What is the weather in Beijing?"}], tools=tools, max_tokens=1024, ) print(response.choices[0].message) ``` # InternVL3.5 Source: https://docs.sglang.io/cookbook/autoregressive/InternVL/InternVL3.5 ## 📝 Community Contribution Welcome This guide is currently under development. We welcome community contributions! If you have experience deploying **InternVL3.5** with SGLang, please help us complete this documentation. ## 🚀 How to Contribute ```shell Command theme={null} git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git cd sglang-cookbook git checkout -b add-internvl3-5-guide # Edit this file and submit a PR ``` ## 📚 Reference * [GLM-4.6V](../GLM/GLM-4.6V) *** **Let's build this together!** 🌟 # Jina-reranker-m0 Source: https://docs.sglang.io/cookbook/autoregressive/Jina/Jina-reranker-m0 ## 📝 Community Contribution Welcome This guide is currently under development. We welcome community contributions! If you have experience deploying **Jina-reranker-m0** with SGLang, please help us complete this documentation. ## 🚀 How to Contribute ```shell Command theme={null} git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git cd sglang-cookbook git checkout -b add-jina-reranker-m0-guide # Edit this file and submit a PR ``` ## 📚 Reference * [DeepSeek-V3.2](../DeepSeek/DeepSeek-V3_2.md) *** **Let's build this together!** 🌟 # LFM2.5 Source: https://docs.sglang.io/cookbook/autoregressive/LiquidAI/LFM2.5 Deploy Liquid AI's LFM2.5 with SGLang — hybrid gated short conv + GQA models from 350M to the 8B-A1B MoE, plus LFM2.5-VL vision, with reasoning and Pythonic tool calling. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` LFM2.5 support — the dense / MoE / VL model classes and the `lfm2` tool-call parser — ships on SGLang `main`. If your installed release predates it, install from source or use the Docker dev image. Then run the **Python** output of the command panel below in that environment. LFM2.5 support ships in the pinned SGLang dev image: ```bash Command theme={null} docker pull lmsysorg/sglang:dev-cu13 ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces): ```bash Command theme={null} docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:dev-cu13 \ sglang serve ``` Every LFM2.5 model runs on a **single GPU (TP=1)** — pick your hardware + model variant to generate the launch command. One recipe covers all operating points per variant; the commands differ only by the parsers a model needs and, on Blackwell, the attention backend. The `lfm2` tool-call parser and each reasoning model's `--reasoning-parser` are already part of the verified command.

Panel controls (top of the command box):

  • Python / Docker — bare sglang serve … for an existing SGLang env, or a docker run … sglang serve … wrap against the dev image from the Install SGLang panel above.
  • ⧉ Copy — copies the current command (with whichever framing is active) to your clipboard.
  • \$ cURL — a sample request against localhost:30000 to confirm the server is up.
  • ⚙ Env — edits the placeholders (HOST\_IP, PORT, HF\_TOKEN) the command and cURL share. Persists in localStorage across cookbooks.
  • Verified / Not Verified badge — green when the (hw, variant, quant, strategy, nodes) combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.
## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change. For LFM2.5 the exposed knob is the **TP override** (every variant is verified at TP=1; TP=2 is available for experimentation on the larger checkpoints). The reasoning and tool-call parsers are not playground toggles here — they are variant-intrinsic and already baked into each verified command. Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.

Panel controls reuse Python / Docker · ⧉ Copy · \$ cURL · ⚙ Env from the Deploy panel, plus one extra:

  • Submit ↗ — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says Not Verified; click it once you've actually run the command on your hardware and confirmed it works.
## 1. Model Introduction LFM2.5 is [Liquid AI](https://www.liquid.ai/)'s family of hybrid models for on-device deployment, released under the [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/LICENSE). It builds on the LFM2 architecture with extended pre-training — 10T → 28T tokens for the dense models, 12T → 38T for the 8B-A1B MoE — and large-scale reinforcement learning. The backbone interleaves **gated short convolution blocks** with a small minority of **grouped query attention (GQA) blocks**. Each convolution block applies input-dependent multiplicative gating around a depthwise short convolution, giving fast local mixing at low compute and memory cost. The GQA blocks handle global context and long-range retrieval. This minimal hybrid layout was selected by a hardware-in-the-loop architecture search under edge latency and memory budgets. On CPUs it delivers up to 2× faster prefill and decode than similarly sized models (see the [LFM2 Technical Report](https://arxiv.org/abs/2511.23404)). **Key Features:** * **Hybrid gated short conv + GQA layout**: the 1.2B / 350M dense models are 16 layers (10 conv + 6 GQA); the 8B-A1B MoE is 24 layers (18 conv + 6 GQA). With only 6 attention layers per model, the KV cache stays small even at long context. * **Block details**: depthwise convolutions with kernel size 3; GQA with 8 KV groups and head size 64, plus RoPE and QK-Norm; pre-norm RMSNorm and SwiGLU MLPs throughout. * **Sparse MoE (8B-A1B)**: 8.3B total / 1.5B active parameters. Every layer except the first two replaces its dense MLP with a 32-expert MoE block; each token is routed to the top-4 SwiGLU experts by a normalized sigmoid router with adaptive bias load balancing. * **New in 2.5 (8B-A1B)**: the blocks are unchanged from LFM2-8B-A1B, but the context window grows from 32K to 128K (a RoPE base-θ increase plus long-context midtraining) and the vocabulary doubles from 65,536 to 128,000 tokens for more efficient non-Latin tokenization. * **Pythonic tool calling**: function calls are emitted as a Python list between `<|tool_call_start|>` and `<|tool_call_end|>` tokens. The `lfm2` tool-call parser surfaces these as standard `message.tool_calls`. * **Reasoning variants**: the 8B-A1B and 1.2B-Thinking checkpoints are reasoning-only models that always emit an explicit `...` chain-of-thought before the answer. The MoE's 1.5B active parameters keep those reasoning tokens cheap. * **Multilingual**: every model except the JP checkpoints covers at least English, Arabic, Chinese, French, German, Japanese, Korean, and Spanish (some variants add more). The dedicated JP chat checkpoints focus on Japanese (Japanese + English only). * **Vision**: LFM2.5-VL-1.6B pairs the 1.2B language backbone with a SigLIP2 So400M NaFlex encoder for OCR, document understanding, and multilingual vision. LFM2.5-VL-450M pairs the 350M backbone with a SigLIP2 Base-86M encoder for captioning and object detection at edge sizes; bounding-box grounding and function calling are new in the 2.5 release. **Available Models:**
Model Parameters Context Role
LFM2.5-8B-A1B 8.3B total / 1.5B active (MoE) 128K Reasoning-tuned, agentic / tool use
LFM2.5-1.2B-Instruct 1.17B (dense) 32K General instruct, RAG, data extraction
LFM2.5-1.2B-Thinking 1.17B (dense) 32K Reasoning (always-on chain-of-thought)
LFM2.5-350M 350M (dense) 32K Compact instruct, structured output
LFM2.5-230M 230M (dense) 32K Most compact; data extraction, structured output
LFM2.5-1.2B-JP-202606 1.17B (dense) 32K Japanese chat (latest)
LFM2.5-1.2B-JP 1.17B (dense) 32K Japanese chat (original)
LFM2.5-VL-1.6B 1.2B LM + SigLIP2 400M 32K Vision-language (OCR, docs, multi-image)
LFM2.5-VL-450M 350M LM + SigLIP2 86M 32K Compact vision-language (captioning, object detection)
LFM2.5-1.2B-Base 1.17B (dense) 32K Pre-trained base (no post-training)
The Deploy panel above covers the eight serving variants; **LFM2.5-1.2B-JP** (original — launch without `--tool-call-parser`) and the **Base** repos (pre-trained only, no post-training — see [§3.5](#3-5-base-checkpoints)) launch the same way with the model path swapped. **Choosing a variant:** * **8B-A1B** — flagship for agentic and tool-calling workloads; the only 128K-context option. * **1.2B-Thinking** — reasoning-heavy tasks: math, tool use, programming. * **1.2B-Instruct** — the recommended pick for chat and creative writing. * **350M** — tool use, data extraction, and structured output; not recommended for math, code, or creative writing. * **230M** — the most compact checkpoint; same use as the 350M, not for math, code, or creative writing. **License:** [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/LICENSE). **Resources:** [LFM2.5 announcement](https://www.liquid.ai/blog/introducing-lfm2-5-the-next-generation-of-on-device-ai), [LFM2.5-8B-A1B blog](https://www.liquid.ai/blog/lfm2-5-8b-a1b), [LFM docs](https://docs.liquid.ai/lfm/getting-started/welcome), [LFM2 Technical Report (arXiv:2511.23404)](https://arxiv.org/abs/2511.23404). ## 2. Configuration Tips * **Reasoning parser**: LFM2.5 reasoning models wrap their chain-of-thought in `...` tags. The command generator passes `--reasoning-parser qwen3` for **8B-A1B** (it emits an explicit opening ``) and `--reasoning-parser qwen3-thinking` for **1.2B-Thinking** (always-on reasoning). This splits the thinking process into `reasoning_content`; without it the chain-of-thought stays inline in `content`. * **Tool calling**: `--tool-call-parser lfm2` surfaces LFM2.5's Pythonic `<|tool_call_start|>[...]<|tool_call_end|>` calls as standard `message.tool_calls`. The original **1.2B-JP** does not expose tool calling; **Base** has no post-training (see [§3.5](#3-5-base-checkpoints)). * **Attention backend on Blackwell (B200/sm100)**: SGLang defaults to the `trtllm_mha` backend on sm100, which is fastest for the dense text models. The **8B-A1B** uses a mamba-style state cache that runs on a page-size-1 backend, so the generator picks `--attention-backend flashinfer` for it. The **VL** language model also uses that state cache and offers two backends: `--attention-backend flashinfer` (keeps prefix/radix caching — what the generator emits), or `--attention-backend trtllm_mha --disable-radix-cache` to run the language model on Blackwell `trtllm_mha` attention (`--disable-radix-cache` lifts the page-size-1 requirement, at the cost of prefix caching). Pair either with `--mm-attention-backend fa4` for the vision tower. * **VL vision tower (`--mm-attention-backend`)**: on sm100 the `trtllm_mha` default is fastest for text but applies *causal* attention to image tokens. For the VL model, pass `--mm-attention-backend fa4` on B200/B300 (or `fa3` on H100/H200) to restore bidirectional image-token attention and full vision quality. * **VL multimodal feature transport**: the generator launches the VL models with `SGLANG_USE_CUDA_IPC_TRANSPORT=1 SGLANG_USE_IPC_POOL_HANDLE_CACHE=1`. The first moves the processor→scheduler image-feature handoff onto CUDA IPC instead of serializing tensors between processes; the second ships the pool handle so the scheduler opens it once and caches it, instead of opening a per-item handle on every request. On the image serving workload (1 image @ 720p, measured on VL-1.6B on H100 and B200) this pair is worth roughly 30–50% higher image throughput and 30–40% lower image TTFT vs running without them (measured on VL-1.6B, H100 and B200); decode speed (TPOT) is unaffected. * **VL-450M memory headroom (`--mem-fraction-static 0.8`)**: with the default memory fraction, the 450M's small weights make SGLang size its static KV/mamba pools to nearly the whole GPU, leaving no headroom for image-feature tensors — under sustained concurrent image load the scheduler can crash with a CUDA OOM in the radix-cache free path. The generator caps `--mem-fraction-static 0.8` for VL-450M; the pool is still far larger than this model ever needs. * **Mamba scheduling**: LFM2.5 runs on the default `no_buffer` mamba scheduler strategy — no `--mamba-radix-cache-strategy` flag is needed. The `extra_buffer` strategy (an overlap-scheduling throughput optimization available for some Gated-DeltaNet hybrids) does not apply to LFM2.5, whose convolution blocks use `mamba_chunk_size=1`. * **Hardware requirements**: all LFM2.5 models run on a single GPU (TP=1) on either Hopper or Blackwell. The 1.2B / 350M dense models fit in a few GB; the 8B-A1B MoE needs roughly 16 GB for bf16 weights plus KV cache. Multi-GPU tensor parallelism is not required for any variant. **Recommended sampling parameters** — pass these explicitly on every request. Some LFM2.5 checkpoints do not ship sampling defaults in `generation_config.json`, so the server will not apply them for you. `top_k`, `min_p`, and `repetition_penalty` are not standard OpenAI `chat.completions` fields — pass them through **`extra_body`** and SGLang forwards them to its sampler. Do not set `max_tokens` unless you intend to cap output, as it can truncate a response (or a reasoning model's chain-of-thought) mid-stream.
Model temperature extra\_body (sampler)
LFM2.5-8B-A1B0.2
LFM2.5-1.2B-Instruct0.1
LFM2.5-1.2B-Thinking0.05
LFM2.5-350M0.1
LFM2.5-230M0.1
LFM2.5-1.2B-JP-2026060.1
LFM2.5-1.2B-JP0.3
LFM2.5-VL-1.6B (text)0.1
LFM2.5-VL-450M (text)0.1
LFM2.5-1.2B-Base0.3
## 3. Advanced Usage ### 3.1 Basic Usage A single client with the recommended sampling presets applied per model (the examples in the following sections reuse this `chat` helper): ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") # Non-OpenAI fields (top_k / min_p / repetition_penalty) ride in extra_body. SAMPLING = { "LiquidAI/LFM2.5-8B-A1B": dict(temperature=0.2, extra_body={"top_k": 80, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-1.2B-Instruct": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-1.2B-Thinking": dict(temperature=0.05, extra_body={"top_k": 50, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-350M": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-230M": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-1.2B-JP-202606": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-VL-1.6B": dict(temperature=0.1, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-VL-450M": dict(temperature=0.1, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}), } def chat(model, messages, **overrides): cfg = SAMPLING[model] body = cfg["extra_body"] | overrides.pop("extra_body", {}) return client.chat.completions.create( model=model, messages=messages, temperature=cfg["temperature"], extra_body=body, **overrides, ) resp = chat( "LiquidAI/LFM2.5-1.2B-Instruct", [{"role": "user", "content": "What is C. elegans? Answer in one sentence."}], ) print(resp.choices[0].message.content) ``` ### 3.2 Reasoning The 8B-A1B and 1.2B-Thinking checkpoints emit chain-of-thought as a built-in behavior. The Deploy panel launches them with the matching `--reasoning-parser`, which separates the thinking process into `reasoning_content`: ```python Example theme={null} resp = chat( "LiquidAI/LFM2.5-8B-A1B", [{"role": "user", "content": "If a train travels 60 km/h for 2.5 hours, how far does it go?"}], ) msg = resp.choices[0].message print("Reasoning:", msg.reasoning_content) print("Answer:", msg.content) ``` ### 3.3 Tool Calling LFM2.5 writes Pythonic tool calls. With `--tool-call-parser lfm2` (already part of the launch command) they are surfaced as standard `message.tool_calls`: ```python Example theme={null} resp = chat( "LiquidAI/LFM2.5-1.2B-Instruct", [{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get current weather for a location", "parameters": { "type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"], }, }, }], ) for call in resp.choices[0].message.tool_calls or []: print(call.function.name, call.function.arguments) ``` Tool calling is supported on 8B-A1B, 1.2B-Thinking, 1.2B-Instruct, 350M, 230M, 1.2B-JP-202606, VL-1.6B, and VL-450M. For the **VL** models it is text-turn-only — do not combine an image and tools in the same turn. ### 3.4 Vision Input The VL models (VL-1.6B and VL-450M) accept images via standard OpenAI multimodal content blocks. Base64 data URIs (`data:image/jpeg;base64,...`) work in place of a URL: ```python Example theme={null} resp = chat( "LiquidAI/LFM2.5-VL-1.6B", [{ "role": "user", "content": [ {"type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"}}, {"type": "text", "text": "What is in this image?"}, ], }], ) print(resp.choices[0].message.content) ``` ### 3.5 Base Checkpoints Each size ships a pre-trained Base repo — [LFM2.5-230M-Base](https://huggingface.co/LiquidAI/LFM2.5-230M-Base), [LFM2.5-1.2B-Base](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base), [LFM2.5-350M-Base](https://huggingface.co/LiquidAI/LFM2.5-350M-Base), and [LFM2.5-8B-A1B-Base](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-Base) — intended for fine-tuning and continued pre-training. The repos ship a ChatML-style chat template, so `chat.completions` requests format normally. The checkpoints have no post-training, though — don't expect instruction following. For raw text continuation: ```python Example theme={null} comp = client.completions.create( model="LiquidAI/LFM2.5-1.2B-Base", prompt="The capital of France is", temperature=0.3, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}, ) print(comp.choices[0].text) ``` # LongCat-2.0 Source: https://docs.sglang.io/cookbook/autoregressive/Meituan/LongCat-2.0 Deploy LongCat-2.0-FP8 with SGLang - config-driven recipes for Meituan's 1.6T sparse MoE model on B300, B200, H200, and H20 GPUs. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). LongCat-2.0 support is on SGLang `main`; use a nightly wheel or rolling nightly Docker image until the next tagged release includes it. The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv # Choose the nightly wheel index for your CUDA runtime. SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu130 # B300 / CUDA 13 # SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu129 # CUDA 12.9 uv pip install --prerelease=allow --extra-index-url "${SGLANG_WHL_INDEX}" "sglang[all]" ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} # Choose the rolling nightly image for your hardware. SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev-cu13 # B300 / CUDA 13 # SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev # Other supported hardware docker pull "${SGLANG_DOCKER_IMAGE}" ``` For how to launch the image, see [Install -> Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + recipe to generate the launch command. LongCat-2.0 currently exposes one model-card-aligned serving strategy: * **Balanced** - the validated B300 recipe and the 2-node H200/B200/H20 topology use TP/EP parallelism with LongCat sparse attention prefill. All recipes here run the LongCat sparse-attention indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on LongCat-2.0. The B300 single-node recipe was validated end-to-end with CUDA graph capture enabled. H200, B200, and H20 are shown as 2-node recipes because LongCat-2.0-FP8 needs 16 ranks for those GPU memory profiles. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction [LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8) is the FP8 checkpoint of Meituan LongCat-2.0, a large sparse Mixture-of-Experts language model with 1.6T total parameters and about 48B activated parameters per token. It combines LongCat Sparse Attention (LSA), expert parallel MoE layers, and an n-gram/token-table embedding path for serving long-context workloads efficiently.
Model Architecture Serving precision
LongCat-2.0-FP8 Sparse MoE · LongCat Sparse Attention · n-gram embedding FP8 weights, BF16 KV cache
**Resources:** [LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8). ## 2. Configuration Tips * **Remote code.** Use `--trust-remote-code` for the Hugging Face checkpoint. * **Topology.** The 8x B300 recipe uses TP=8 and EP=8. H200, B200, and H20 use a 2-node 16 GPU layout with TP=16 and EP=16; the command panel injects the multi-node rank flags for you. * **LongCat sparse attention.** Keep `--dsa-prefill-backend fa3` with `--chunked-prefill-size 2048` for the model-card-aligned prefill path. * **Memory.** The recipe uses `--kv-cache-dtype bfloat16` and starts at `--mem-fraction-static 0.92`. Tune memory only after the generated command launches cleanly on your cluster. * **Weight loading.** `--model-loader-extra-config '{"enable_multithread_load":true,"num_threads":12}'` loads checkpoint shards in parallel and reduces startup time. * **FP8 backend selection.** Do not pass `--fp8-gemm-runner-backend` manually. SGLang selects the correct backend for the LongCat FP8 scale layout. * **Host, port, and ranks.** Use the command panel environment fields for `HOST_IP`, `PORT`, `NODE0_IP`, and `NODE_RANK` instead of hardcoding them in the recipe. ## 3. Advanced Usage ### 3.1 Test the deployment ```bash Command theme={null} curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meituan-longcat/LongCat-2.0-FP8", "messages": [ {"role": "user", "content": "A shop has 17 apples and sells 8. Then it buys 6 more. How many apples are there? Answer with only the final number."} ], "max_tokens": 32, "chat_template_kwargs": {"enable_thinking": false} }' ``` ```text Output theme={null} 15 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="meituan-longcat/LongCat-2.0-FP8", messages=[ { "role": "user", "content": "Solve: A shop has 17 apples and sells 8, then buys 6 more. Answer with only the final number.", } ], max_tokens=32, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print(response.choices[0].message.content) ``` ```text Output theme={null} 15 ``` ## 4. Validation The B300 recipe was validated with `meituan-longcat/LongCat-2.0-FP8` on 8x B300 using the command generated above.
Evaluation Examples Accuracy
GSM8K 200 98.0%
GSM8K 1314 95.8904109589041%
CUDA graph was enabled, and decode CUDA graph capture completed successfully during serving validation. # Llama-3.1 Source: https://docs.sglang.io/cookbook/autoregressive/Meta/Llama3.1 ## 1. Model Introduction Llama 3.1 is a collection of pretrained and instruction tuned generative models, released in July 2024 by Meta. These models are available in 8B, 70B and 405B sizes, with the 405B variant being the most capable fully-open source model at the time. These models bring open intelligence to all, with several new features and improvements: * **Stronger General Intelligence**: These models showcase significant improvements in coding, state-of-the-art tool use, and overall stronger reasoning capabilities. * **Extended Context Length**: Llama 3.1 extends the context length to 128K tokens to improve performance over long context tasks such as summarization and code reasoning. * **Tool Use**: Llama 3.1 is trained to interact with a search engine, python interpreter and mathematical engine, and also improves zero-shot tool use capabilities to interact with potentially unseen tools. * **Multilinguality**: Llama 3.1 supports 7 languages in addition to English: French, German, Hindi, Italian, Portuguese, Spanish, and Thai. For further details, please refer to the [Llama 3.1 blog](https://ai.meta.com/blog/meta-llama-3-1/) and the [Llama 3.1 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md).note ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to generate a launch command for Llama 3.1 collection of models. ### 3.2 Configuration Tips **Speculative Decoding (NVIDIA GPUs):** * Using Speculative Decoding for latency-sensitive scenarios: * `--speculative-algorithm EAGLE3`: Speculative decoding algorithm * `--speculative-num-steps 3`: Number of speculative verification rounds * `--speculative-eagle-topk 1`: Top-k sampling for draft tokens * `--speculative-num-draft-tokens 4`: Number of draft tokens per step * `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`yuhuili/EAGLE3-LLaMA3.1-Instruct-8B`](https://huggingface.co/yuhuili/EAGLE3-LLaMA3.1-Instruct-8B). **AMD GPU Deployment:** * **Hardware-Aware TP**: MI355X (256GB memory) supports lower TP values compared to MI300X/MI325X (192GB) * **Verified TP Configurations**: * MI300X/MI325X: 405B BF16 (TP=8), 405B FP8 (TP=4), 70B/8B (TP=1) * MI355X: 405B BF16 (TP=4), 405B FP8 (TP=2), 70B/8B (TP=1) * **FP8 Model Variants**: * 405B: Use Meta's official `meta-llama/Llama-3.1-405B-Instruct-FP8` * 70B/8B: Use AMD's optimized `amd/Llama-3.1-{size}-Instruct-FP8-KV` * **Tool Calling**: Enable with `--tool-call-parser llama3` for Instruct models **Xeon CPU Deployment:** * Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage SGLang exposes an OpenAI-compatible endpoint. First, start the server ```shell Command theme={null} sglang serve \ --model-path Meta-Llama/Llama-3.1-405B-Instruct \ --tp 8 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="Meta-Llama/Llama-3.1-405B-Instruct", messages=[ {"role": "system", "content": "You are a helpful coding assistant."}, {"role": "user", "content": "Write a Python function that retries a request with exponential backoff."}, ], temperature=0.2, max_tokens=512, ) print(resp.choices[0].message.content) ``` **Output Example:** ````text Output theme={null} **Exponential Backoff Retry Function in Python** ===================================================== Below is a Python function that uses the `requests` library to retry a request with exponential backoff. ```python import requests import time import random def exponential_backoff_retry(url, method, retries=3, backoff_factor=1, max_delay=60): """ Retry a request with exponential backoff. Args: url (str): The URL to make the request to. method (str): The HTTP method to use (e.g. 'GET', 'POST', etc.). retries (int): The number of retries to attempt. Defaults to 3. backoff_factor (int): The factor to multiply the delay by for each retry. Defaults to 1. max_delay (int): The maximum delay to wait between retries in seconds. Defaults to 60. Returns: The response object from the successful request. """ delay = 1 for attempt in range(retries + 1): try: response = requests.request(method, url) response.raise_for_status() # Raise an exception for HTTP errors return response except requests.RequestException as e: if attempt < retries: # Calculate the delay for this retry delay = min(delay * backoff_factor, max_delay) # Add a random jitter to the delay to prevent thundering herd problem delay += random.uniform(0, delay * 0.1) # Wait for the calculated delay before retrying time.sleep(delay) else: # If all retries have failed, raise the exception raise e ... ```` ### 4.2 Advanced Usage #### 4.2.1 Tool Calling Llama3 supports tool calling capabilities. First, start the server with tool call parser enabled: ```shell Command theme={null} sglang serve \ --model-path Meta-Llama/Llama-3.1-405B-Instruct \ --tool-call-parser llama3 \ --tp 8 ``` **Python Example** ```python Example theme={null} from openai import OpenAI client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:8000/v1") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the weather in a given location", "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": "The city to find the weather for, e.g. 'San Francisco'", }, "unit": { "type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"], }, }, "required": ["city", "unit"], }, }, } ] response = client.chat.completions.create( model="meta-llama/Llama-3.1-405B-Instruct", messages=[ { "role": "user", "content": "What's the weather like in Boston today?", } ], temperature=0.7, stream=True, tools=tools, ) arguments = [] tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'tool_calls') and delta.tool_calls: for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` Reference: [SGLang Tool Parser Documentation](../../../docs/advanced_features/tool_parser#openai-compatible-api) **Output Example** ```text Output theme={null} 🔧 Tool Call: get_weather Arguments: {"city": "Boston", "unit": "fahrenheit"} ``` **Handling Tool Call Results** After getting the tool call, you can execute the function: ```python Example theme={null} def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather like in Boston today?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Boston", "unit": "fahrenheit"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Boston", "fahrenheit") } ] final_response = client.chat.completions.create( model="Meta-Llama/Llama-3.1-405B-Instruct", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The current weather in Boston is **22°C** and **sunny**. A perfect day to spend outside" ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA A100 GPU (8x) * Model: Meta-Llama/Llama-3.1-70B * Tensor Parallelism: 8 * sglang version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. #### 5.1.1 Standard Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path Meta-Llama/Llama-3.1-70B \ --tp 8 ``` ##### 5.1.1.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} sglang serve \ --backend sglang \ --model Meta-Llama/Llama-3.1-70B \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 79.81 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4208 Request throughput (req/s): 0.13 Input token throughput (tok/s): 76.44 Output token throughput (tok/s): 52.88 Peak output token throughput (tok/s): 54.00 Peak concurrent requests: 2 Total token throughput (tok/s): 129.32 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7977.81 Median E2E Latency (ms): 6373.48 ---------------Time to First Token---------------- Mean TTFT (ms): 131.61 Median TTFT (ms): 131.77 P99 TTFT (ms): 163.88 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 18.63 Median TPOT (ms): 18.63 P99 TPOT (ms): 18.65 ---------------Inter-Token Latency---------------- Mean ITL (ms): 18.64 Median ITL (ms): 18.64 P95 ITL (ms): 18.69 P99 ITL (ms): 18.74 Max ITL (ms): 21.95 ================================================== ``` ##### 5.1.1.2 Medium Concurrency ```shell Command theme={null} sglang serve \ --backend sglang \ --model-path Meta-Llama/Llama-3.1-70B \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 79.47 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 38450 Request throughput (req/s): 1.01 Input token throughput (tok/s): 499.17 Output token throughput (tok/s): 513.48 Peak output token throughput (tok/s): 674.00 Peak concurrent requests: 20 Total token throughput (tok/s): 1012.65 Concurrency: 13.47 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 13376.67 Median E2E Latency (ms): 14130.48 ---------------Time to First Token---------------- Mean TTFT (ms): 264.84 Median TTFT (ms): 147.02 P99 TTFT (ms): 791.93 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 26.09 Median TPOT (ms): 26.08 P99 TPOT (ms): 34.65 ---------------Inter-Token Latency---------------- Mean ITL (ms): 25.76 Median ITL (ms): 23.95 P95 ITL (ms): 24.72 P99 ITL (ms): 98.32 Max ITL (ms): 478.92 ================================================== ``` ##### 5.1.1.3 High Concurrency ```shell Command theme={null} sglang serve \ --backend sglang \ --model-path Meta-Llama/Llama-3.1-70B \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 131.64 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 243641 Request throughput (req/s): 3.80 Input token throughput (tok/s): 1897.87 Output token throughput (tok/s): 1919.38 Peak output token throughput (tok/s): 3100.00 Peak concurrent requests: 107 Total token throughput (tok/s): 3817.25 Concurrency: 89.70 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 23616.71 Median E2E Latency (ms): 22770.44 ---------------Time to First Token---------------- Mean TTFT (ms): 245.98 Median TTFT (ms): 184.22 P99 TTFT (ms): 1251.67 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 47.19 Median TPOT (ms): 48.67 P99 TPOT (ms): 56.37 ---------------Inter-Token Latency---------------- Mean ITL (ms): 46.34 Median ITL (ms): 33.46 P95 ITL (ms): 108.61 P99 ITL (ms): 166.11 Max ITL (ms): 1107.09 ================================================== ``` #### 5.1.2 Summarization Scenario Benchmark ##### 5.1.2.1 Low Concurrency ```shell Command theme={null} sglang serve \ --backend sglang \ --model-path Meta-Llama/Llama-3.1-70B\ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 83.25 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.12 Input token throughput (tok/s): 503.77 Output token throughput (tok/s): 50.69 Peak output token throughput (tok/s): 54.00 Peak concurrent requests: 2 Total token throughput (tok/s): 554.46 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8322.45 Median E2E Latency (ms): 6873.36 ---------------Time to First Token---------------- Mean TTFT (ms): 395.25 Median TTFT (ms): 318.02 P99 TTFT (ms): 850.80 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 18.80 Median TPOT (ms): 18.81 P99 TPOT (ms): 19.03 ---------------Inter-Token Latency---------------- Mean ITL (ms): 18.83 Median ITL (ms): 18.81 P95 ITL (ms): 19.06 P99 ITL (ms): 19.08 Max ITL (ms): 23.08 ================================================== ``` ##### 5.1.2.2 Medium Concurrency ```shell Command theme={null} sglang serve \ --backend sglang \ --model-path Meta-Llama/Llama-3.1-70B \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 107.12 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41669 Total generated tokens (retokenized): 41603 Request throughput (req/s): 0.75 Input token throughput (tok/s): 2800.81 Output token throughput (tok/s): 389.00 Peak output token throughput (tok/s): 624.00 Peak concurrent requests: 19 Total token throughput (tok/s): 3189.81 Concurrency: 14.18 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 18988.30 Median E2E Latency (ms): 20290.66 ---------------Time to First Token---------------- Mean TTFT (ms): 603.42 Median TTFT (ms): 531.82 P99 TTFT (ms): 2607.95 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 36.94 Median TPOT (ms): 36.73 P99 TPOT (ms): 79.19 ---------------Inter-Token Latency---------------- Mean ITL (ms): 35.36 Median ITL (ms): 25.72 P95 ITL (ms): 27.07 P99 ITL (ms): 439.74 Max ITL (ms): 2529.51 ================================================== ``` ##### 5.1.2.3 High Concurrency ```shell Command theme={null} sglang serve \ --backend sglang \ --model-path Meta-Llama/Llama-3.1-70B \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 215.66 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 170000 Total generated tokens (retokenized): 169035 Request throughput (req/s): 1.48 Input token throughput (tok/s): 5906.92 Output token throughput (tok/s): 788.27 Peak output token throughput (tok/s): 1920.00 Peak concurrent requests: 69 Total token throughput (tok/s): 6695.19 Concurrency: 60.01 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 40443.85 Median E2E Latency (ms): 39813.12 ---------------Time to First Token---------------- Mean TTFT (ms): 633.32 Median TTFT (ms): 616.38 P99 TTFT (ms): 1912.97 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 74.95 Median TPOT (ms): 82.85 P99 TPOT (ms): 118.46 ---------------Inter-Token Latency---------------- Mean ITL (ms): 75.08 Median ITL (ms): 34.12 P95 ITL (ms): 261.18 P99 ITL (ms): 828.12 Max ITL (ms): 1970.03 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * **Results**: ```text Output theme={null} Accuracy: 0.830 Invalid: 0.000 Latency: 11.794 s Output throughput: 1406.961 token/s ``` # Llama-3.3-70B Source: https://docs.sglang.io/cookbook/autoregressive/Meta/Llama3.3-70B ## 1. Model Introduction [Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) is Meta's latest 70 billion parameter instruction-tuned language model, featuring improved performance and efficiency over Llama 3.1. With a 128K token context window and enhanced capabilities across reasoning, coding, and multilingual tasks, Llama 3.3 delivers state-of-the-art results while maintaining accessibility for production deployment. **Key Features:** * **Enhanced Performance**: Improved instruction following, reasoning, and task completion over Llama 3.1 * **Tool Calling**: Native support for function calling and tool use scenarios * **Multilingual Support**: Optimized for 8 languages (English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai) * **Extended Context**: 128K token context window for processing long documents and complex tasks * **Efficient Deployment**: 70B parameters enable deployment on single GPU with AMD MI300X **License:** Llama 3.3 is licensed under the Llama 3.3 Community License. See [LICENSE](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/blob/main/LICENSE) for details. For more details, please refer to the [official Llama models repository](https://github.com/meta-llama/llama-models). ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X), Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)) and Intel Xeon CPUs. ### 3.1 Interactive Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU, Intel Arc Pro B-Series GPUs or Intel Xeon CPUs setup. ### 3.2 Configuration Tips **AMD GPU Deployment:** * All AMD GPUs (MI300X, MI325X, MI355X) support TP=1 for both BF16 and FP8 variants * **FP8 Model Variant**: Use AMD's optimized `amd/Llama-3.3-70B-Instruct-FP8-KV` * **Tool Calling**: Enable with `--tool-call-parser llama3` for function calling support * **Higher Throughput**: Optional TP=2 or TP=4 can be used for increased throughput **Xeon CPU Deployment:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Tool Calling Llama 3.3 70B Instruct supports native tool calling. Enable the tool parser during deployment: ```shell Command theme={null} python -m sglang.launch_server \ --model-path meta-llama/Llama-3.3-70B-Instruct \ --tool-call-parser llama3 \ --tp 1 \ --host 0.0.0.0 \ --port 30000 ``` **Python Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request response = client.chat.completions.create( model="meta-llama/Llama-3.3-70B-Instruct", messages=[ {"role": "user", "content": "What's the weather in Tokyo?"} ], tools=tools, temperature=0.7 ) # Check for tool calls message = response.choices[0].message if message.tool_calls: tool_call = message.tool_calls[0] print(f"Function: {tool_call.function.name}") print(f"Arguments: {tool_call.function.arguments}") ``` **Handling Tool Call Results:** ```python Example theme={null} # After executing the function, send the result back def get_weather(location, unit="celsius"): # Your weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Build conversation with tool result messages = [ {"role": "user", "content": "What's the weather in Tokyo?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Tokyo", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Tokyo", "celsius") } ] final_response = client.chat.completions.create( model="meta-llama/Llama-3.3-70B-Instruct", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The current weather in Tokyo is 22°C and sunny. A perfect day!" ``` #### 4.2.2 Long Context Processing Leverage the 128K context window for processing long documents: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Example with long document long_document = "..." * 10000 # Your long document here response = client.chat.completions.create( model="meta-llama/Llama-3.3-70B-Instruct", messages=[ {"role": "user", "content": f"Summarize this document:\n\n{long_document}"} ], temperature=0.7, max_tokens=1000 ) print(response.choices[0].message.content) ``` ## 5. Benchmarking Use the SGLang benchmarking suite to test model performance with different workload patterns: ### 5.1 Basic Benchmark Command ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 1000 \ --random-input 1024 \ --random-output 1024 \ --max-concurrency 16 ``` ### 5.2 Adjusting Benchmark Parameters **Input/Output Length**: Adjust `--random-input` and `--random-output` to test different workload patterns: * Short conversations: `--random-input 1024 --random-output 1024` * Long outputs: `--random-input 1024 --random-output 8192` * Long inputs: `--random-input 8192 --random-output 1024` **Concurrency Levels**: Adjust `--max-concurrency` to test different load scenarios: * Low concurrency (latency-focused): `--max-concurrency 1 --num-prompts 100` * Medium concurrency (balanced): `--max-concurrency 16 --num-prompts 1000` * High concurrency (throughput-focused): `--max-concurrency 100 --num-prompts 2000` *** ## 📚 Additional Resources * [Meta Llama Models Repository](https://github.com/meta-llama/llama-models) * [Llama 3.3 Model Card](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) * [SGLang Documentation](/) * [AMD ROCm Documentation](https://rocm.docs.amd.com/) # Llama 4 Source: https://docs.sglang.io/cookbook/autoregressive/Meta/Llama4 ## 1. Model Introduction [Llama 4](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md) is Meta's latest generation of open-source LLM model with industry-leading performance. SGLang has supported Llama 4 Scout (109B) and Llama 4 Maverick (400B) since [v0.4.5](https://github.com/sgl-project/sglang/releases/tag/v0.4.5). Ongoing optimizations are tracked in the [Roadmap](https://github.com/sgl-project/sglang/issues/5118). This generation delivers comprehensive upgrades across the board: The highly capable Llama 4 Maverick with 17B active parameters out of \~400B total, with 128 experts. The efficient Llama 4 Scout also has 17B active parameters out of \~109B total, using just 16 experts. Both models leverage early fusion for native multimodality, enabling them to process text and image inputs. Maverick and Scout are both trained on up to 40 trillion tokens on data encompassing 200 languages (with specific fine-tuning support for 12 languages including Arabic, Spanish, German, and Hindi). For more details, please refer to the official llama4 Repository:[https://www.llama.com/models/llama-4/](https://www.llama.com/models/llama-4/) ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips * **OOM Mitigation:** Reduce `--context-length` to avoid GPU out-of-memory. Recommended: Scout up to 1M on 8×H100, up to 2.5M on 8×H200; Maverick doesn't need context-length set on 8×H200. With hybrid KV cache enabled, Scout can reach 5M on 8×H100 and 10M on 8×H200. * **Attention Backend Auto-Selection:** SGLang automatically picks the optimal backend. Manual override with `--attention-backend`: * Blackwell (B200/GB200): `trtllm_mha` * Hopper (H100/H200): `fa3` * AMD GPUs: `aiter` * Intel XPU: `intel_xpu` * Other: `triton` * **Chat Template:** Add `--chat-template llama-4` for chat completion tasks. * **Multi-Modal:** Add `--enable-multimodal` to enable image input support. * **Hybrid KV Cache:** Set `--swa-full-tokens-ratio` to control the ratio of SWA (local attention) KV tokens to full-attention KV tokens (default: 0.8, range: 0–1). * **EAGLE Speculative Decoding:** Supported for Llama 4 Scout and Maverick via EAGLE3. Enable with the interactive command generator above. * **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Launch the docker ```shell Command theme={null} docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x ``` ```shell Command theme={null} docker run -d -it --ipc=host --network=host --privileged \ --cap-add=CAP_SYS_ADMIN \ --device=/dev/kfd --device=/dev/dri --device=/dev/mem \ --group-add video --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v /:/work \ -e SHELL=/bin/bash \ --name Llama4 \ lmsysorg/sglang:v0.5.9-rocm720-mi30x \ /bin/bash ``` #### 4.2.2 Launch the server ### Llama-4-Scout 8-GPU deployment command: ```bash Command theme={null} sglang serve \ --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \ --tp 8 \ --context-length 1000000 \ --trust-remote-code ``` ### Llama-4-Maverick 8-GPU deployment command: ```bash Command theme={null} sglang serve \ --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --tp 8 \ --trust-remote-code ``` #### 4.2.3 EAGLE Speculative Decoding SGLang supports Llama 4 Maverick (400B) with [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding). Enable with the EAGLE3 algorithm and the SGLang EAGLE3 draft model: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path lmsys/sglang-EAGLE3-Llama-4-Maverick-17B-128E-Instruct-v1 \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --trust-remote-code \ --tp 8 ``` ## 5. Benchmark ### 5.1 Speed Benchmark (Scout) Test Environment: Hardware: AMD MI300x GPU Model: Llama-4-Scout Tensor Parallelism: 8 sglang version: 0.5.9 * **Model Deployment** ```bash Command theme={null} sglang serve \ --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \ --tp 8 \ --context-length 1000000 \ --trust-remote-code ``` ### 5.1.1 Low Concurrency (Latency-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model meta-llama/Llama-4-Scout-17B-16E-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 74.62 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4211 Request throughput (req/s): 0.14 Input token throughput (tok/s): 82.88 Output token throughput (tok/s): 57.42 Peak output token throughput (tok/s): 146.00 Peak concurrent requests: 2 Total token throughput (tok/s): 140.20 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7459.48 Median E2E Latency (ms): 4489.77 ---------------Time to First Token---------------- Mean TTFT (ms): 4246.98 Median TTFT (ms): 68.57 P99 TTFT (ms): 48091.05 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.49 Median TPOT (ms): 7.40 P99 TPOT (ms): 7.40 ---------------Inter-Token Latency---------------- Mean ITL (ms): 7.49 Median ITL (ms): 7.49 P95 ITL (ms): 7.47 P99 ITL (ms): 7.52 Max ITL (ms): 10.44 ================================================== ``` ### 5.1.2 Medium Concurrency (Balanced) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model meta-llama/Llama-4-Scout-17B-16E-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 45.41 Total input tokens: 49668 Total input text tokens: 49668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40516 Request throughput (req/s): 2.26 Input token throughput (tok/s): 1120.46 Output token throughput (tok/s): 1152.47 Peak output token throughput (tok/s): 1520.00 Peak concurrent requests: 21 Total token throughput (tok/s): 2272.84 Concurrency: 14.76 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6089.22 Median E2E Latency (ms): 6568.80 ---------------Time to First Token---------------- Mean TTFT (ms): 124.44 Median TTFT (ms): 87.42 P99 TTFT (ms): 268.72 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 11.88 Median TPOT (ms): 12.00 P99 TPOT (ms): 15.49 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.72 Median ITL (ms): 10.54 P95 ITL (ms): 11.22 P99 ITL (ms): 67.88 Max ITL (ms): 74.05 ================================================== ``` ### 5.1.3 High Concurrency (Throughput-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model meta-llama/Llama-4-Scout-17B-16E-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 85.84 Total input tokens: 249841 Total input text tokens: 249841 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 250498 Request throughput (req/s): 5.84 Input token throughput (tok/s): 2910.84 Output token throughput (tok/s): 2944.82 Peak output token throughput (tok/s): 4100.00 Peak concurrent requests: 110 Total token throughput (tok/s): 5854.65 Concurrency: 92.24 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 15844.00 Median E2E Latency (ms): 15262.56 ---------------Time to First Token---------------- Mean TTFT (ms): 204.46 Median TTFT (ms): 129.96 P99 TTFT (ms): 528.54 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 41.56 Median TPOT (ms): 42.90 P99 TPOT (ms): 47.48 ---------------Inter-Token Latency---------------- Mean ITL (ms): 40.99 Median ITL (ms): 24.46 P95 ITL (ms): 84.46 P99 ITL (ms): 87.64 Max ITL (ms): 226.06 ================================================== ``` ### 5.2 Speed Benchmark (Maverick) Test Environment: Hardware: AMD MI300x GPU Model: Llama-4-Maverick Tensor Parallelism: 8 sglang version: 0.5.9 * **Model Deployment** ```bash Command theme={null} sglang serve \ --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --tp 8 \ --context-length 1000000 \ --trust-remote-code ``` ### 5.2.1 Low Concurrency (Latency-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 68.08 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4202 Request throughput (req/s): 0.15 Input token throughput (tok/s): 89.62 Output token throughput (tok/s): 61.99 Peak output token throughput (tok/s): 168.00 Peak concurrent requests: 2 Total token throughput (tok/s): 151.61 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6805.62 Median E2E Latency (ms): 2733.91 ---------------Time to First Token---------------- Mean TTFT (ms): 4296.56 Median TTFT (ms): 57.45 P99 TTFT (ms): 38633.95 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.95 Median TPOT (ms): 5.96 P99 TPOT (ms): 5.97 ---------------Inter-Token Latency---------------- Mean ITL (ms): 5.96 Median ITL (ms): 5.96 P95 ITL (ms): 6.02 P99 ITL (ms): 6.08 Max ITL (ms): 7.02 ================================================== ``` ### 5.2.2 Medium Concurrency (Balanced) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 30.72 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40923 Request throughput (req/s): 2.60 Input token throughput (tok/s): 1291.39 Output token throughput (tok/s): 1328.41 Peak output token throughput (tok/s): 1760.00 Peak concurrent requests: 22 Total token throughput (tok/s): 2619.80 Concurrency: 13.92 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5345.15 Median E2E Latency (ms): 5679.73 ---------------Time to First Token---------------- Mean TTFT (ms): 259.30 Median TTFT (ms): 72.60 P99 TTFT (ms): 1063.45 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.53 Median TPOT (ms): 10.22 P99 TPOT (ms): 20.27 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.99 Median ITL (ms): 9.10 P95 ITL (ms): 9.87 P99 ITL (ms): 55.62 Max ITL (ms): 868.54 ================================================== ``` ### 5.2.3 High Concurrency (Throughput-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 90.95 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 251625 Request throughput (req/s): 5.50 Input token throughput (tok/s): 2746.77 Output token throughput (tok/s): 2777.90 Peak output token throughput (tok/s): 3700.00 Peak concurrent requests: 109 Total token throughput (tok/s): 5524.67 Concurrency: 93.04 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16924.17 Median E2E Latency (ms): 16294.85 ---------------Time to First Token---------------- Mean TTFT (ms): 188.19 Median TTFT (ms): 128.96 P99 TTFT (ms): 534.81 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 33.63 Median TPOT (ms): 35.37 P99 TPOT (ms): 38.26 ---------------Inter-Token Latency---------------- Mean ITL (ms): 33.19 Median ITL (ms): 27.66 P95 ITL (ms): 76.91 P99 ITL (ms): 78.82 Max ITL (ms): 268.17 ================================================== ``` ### 5.3 Accuracy Benchmark #### 5.3.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * Llama-4-Scout-17B-16E-Instruct ```text Output theme={null} Accuracy: 0.945 Invalid: 0.000 Latency: 12.731 s Output throughput: 1595.418 token/s ``` * Llama-4-Maverick-17B-128E-Instruct ```text Output theme={null} Accuracy: 0.895 Invalid: 0.000 Latency: 9.739 s Output throughput: 2405.505 token/s ``` #### 5.3.2 MMLU-Pro with sgl-eval The table below records historical lm-eval measurements on 8×H100 alongside [Meta's official benchmark numbers](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) (reproduction details: [PR #5092](https://github.com/sgl-project/sglang/pull/5092)). These scores were not produced by the sgl-eval commands below; rerun the evaluation for results using sgl-eval's prompts and grading.
Model Official SGLang
Llama-4-Scout-17B-16E-Instruct 74.3 75.2
Llama-4-Maverick-17B-128E-Instruct 80.5 80.7
Install the evaluation package before running either command: ```bash theme={null} pip install sgl-eval==0.1.0 ``` **Scout:** ```bash Command theme={null} # Start the server python -m sglang.launch_server \ --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \ --port 30000 \ --tp 8 \ --mem-fraction-static 0.8 \ --context-length 65536 # Run MMLU-Pro sgl-eval run mmlu_pro \ --base-url http://localhost:30000/v1 \ --model meta-llama/Llama-4-Scout-17B-16E-Instruct \ --max-tokens 2048 \ --num-threads 128 ``` **Maverick:** ```bash Command theme={null} # Start the server python -m sglang.launch_server \ --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --port 30000 \ --tp 8 \ --mem-fraction-static 0.8 \ --context-length 65536 # Run MMLU-Pro sgl-eval run mmlu_pro \ --base-url http://localhost:30000/v1 \ --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --max-tokens 2048 \ --num-threads 128 ``` # Muse Glimmer Source: https://docs.sglang.io/cookbook/autoregressive/Meta/MuseGlimmer A multimodal reasoning model served from a BF16, NVFP4 + MXFP8, vendor GGUF, or MLX checkpoint. ## Deployment See the [official SGLang installation guide](../../../docs/get-started/install) for all installation methods and hardware platforms. The steps below match the **Python** and **Docker** options in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv # Muse Glimmer support is not in a release yet -- build the PR branch: # https://github.com/sgl-project/sglang/pull/34262 git clone -b muse-glimmer https://github.com/sgl-project/sglang.git cd sglang uv pip install --prerelease=allow -e "python[all]" ``` Run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev-muse-glimmer ``` See [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker) to start the image. Replace the inner `sglang serve ...` command with the command from the panel below. Select a checkpoint format. Select whether to use speculative decoding: * **Standard**: Use normal autoregressive decoding. * **DFlash**: Use speculative decoding with the DFlash draft model. The draft serves as published, with no conversion step. See [§2](#2-configuration-tips). ## Playground Use the Playground to test SGLang features that are not in the verified matrix. The Deploy panel above shows only combinations that the SGLang team has verified. The Playground lets you add more options to the command from the Deploy panel. ## 1. Model Introduction Muse Glimmer is a multimodal reasoning model. You can serve Muse Glimmer in four formats: * A BF16 checkpoint (`MuseGlimmerForConditionalGeneration`). * A set of vendor GGUF files. * A ready-to-serve NVFP4 + MXFP8 checkpoint. * Three MLX repacks for Apple Silicon.
Form Source Notes
BF16 meta-models/Muse-Glimmer-30B Supports image input.
GGUF Q4\_K\_M meta-models/Muse-Glimmer-30B-GGUF Text only. This path is not optimized. SGLang shows a warning at startup.
NVFP4 RadixArk/Muse-Glimmer-NVFP4 Text only. Ready to serve, no conversion needed.
MLX Q4 RadixArk/Muse-Glimmer-q4-MLX Text only. Apple Silicon (MLX backend). Same serve recipe as gs128, no measured round yet. See §3.4.
MLX Q4\_K\_M (gs128) RadixArk/Muse-Glimmer-q4km-gs128-MLX Text only. Apple Silicon (MLX backend). Carries the vendor GGUF's exact quantization codes in MLX format. The measured MLX artifact. See §3.4.
MLX Q4\_K (dynamic) RadixArk/Muse-Glimmer-q4k-dynamic-MLX Text only. Apple Silicon (MLX backend). Same serve recipe as gs128, no measured round yet. See §3.4.
**Resources:** [Muse-Glimmer-30B (BF16)](https://huggingface.co/meta-models/Muse-Glimmer-30B) · [Muse-Glimmer-30B-assistant (DFlash draft)](https://huggingface.co/meta-models/Muse-Glimmer-30B-assistant) · [Muse-Glimmer-30B-GGUF](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF) · [Muse-Glimmer-NVFP4](https://huggingface.co/RadixArk/Muse-Glimmer-NVFP4) · MLX · [q4](https://huggingface.co/RadixArk/Muse-Glimmer-q4-MLX) · [q4km-gs128](https://huggingface.co/RadixArk/Muse-Glimmer-q4km-gs128-MLX) · [q4k-dynamic](https://huggingface.co/RadixArk/Muse-Glimmer-q4k-dynamic-MLX). ## 2. Configuration Tips **The GGUF format is text only.** SGLang has no `mmproj` path. You cannot use the vision GGUF files. Use the BF16 checkpoint for multimodal input. **The NVFP4 checkpoint.** `RadixArk/Muse-Glimmer-NVFP4` is a ready-to-serve NVFP4 + MXFP8 checkpoint. No conversion needed — point `--model-path` straight at it. **The DFlash draft.** `meta-models/Muse-Glimmer-30B-assistant` is the vendor's native draft export and serves directly. No conversion needed. **DFlash with a GGUF target model** needs `--speculative-draft-load-format auto`. Without this flag, the draft model uses the `gguf` load format from the target model. The loader then rejects the draft directory. **Apple Silicon uses an MLX checkpoint, not the GGUF files.** The MLX backend has no GGUF path. Serve one of the three `RadixArk/Muse-Glimmer-*-MLX` artifacts with `SGLANG_USE_MLX=1` (see the Apple Silicon cells in the command panel). All three take the same flags; `q4km-gs128` is the one with a measured round. Keep `--disable-radix-cache` — the windowed KV storage for the sliding-window layers requires it — and set `SGLANG_MLX_CACHE_LIMIT_GB=8` so the MLX buffer cache does not grow the footprint under concurrent load. Speculative decoding is not available on the MLX backend. ## 3. Advanced Usage ### 3.1 Reasoning Muse Glimmer enables the `muse` reasoning parser by default. This parser separates the reasoning text from the final answer. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="meta-models/Muse-Glimmer-30B", messages=[{"role": "user", "content": "What is 15% of 240?"}], ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ### 3.2 Tool Calling Muse Glimmer enables the `muse` tool-call parser by default. This parser sends structured tool calls in `message.tool_calls`. ### 3.3 Multimodal The BF16 checkpoint supports image input. It defaults to text only. To switch, select **Modality** in the command panel above. **Text only** adds `--language-model-only`. This flag turns off the vision tower. SGLang does not build or load the vision weights. This frees memory for the KV cache. SGLang rejects image requests in this mode. Select **Image + text** to turn on image input. NVFP4, GGUF, and the MLX artifacts are text only. The Modality option does not appear for GGUF or MLX; NVFP4 only offers **Text only**. ### 3.4 Apple Silicon (MLX) The MLX backend serves three Muse Glimmer artifacts on Apple Silicon Macs (48 GB unified memory or more). All three are text only — the MLX backend has no vision path — and all three take the same flags, so pick one in the command panel: * `RadixArk/Muse-Glimmer-q4-MLX` — no measured round yet. * `RadixArk/Muse-Glimmer-q4km-gs128-MLX` — a lossless repack of the vendor's Q4\_K\_M (gs128) GGUF: every weight keeps the GGUF's exact quantization code, with the group scales re-expressed in MLX affine bf16 (≤2⁻⁸ relative rounding). The numbers below are for this artifact. * `RadixArk/Muse-Glimmer-q4k-dynamic-MLX` — no measured round yet. Choose along the speed-versus-accuracy axis: footprint and expected accuracy both grow `q4` → `q4km-gs128` → `q4k-dynamic`, and decode speed moves the other way. Decode on Apple Silicon is memory-bandwidth-bound, so a smaller artifact reads fewer weight bytes per token — more tokens per second, and more unified memory left over for the KV cache. Take `q4` for the fastest responses on the smallest machine, `q4k-dynamic` to stay closest to BF16, and `q4km-gs128` for the middle ground — it is also the only one of the three with a measured round, below. This table shows accuracy for the gs128 checkpoint, with the vendor llama.cpp fork serving the source GGUF on the same machine as the reference. GSM8K: 200 questions, no-thinking chat template, temperature 0, max 2048 new tokens. CIMemories: 1 profile, full combo, single trial, DeepSeek-R1-0528 judge.
Benchmark SGLang MLX llama.cpp (same GGUF)
GSM8K (200q, no-thinking, greedy) 0.970 0.970
CIMemories — violation rate (lower is better) 0.00% 8.27%
CIMemories — coverage (higher is better) 76.0% 68.4%
CIMemories is a single-trial benchmark with a nondeterministic judge; treat the SGLang-vs-llama.cpp gap on that row as run noise, not a runtime effect. GSM8K parity is exact. Decode throughput for gs128 on an M5 Pro (64 GB), 1k-in/1k-out greedy: 15.3 tok/s at batch 1, rising to 52.6 tok/s aggregate at batch 8 — ahead of llama.cpp on the same GGUF codes at every batch size above 1. # MiniMax-M2 Source: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M2 ## 1. Model Introduction [MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) is a compact, fast, and cost-effective MoE model (230 billion total parameters with 10 billion active parameters) built for elite performance in coding and agentic tasks, all while maintaining powerful general intelligence. This generation delivers comprehensive upgrades across the board: * **Superior Intelligence**: MiniMax-M2 demonstrates highly competitive general intelligence across mathematics, science, instruction following, coding, and agentic tool use in [Artificial Analysis](https://artificialanalysis.ai/). Its composite score ranks #1 among open-source models globally. * **Advanced Coding**: Engineered for end-to-end developer workflows, MiniMax-M2 excels at multi-file edits, coding-run-fix loops, and test-validated repairs. Strong performance on Terminal-Bench and (Multi-)SWE-Bench–style tasks demonstrates practical effectiveness in terminals, IDEs, and CI across languages. * **Agent Performance**: MiniMax-M2 plans and executes complex, long-horizon toolchains across shell, browser, retrieval, and code runners. In BrowseComp-style evaluations, it consistently locates hard-to-surface sources, maintains evidence traceable, and gracefully recovers from flaky steps. * **Efficient Design**: With 10 billion activated parameters (230 billion in total), MiniMax-M2 delivers lower latency, lower cost, and higher throughput for interactive agents and batched sampling—perfectly aligned with the shift toward highly deployable models that still shine on coding and agentic tasks. For more details, please refer to the [official Minimax GitHub Repository](https://github.com/MiniMax-AI). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. The AMD environment is currently available in SGLang via Docker image install. ### 2.1 AMD Docker #### 2.1.1 Launch docker ```shell Command theme={null} docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x ``` ```shell Command theme={null} docker run -d -it --ipc=host --network=host --privileged \ --cap-add=CAP_SYS_ADMIN \ --device=/dev/kfd --device=/dev/dri --device=/dev/mem \ --group-add video --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v /:/work \ -e SHELL=/bin/bash \ --name Minimax \ lmsysorg/sglang:v0.5.9-rocm720-mi30x \ /bin/bash ``` #### 2.1.2 Make modifications inside the docker ```shell Command theme={null} mv /sgl-workspace/sglang/python/sglang/srt/models/transformers.py \ /sgl-workspace/sglang/python/sglang/srt/models/hf_transformers_model.py ``` #### 2.1.3 Fix torch compile Comment out the following line: @torch.compile(dynamic=True, backend=get\_compiler\_backend()) in /sgl-workspace/sglang/python/sglang/srt/models/minimax\_m2.py ```shell Command theme={null} #@torch.compile(dynamic=True, backend=get_compiler_backend()) ``` ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. #### 3.1.1 NVIDIA GPU Deployment The interactive command generator above covers AMD deployments. For NVIDIA GPUs (H100/H200/B200), use these explicit commands: **4-GPU deployment (up to 400K context):** ```bash Command theme={null} python -m sglang.launch_server \ --model-path MiniMaxAI/MiniMax-M2 \ --tp-size 4 \ --tool-call-parser minimax-m2 \ --reasoning-parser minimax-append-think \ --host 0.0.0.0 \ --trust-remote-code \ --port 30000 \ --mem-fraction-static 0.85 ``` **8-GPU deployment (up to 3M context):** ```bash Command theme={null} python -m sglang.launch_server \ --model-path MiniMaxAI/MiniMax-M2 \ --tp-size 8 \ --ep-size 8 \ --tool-call-parser minimax-m2 \ --reasoning-parser minimax-append-think \ --host 0.0.0.0 \ --trust-remote-code \ --port 30000 \ --mem-fraction-static 0.85 ``` ### 3.2 System Requirements Recommended configurations — actual requirements depend on workload:
GPUs Context Length Support
4× 96 GB GPUs Up to 400K tokens
8× 144 GB GPUs Up to 3M tokens
### 3.3 Testing Deployment After the server starts, verify with: ```shell Command theme={null} curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-M2", "messages": [ {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]} ] }' ``` ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser Server Command: ```shell Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2 \ --tp-size 4 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 ``` Test Code: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.6, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` Output Example: ```text Output theme={null} First, the user asks: "What is 15% of 240?" This is a straightforward percentage calculation. I need to solve it step by step as per the instruction. The problem is: What is 15% of 240? To find a percentage of a number, I multiply the number by the percentage divided by 100. So, 15% is 15/100, which simplifies to 0.15. Therefore, 15% of 240 is 240 times 0.15. Let me calculate that: 240 × 0.15. I can break it down: 240 × 0.15 = 240 × (15/100) = (240 × 15) / 100. Now, 240 × 15. 200 × 15 = 3000, and 40 × 15 = 600, so total 3000 + 600 = 3600. Then, divide by 100: 3600 / 100 = 36. So, 15% of 240 is 36. I should confirm this with another method. For example, 10% of 240 is 24, and 5% is half of that, which is 12. Then 15% is 10% + 5% = 24 + 12 = 36. Same answer. Or, using fractions: 15% = 3/20, so 240 × 3/20 = (240 / 20) × 3 = 12 × 3 = 36. All methods confirm it's 36. The user said "solve this problem step by step," so I should present the steps clearly. Step 1: Understand that "15% of 240" means 15 per hundred of 240. Step 2: Convert percentage to decimal: 15% = 15/100 = 0.15. Step 3: Multiply the number by the decimal: 240 × 0.15. Step 4: Calculate the multiplication: 240 × 0.15 = 36. Step 5: Therefore, 15% of 240 is 36. I should also mention that percentage means per hundred, so it's straightforward. Finally, I need to box the answer as per the instruction. So, the final answer is 36. To find 15% of 240, follow these steps: 1. **Understand the percentage**: "15%" means 15 per hundred, or 15/100. 2. **Convert to a decimal**: 15/100 = 0.15. 3. **Multiply by the number**: 240 × 0.15. 4. **Calculate the result**: - 240 × 0.15 = 36. Alternatively, you can break it down: - 10% of 240 is 24 (since 240 ÷ 10 = 24). - 5% of 240 is half of 10%, which is 12. - Therefore, 15% is 10% + 5% = 24 + 12 = 36. Both methods confirm the result. **Answer**: 36 ``` ### 4.2.2 Tool Calling Server Command: ```shell Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2 \ --tp-size 4 \ --tool-call-parser minimax-m2 \ --trust-remote-code \ --mem-fraction-static 0.85 ``` Test Code: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` Output Example: ```text Output theme={null} Alright, the user is asking about the weather in Beijing. This is a straightforward request that I can help with using the get_weather tool that's available to me. Let me think about what I need to do here. The user wants to know the current weather conditions in Beijing, which is the capital city of China. To provide this information, I need to use the get_weather tool that's been provided to me. Looking at the tool's parameters, I can see it requires: 1. location - which is required and should be a string representing the city name 2. unit - which is optional and can be either "celsius" or "fahrenheit" For the location parameter, I'll use "Beijing" since that's what the user asked about. For the unit parameter, the user didn't specify their preference between celsius and fahrenheit. Since Beijing is in China, which primarily uses celsius, and celsius is the more standard unit internationally, I'll default to celsius. If the user wants the temperature in fahrenheit instead, they can ask in a follow-up message and I can provide that information. So I need to make a tool call to get_weather with the following parameters: - location: "Beijing" - unit: "celsius" This should return the current weather information for Beijing, which I can then share with the user. I'll format my response using the required XML tags for tool calls as specified in my instructions. 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment**: * Hardware: AMD MI300X GPU(4x) * Model: MiniMax-M2 * Tensor Parallelism: 4 * sglang version: 0.5.7 **Model Deployment**: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2 \ --tp-size 4 \ --trust-remote-code \ --mem-fraction-static 0.85 ``` ### 5.1.1 Low Concurrency (Latency-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 138.91 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.07 Input token throughput (tok/s): 43.92 Output token throughput (tok/s): 30.38 Peak output token throughput (tok/s): 46.00 Peak concurrent requests: 2 Total token throughput (tok/s): 74.30 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 13887.62 Median E2E Latency (ms): 10377.26 ---------------Time to First Token---------------- Mean TTFT (ms): 4528.94 Median TTFT (ms): 385.23 P99 TTFT (ms): 38338.51 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 22.21 Median TPOT (ms): 22.24 P99 TPOT (ms): 22.25 ---------------Inter-Token Latency---------------- Mean ITL (ms): 22.23 Median ITL (ms): 22.24 P95 ITL (ms): 22.35 P99 ITL (ms): 22.41 Max ITL (ms): 23.64 ================================================== ``` ### 5.1.2 Medium Concurrency (Balanced) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 81.07 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40803 Request throughput (req/s): 0.99 Input token throughput (tok/s): 489.29 Output token throughput (tok/s): 503.32 Peak output token throughput (tok/s): 704.00 Peak concurrent requests: 19 Total token throughput (tok/s): 992.61 Concurrency: 13.74 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 13925.95 Median E2E Latency (ms): 14348.75 ---------------Time to First Token---------------- Mean TTFT (ms): 532.32 Median TTFT (ms): 147.69 P99 TTFT (ms): 1978.48 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 27.49 Median TPOT (ms): 26.56 P99 TPOT (ms): 46.52 ---------------Inter-Token Latency---------------- Mean ITL (ms): 26.31 Median ITL (ms): 23.47 P95 ITL (ms): 24.37 P99 ITL (ms): 125.10 Max ITL (ms): 1192.51 ================================================== ``` ### 5.1.3 High Concurrency (Throughput-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 153.71 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 250982 Request throughput (req/s): 3.25 Input token throughput (tok/s): 1625.33 Output token throughput (tok/s): 1643.75 Peak output token throughput (tok/s): 2597.00 Peak concurrent requests: 107 Total token throughput (tok/s): 3269.09 Concurrency: 91.14 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 28017.24 Median E2E Latency (ms): 26865.28 ---------------Time to First Token---------------- Mean TTFT (ms): 387.41 Median TTFT (ms): 183.90 P99 TTFT (ms): 1192.44 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 55.23 Median TPOT (ms): 57.84 P99 TPOT (ms): 70.23 ---------------Inter-Token Latency---------------- Mean ITL (ms): 54.79 Median ITL (ms): 39.01 P95 ITL (ms): 143.10 P99 ITL (ms): 150.46 Max ITL (ms): 986.14 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Server Command**: ```shell Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2 \ --tp-size 4 \ --trust-remote-code \ --mem-fraction-static 0.85 ``` * **Benchmark Command**: ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * **Result**: * MiniMax-M2 ```text Output theme={null} Accuracy: 0.950 Invalid: 0.000 Latency: 15.120 s Output throughput: 1306.711 token/s ``` # MiniMax-M2.5 Source: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M2.5 ## 1. Model Introduction [MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5) is a powerful language model developed by MiniMax, built for real-world productivity with state-of-the-art performance across coding, reasoning, agentic tasks, and tool use. As the latest iteration in the MiniMax model series, MiniMax-M2.5 achieves comprehensive enhancements across multiple domains. Details are as follows: * **Superior coding performance**: Achieves 79.7 on Droid and 76.1 on OpenCode, surpassing Opus 4.6 (78.9 and 75.9 respectively). Strong results on SWE-bench Verified, SWE-bench Multilingual, SWE-bench-pro, and Multi-SWE-bench. * **Advanced reasoning**: Demonstrates strong performance on AIME25 and other reasoning benchmarks, with robust tool use during inference. * **More capable agents**: Excels in agentic tasks including web browsing (BrowseComp, Wide Search), information retrieval (RISE), and complex tool use scenarios (Terminal Bench 2, MEWC, Finance Modeling). * **Real-world productivity**: Designed for production-grade workloads with strong performance on practical coding, data analysis, and multi-step reasoning tasks. For more details, please refer to the [official MiniMax-M2.5 announcement](https://www.minimax.io/news/minimax-m25). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. **For AMD MI300X/MI325X/MI355X GPUs:** ```bash Command theme={null} # Docker (AMD MI300X/MI325X) docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x # Docker (AMD MI355X) docker pull lmsysorg/sglang:v0.5.9-rocm720-mi35x ``` ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and feature capabilities. ### 3.2 Configuration Tips **Key Parameters:**
Parameter Description Recommended Value
`--tool-call-parser` Tool call parser for function calling support `minimax-m2`
`--reasoning-parser` Reasoning parser for thinking mode `minimax-append-think`
`--trust-remote-code` Required for MiniMax model loading Always enabled
`--mem-fraction-static` Static memory fraction for KV cache `0.85`
`--tp` Tensor parallelism size `2` (2-GPU) or `4` (4-GPU) or `8` (8-GPU)
`--ep` Expert parallelism size `8` (NVIDIA 8-GPU) or EP=TP (AMD)
`--kv-cache-dtype` KV cache data type (AMD only) `fp8_e4m3`
`--attention-backend` Attention backend (AMD only) `triton`
**Hardware Requirements: NVIDIA** * **4-GPU deployment**: Requires 4× high-memory GPUs (e.g., H200, B200, A100, H100) with TP=4 * **8-GPU deployment**: Requires 8× GPUs (e.g., H200, B200, A100, H100) with TP=8 and EP=8 **Hardware Requirements: AMD** * **2-GPU deployment**: Requires 2× high-memory GPUs (e.g., MI300X, MI325X, MI355X) with TP=2, EP=2 * **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4 * **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8 ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) **Testing Deployment:** After startup, you can test the SGLang OpenAI-compatible API with the following command: ```bash Command theme={null} curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-M2.5", "messages": [ {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]} ] }' ``` **Simple Completion Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.5", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"} ], max_tokens=1024 ) print(response.choices[0].message.content) ``` **Example Output**: ```text Output theme={null} The user asks: "Who won the world series in 2020?" That is a straightforward factual question. The answer: the Los Angeles Dodgers. They won the 2020 World Series, beating the Tampa Bay Rays. The user is presumably expecting that answer. We must follow the policies. The question is safe: no disallowed content. It's just a factual question. Provide answer. We must ensure compliance: Use no disallowed content. Should we provide context? Just answer straightforwardly. The user simply asks "Who won the world series in 2020?" We'll answer: The Los Angeles Dodgers. No additional relevant info needed, but could elaborate briefly: They beat the Tampa Bay Rays in six games, the series was played in a bubble at Globe Life Field in Arlington, Texas due to COVID-19. No need for any extra. That's it. The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays in six games. ``` ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser MiniMax-M2.5 supports Thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model-path MiniMaxAI/MiniMax-M2.5 \ --tp 4 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 ``` **Streaming with Thinking Process** With `minimax-append-think`, the thinking content is wrapped in `...` tags within the `content` field. You can parse these tags on the client side to separate the thinking and content sections: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.5", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream, separating ... from content in_think = False think_printed_header = False content_printed_header = False buffer = "" for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: buffer += delta.content while buffer: if in_think: # Look for closing tag end_idx = buffer.find("") if end_idx != -1: print(buffer[:end_idx], end="", flush=True) buffer = buffer[end_idx + len(""):] in_think = False else: # Still in thinking, print what we have print(buffer, end="", flush=True) buffer = "" else: # Look for opening tag start_idx = buffer.find("") if start_idx != -1: # Print any content before before = buffer[:start_idx] if before: if not content_printed_header: print("=============== Content =================", flush=True) content_printed_header = True print(before, end="", flush=True) buffer = buffer[start_idx + len(""):] in_think = True if not think_printed_header: print("=============== Thinking =================", flush=True) think_printed_header = True else: # No tag, print as content if not content_printed_header and think_printed_header: print("\n=============== Content =================", flush=True) content_printed_header = True print(buffer, end="", flush=True) buffer = "" print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user asks: "Solve this problem step by step: What is 15% of 240?" This is straightforward: 15% = 0.15; 0.15*240 = 36. So answer: 36. Provide step-by-step: convert percent to decimal, multiply. We need to obey policies. There's no policy violation. Just answer. Provide step by step. Should respond with solution. We can also mention alternative method: 15% = 15/100 = 3/20. Multiply 240 * 3/20 = (240/20)*3 = 12*3 = 36. Thus answer 36. We can add step-by-step. That's it. =============== Content ================= **Step‑by‑step solution** 1. **Convert the percent to a decimal** \[ 15\% = \frac{15}{100}=0.15 \] 2. **Multiply the decimal by the number** \[ 0.15 \times 240 = 36 \] (You can also think of it as \(15\% = \frac{3}{20}\) and then \(240 \times \frac{3}{20}=12 \times 3 = 36\).) \[ \boxed{36} \] ``` **Note:** The `minimax-append-think` reasoning parser embeds the thinking process in `...` tags within the `content` field. The code above parses these tags in real-time to display thinking and content separately. #### 4.2.2 Tool Calling MiniMax-M2.5 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model-path MiniMaxAI/MiniMax-M2.5 \ --tp 4 \ --tool-call-parser minimax-m2 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 ``` **Python Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Non-streaming request response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.5", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7 ) message = response.choices[0].message # Check for tool calls if message.tool_calls: for tool_call in message.tool_calls: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") else: print(message.content) ``` **Output Example**: ```text Output theme={null} Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Note:** * Tool calls are returned in `message.tool_calls` with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.5", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment**: * Hardware: NVIDIA B200 GPU (8x) * Model: MiniMax-M2.5 * Tensor Parallelism: 8 * Expert Parallelism: 8 * sglang version: 0.5.8 #### 5.1.1 Standard Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2.5 \ --tp 8 \ --ep 8 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 \ --tool-call-parser minimax-m2 ``` ##### 5.1.1.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 42.99 Total input tokens: 6091 Total input text tokens: 6091 Total generated tokens: 4220 Total generated tokens (retokenized): 3804 Request throughput (req/s): 0.23 Input token throughput (tok/s): 141.70 Output token throughput (tok/s): 98.17 Peak output token throughput (tok/s): 102.00 Peak concurrent requests: 2 Total token throughput (tok/s): 239.87 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4295.92 Median E2E Latency (ms): 3419.28 P90 E2E Latency (ms): 7832.04 P99 E2E Latency (ms): 9601.40 ---------------Time to First Token---------------- Mean TTFT (ms): 130.57 Median TTFT (ms): 116.10 P99 TTFT (ms): 190.90 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.89 Median TPOT (ms): 9.89 P99 TPOT (ms): 9.91 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.89 Median ITL (ms): 9.89 P95 ITL (ms): 10.15 P99 ITL (ms): 10.32 Max ITL (ms): 14.46 ================================================== ``` ##### 5.1.1.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 48.43 Total input tokens: 39588 Total input text tokens: 39588 Total generated tokens: 40805 Total generated tokens (retokenized): 37142 Request throughput (req/s): 1.65 Input token throughput (tok/s): 817.37 Output token throughput (tok/s): 842.49 Peak output token throughput (tok/s): 1184.00 Peak concurrent requests: 21 Total token throughput (tok/s): 1659.86 Concurrency: 13.67 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8274.32 Median E2E Latency (ms): 8692.90 P90 E2E Latency (ms): 13690.70 P99 E2E Latency (ms): 16104.18 ---------------Time to First Token---------------- Mean TTFT (ms): 305.44 Median TTFT (ms): 106.75 P99 TTFT (ms): 1053.26 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 16.20 Median TPOT (ms): 16.06 P99 TPOT (ms): 26.75 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.65 Median ITL (ms): 13.63 P95 ITL (ms): 14.90 P99 ITL (ms): 87.99 Max ITL (ms): 483.53 ================================================== ``` ##### 5.1.1.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 92.31 Total input tokens: 249331 Total input text tokens: 249331 Total generated tokens: 252662 Total generated tokens (retokenized): 218975 Request throughput (req/s): 5.42 Input token throughput (tok/s): 2700.94 Output token throughput (tok/s): 2737.02 Peak output token throughput (tok/s): 4479.00 Peak concurrent requests: 109 Total token throughput (tok/s): 5437.97 Concurrency: 91.19 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16835.82 Median E2E Latency (ms): 16042.08 P90 E2E Latency (ms): 31027.63 P99 E2E Latency (ms): 34787.91 ---------------Time to First Token---------------- Mean TTFT (ms): 391.06 Median TTFT (ms): 133.12 P99 TTFT (ms): 1712.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 33.04 Median TPOT (ms): 34.29 P99 TPOT (ms): 41.98 ---------------Inter-Token Latency---------------- Mean ITL (ms): 32.61 Median ITL (ms): 21.67 P95 ITL (ms): 87.76 P99 ITL (ms): 118.81 Max ITL (ms): 1145.62 ================================================== ``` #### 5.1.2 Summarization Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2.5 \ --tp 8 \ --ep 8 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 \ --tool-call-parser minimax-m2 ``` ##### 5.1.2.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 43.49 Total input tokens: 41941 Total input text tokens: 41941 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.23 Input token throughput (tok/s): 964.42 Output token throughput (tok/s): 97.04 Peak output token throughput (tok/s): 102.00 Peak concurrent requests: 2 Total token throughput (tok/s): 1061.46 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4346.83 Median E2E Latency (ms): 3508.84 P90 E2E Latency (ms): 7972.23 P99 E2E Latency (ms): 9659.71 ---------------Time to First Token---------------- Mean TTFT (ms): 131.50 Median TTFT (ms): 126.76 P99 TTFT (ms): 182.52 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.00 Median TPOT (ms): 10.01 P99 TPOT (ms): 10.12 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.01 Median ITL (ms): 10.02 P95 ITL (ms): 10.29 P99 ITL (ms): 10.44 Max ITL (ms): 14.11 ================================================== ``` ##### 5.1.2.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 50.12 Total input tokens: 300020 Total input text tokens: 300020 Total generated tokens: 41669 Total generated tokens (retokenized): 41662 Request throughput (req/s): 1.60 Input token throughput (tok/s): 5986.00 Output token throughput (tok/s): 831.38 Peak output token throughput (tok/s): 1152.00 Peak concurrent requests: 20 Total token throughput (tok/s): 6817.38 Concurrency: 13.93 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8727.66 Median E2E Latency (ms): 9170.52 P90 E2E Latency (ms): 14220.00 P99 E2E Latency (ms): 16896.54 ---------------Time to First Token---------------- Mean TTFT (ms): 282.56 Median TTFT (ms): 149.37 P99 TTFT (ms): 1278.62 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 16.60 Median TPOT (ms): 16.61 P99 TPOT (ms): 25.17 ---------------Inter-Token Latency---------------- Mean ITL (ms): 16.24 Median ITL (ms): 13.89 P95 ITL (ms): 15.96 P99 ITL (ms): 105.79 Max ITL (ms): 1065.02 ================================================== ``` ##### 5.1.2.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 93.92 Total input tokens: 1273893 Total input text tokens: 1273893 Total generated tokens: 170000 Total generated tokens (retokenized): 169999 Request throughput (req/s): 3.41 Input token throughput (tok/s): 13563.30 Output token throughput (tok/s): 1810.01 Peak output token throughput (tok/s): 2881.00 Peak concurrent requests: 71 Total token throughput (tok/s): 15373.31 Concurrency: 58.87 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 17277.69 Median E2E Latency (ms): 16827.33 P90 E2E Latency (ms): 29045.40 P99 E2E Latency (ms): 33496.77 ---------------Time to First Token---------------- Mean TTFT (ms): 692.26 Median TTFT (ms): 188.46 P99 TTFT (ms): 4932.70 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 32.19 Median TPOT (ms): 32.69 P99 TPOT (ms): 50.46 ---------------Inter-Token Latency---------------- Mean ITL (ms): 31.28 Median ITL (ms): 21.59 P95 ITL (ms): 101.35 P99 ITL (ms): 136.74 Max ITL (ms): 4649.23 ================================================== ``` #### 5.1.3 H100 Benchmark **Test Environment**: * Hardware: NVIDIA H100 80GB HBM3 GPU (8x) * Model: MiniMax-M2.5 * Tensor Parallelism: 8 * Expert Parallelism: 8 * sglang version: 0.5.9 * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2.5 \ --tp 8 \ --ep 8 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 \ --tool-call-parser minimax-m2 ``` ##### 5.1.3.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 35.44 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.28 Input token throughput (tok/s): 172.16 Output token throughput (tok/s): 119.08 Peak output token throughput (tok/s): 127.00 Peak concurrent requests: 2 Total token throughput (tok/s): 291.24 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3542.38 Median E2E Latency (ms): 2791.92 P90 E2E Latency (ms): 6317.77 P99 E2E Latency (ms): 7780.15 ---------------Time to First Token---------------- Mean TTFT (ms): 145.20 Median TTFT (ms): 80.38 P99 TTFT (ms): 633.08 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.05 Median TPOT (ms): 8.08 P99 TPOT (ms): 8.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 8.07 Median ITL (ms): 8.08 P95 ITL (ms): 8.12 P99 ITL (ms): 8.16 Max ITL (ms): 10.10 ================================================== ``` ##### 5.1.3.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 43.68 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40805 Request throughput (req/s): 1.83 Input token throughput (tok/s): 908.19 Output token throughput (tok/s): 934.22 Peak output token throughput (tok/s): 1184.00 Peak concurrent requests: 20 Total token throughput (tok/s): 1842.42 Concurrency: 13.83 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7551.91 Median E2E Latency (ms): 8094.28 P90 E2E Latency (ms): 12606.99 P99 E2E Latency (ms): 14977.84 ---------------Time to First Token---------------- Mean TTFT (ms): 116.86 Median TTFT (ms): 82.33 P99 TTFT (ms): 240.59 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.81 Median TPOT (ms): 14.98 P99 TPOT (ms): 17.98 ---------------Inter-Token Latency---------------- Mean ITL (ms): 14.61 Median ITL (ms): 13.50 P95 ITL (ms): 14.15 P99 ITL (ms): 66.52 Max ITL (ms): 107.39 ================================================== ``` ##### 5.1.3.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 80.63 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 252331 Request throughput (req/s): 6.20 Input token throughput (tok/s): 3098.45 Output token throughput (tok/s): 3133.56 Peak output token throughput (tok/s): 4800.00 Peak concurrent requests: 113 Total token throughput (tok/s): 6232.01 Concurrency: 90.56 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 14604.59 Median E2E Latency (ms): 14044.04 P90 E2E Latency (ms): 26456.53 P99 E2E Latency (ms): 30136.68 ---------------Time to First Token---------------- Mean TTFT (ms): 149.32 Median TTFT (ms): 95.16 P99 TTFT (ms): 374.62 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 28.92 Median TPOT (ms): 30.09 P99 TPOT (ms): 34.31 ---------------Inter-Token Latency---------------- Mean ITL (ms): 28.66 Median ITL (ms): 21.52 P95 ITL (ms): 66.90 P99 ITL (ms): 96.76 Max ITL (ms): 376.34 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * Benchmark Command: ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. * Test Results: ```text Output theme={null} Accuracy: 0.950 Invalid: 0.000 Latency: 18.033 s Output throughput: 1130.161 token/s ``` #### 5.2.2 MMLU Benchmark * Benchmark Command: ```shell Command theme={null} sgl-eval run mmlu \ --base-url http://127.0.0.1:30000/v1 \ --num-threads 64 --max-tokens 2048 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. * Test Results: ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.620 subject: anatomy, #q:135, acc: 0.830 subject: astronomy, #q:152, acc: 0.928 subject: business_ethics, #q:100, acc: 0.810 subject: clinical_knowledge, #q:265, acc: 0.891 subject: college_biology, #q:144, acc: 0.951 subject: college_chemistry, #q:100, acc: 0.670 subject: college_computer_science, #q:100, acc: 0.820 subject: college_mathematics, #q:100, acc: 0.660 subject: college_medicine, #q:173, acc: 0.832 subject: college_physics, #q:102, acc: 0.814 subject: computer_security, #q:100, acc: 0.880 subject: conceptual_physics, #q:235, acc: 0.915 subject: econometrics, #q:114, acc: 0.719 subject: electrical_engineering, #q:145, acc: 0.834 subject: elementary_mathematics, #q:378, acc: 0.902 subject: formal_logic, #q:126, acc: 0.698 subject: global_facts, #q:100, acc: 0.710 subject: high_school_biology, #q:310, acc: 0.926 subject: high_school_chemistry, #q:203, acc: 0.793 subject: high_school_computer_science, #q:100, acc: 0.910 subject: high_school_european_history, #q:165, acc: 0.879 subject: high_school_geography, #q:198, acc: 0.955 subject: high_school_government_and_politics, #q:193, acc: 0.964 subject: high_school_macroeconomics, #q:390, acc: 0.908 subject: high_school_mathematics, #q:270, acc: 0.600 subject: high_school_microeconomics, #q:238, acc: 0.954 subject: high_school_physics, #q:151, acc: 0.781 subject: high_school_psychology, #q:545, acc: 0.956 subject: high_school_statistics, #q:216, acc: 0.847 subject: high_school_us_history, #q:204, acc: 0.922 subject: high_school_world_history, #q:237, acc: 0.916 subject: human_aging, #q:223, acc: 0.839 subject: human_sexuality, #q:131, acc: 0.893 subject: international_law, #q:121, acc: 0.934 subject: jurisprudence, #q:108, acc: 0.861 subject: logical_fallacies, #q:163, acc: 0.890 subject: machine_learning, #q:112, acc: 0.750 subject: management, #q:103, acc: 0.883 subject: marketing, #q:234, acc: 0.944 subject: medical_genetics, #q:100, acc: 0.920 subject: miscellaneous, #q:783, acc: 0.936 subject: moral_disputes, #q:346, acc: 0.829 subject: moral_scenarios, #q:895, acc: 0.632 subject: nutrition, #q:306, acc: 0.863 subject: philosophy, #q:311, acc: 0.833 subject: prehistory, #q:324, acc: 0.907 subject: professional_accounting, #q:282, acc: 0.720 subject: professional_law, #q:1534, acc: 0.640 subject: professional_medicine, #q:272, acc: 0.923 subject: professional_psychology, #q:612, acc: 0.871 subject: public_relations, #q:110, acc: 0.773 subject: security_studies, #q:245, acc: 0.845 subject: sociology, #q:201, acc: 0.930 subject: us_foreign_policy, #q:100, acc: 0.940 subject: virology, #q:166, acc: 0.614 subject: world_religions, #q:171, acc: 0.895 Total latency: 81.468 Average accuracy: 0.825 ``` # MiniMax-M2.7 Source: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M2.7 ## 1. Model Introduction [MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) is MiniMax's first model deeply participating in its own evolution. Built for real-world productivity, M2.7 excels at building complex agent harnesses and completing highly elaborate productivity tasks, leveraging Agent Teams, complex Skills, and dynamic tool search. Key highlights: * **Model Self-Evolution**: During development, M2.7 updates its own memory, builds complex skills for RL experiments, and improves its own learning process. An internal version autonomously optimized a programming scaffold over 100+ rounds, achieving a **30% performance improvement**. On MLE Bench Lite, M2.7 achieved a **66.6% medal rate**. * **Professional Software Engineering**: Delivers outstanding real-world programming capabilities. On SWE-Pro, M2.7 achieved **56.22%**, with strong results on SWE Multilingual (76.5) and Multi SWE Bench (52.7). On Terminal Bench 2 (57.0%) and NL2Repo (39.8%), M2.7 demonstrates deep understanding of complex engineering systems. * **Professional Work**: Achieved an ELO score of **1495** on GDPval-AA (highest among open-source models). On Toolathon, M2.7 reached **46.3%** accuracy (global top tier). * **Native Agent Teams**: Supports multi-agent collaboration with stable role identity and autonomous decision-making. For more details, see the [official MiniMax-M2.7 blog post](https://www.minimax.io/news/minimax-m27-en). **License**: [Modified-MIT (MiniMax Model License)](https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE) ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). **Docker Images by Hardware Platform:**
Hardware Platform Docker Image
NVIDIA A100 / H100 / H200 / B200 `lmsysorg/sglang:v0.5.10.post1`
NVIDIA B300 / GB300 `lmsysorg/sglang:v0.5.10.post1-cu130`
AMD MI300X / MI325X `lmsysorg/sglang:v0.5.10.post1-rocm720-mi30x`
AMD MI355X `lmsysorg/sglang:v0.5.10.post1-rocm720-mi35x`
## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and feature capabilities. ### 3.2 Configuration Tips **Key Parameters:**
Parameter Description Recommended Value
`--tool-call-parser` Tool call parser for function calling support `minimax-m2`
`--reasoning-parser` Reasoning parser for thinking mode `minimax-append-think`
`--trust-remote-code` Required for MiniMax model loading Always enabled
`--mem-fraction-static` Static memory fraction for KV cache `0.85`
`--tp` Tensor parallelism size `2` / `4` / `8` depending on hardware
`--ep` Expert parallelism size `8` (NVIDIA 8-GPU) or EP=TP (AMD)
`--kv-cache-dtype` KV cache data type (AMD only) `fp8_e4m3`
`--attention-backend` Attention backend (AMD only) `triton`
**Hardware Requirements: NVIDIA** * **4-GPU deployment**: Requires 4× high-memory GPUs (e.g., H200, B200, A100, H100) with TP=4 * **8-GPU deployment**: Requires 8× GPUs (e.g., H200, B200, A100, H100) with TP=8 and EP=8 **Hardware Requirements: NVIDIA GB300** * **2-GPU deployment**: GB300 (275GB per die) can host the model with TP=2 * **4-GPU deployment**: Maximum single-node TP for GB300, recommended for higher throughput **Hardware Requirements: AMD** * **2-GPU deployment**: Requires 2× high-memory GPUs (e.g., MI300X, MI325X, MI355X) with TP=2, EP=2 * **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4 * **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8 **Hardware Requirements: Intel Xeon CPU** * It is recommended to run the model service on a Granite Rapids (GNR) AP 2-Socket server. * For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) **Deployment Command:** ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2.7 \ --tp 4 \ --tool-call-parser minimax-m2 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 ``` **Testing Deployment:** After startup, you can test the SGLang OpenAI-compatible API with the following command: ```bash Command theme={null} curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-M2.7", "messages": [ {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]} ] }' ``` **Simple Completion Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.7", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"} ], max_tokens=1024 ) print(response.choices[0].message.content) ``` **Example Output**: ```text Output theme={null} The user asks: "Who won the World Series in 2020?" That's a simple factual question. The answer: the Los Angeles Dodgers won the 2020 MLB World Series, defeating the Tampa Bay Rays. So answer accordingly. We must be mindful of policy: it's a factual question about sports. It's allowed. Provide answer with brief context. We should answer concisely. Hence final answer: The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays in six games (best-of-seven series). Possibly mention it was played at a neutral site due to COVID-19, at Globe Life Field in Arlington, Texas. We must avoid disallowed content, no issue. Thus final. The **Los Angeles Dodgers** won the 2020 World Series. They defeated the **Tampa Bay Rays** in six games (4‑2) in a best‑of‑seven series that was played at Globe Life Field in Arlington, Texas, under the MLB bubble‑like arrangements for the COVID‑19 pandemic. ``` ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser MiniMax-M2.7 supports Thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2.7 \ --tp 4 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 ``` **Streaming with Thinking Process** With `minimax-append-think`, the thinking content is wrapped in `...` tags within the `content` field. You can parse these tags on the client side to separate the thinking and content sections: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.7", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True ) # Process the stream, separating ... from content in_think = False think_printed_header = False content_printed_header = False buffer = "" for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: buffer += delta.content while buffer: if in_think: # Look for closing tag end_idx = buffer.find("") if end_idx != -1: print(buffer[:end_idx], end="", flush=True) buffer = buffer[end_idx + len(""):] in_think = False else: # Still in thinking, print what we have print(buffer, end="", flush=True) buffer = "" else: # Look for opening tag start_idx = buffer.find("") if start_idx != -1: # Print any content before before = buffer[:start_idx] if before: if not content_printed_header: print("=============== Content =================", flush=True) content_printed_header = True print(before, end="", flush=True) buffer = buffer[start_idx + len(""):] in_think = True if not think_printed_header: print("=============== Thinking =================", flush=True) think_printed_header = True else: # No tag, print as content if not content_printed_header and think_printed_header: print("\n=============== Content =================", flush=True) content_printed_header = True print(buffer, end="", flush=True) buffer = "" print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user asks: "Solve this problem step by step: What is 15% of 240?" Straightforward. Provide solution: 15% = 15/100 = 0.15. Multiply 240 * 0.15 = 36. Show steps. So answer: 36. Provide explanation. But also ensure we follow any policy? No issues. Just straightforward. I'll provide a step-by-step solution. Also could show fraction: 15% = 15/100 = 3/20, multiply 240 * 3/20 = (240/20)*3 = 12*3 = 36. Yes. Provide final answer. Also show verification: 10% of 240 is 24, 5% is 12, total 36. All good. =============== Content ================= **Step‑by‑step solution** 1. **Convert the percent to a decimal (or a fraction).** 15% = 15/100 = 0.15 = 3/20 2. **Multiply the original number (240) by this decimal/fraction.** Using the decimal: 240 × 0.15 = 36 Or using the fraction: 240 × 3/20 = (240/20) × 3 = 12 × 3 = 36 3. **Result:** 15% of 240 = **36** *Check:* - 10% of 240 = 24 - 5% of 240 = 12 - Adding them: 24 + 12 = 36, which matches the calculation. ``` **Note:** The `minimax-append-think` reasoning parser embeds the thinking process in `...` tags within the `content` field. The code above parses these tags in real-time to display thinking and content separately. #### 4.2.2 Tool Calling MiniMax-M2.7 supports tool calling capabilities. Enable the tool call parser: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2.7 \ --tp 4 \ --tool-call-parser minimax-m2 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 ``` **Python Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Non-streaming request response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.7", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools ) message = response.choices[0].message # Check for tool calls if message.tool_calls: for tool_call in message.tool_calls: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") else: print(message.content) ``` **Output Example**: ```text Output theme={null} Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.7", messages=messages ) print(final_response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} The weather in Beijing is currently 22°C and sunny. ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. **Test Environment**: * Hardware: 2× NVIDIA GB300 (275GB per die) * Docker Image: `lmsysorg/sglang:v0.5.10.post1-cu130` * Model: MiniMax-M2.7 (FP8) * Tensor Parallelism: 2 * SGLang version: 0.5.10.post1 ### 5.1 Accuracy Benchmark **Evaluation Tool**: [sgl-eval](https://github.com/sgl-project/sgl-eval) **Evaluation Settings**: temperature=0.6, top\_p=0.95, 8 repeats, max\_tokens=120,000 (GPQA and AIME; MMLU-Pro is greedy and single-pass, see below) #### 5.1.1 GPQA Diamond * Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions) * Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking)) -- sgl-eval's default for this benchmark * Evaluation command: ```bash Command theme={null} pip install sgl-eval sgl-eval run gpqa \ --base-url http://localhost:30000/v1 \ --model MiniMaxAI/MiniMax-M2.7 \ --n-repeats 8 \ --max-tokens 120000 \ --temperature 0.6 \ --top-p 0.95 ``` * Test Results:
Evaluation Mode Accuracy No Answer
pass\@1 (avg-of-8) 84.91% 3.54%
**majority\@8** **88.89%** 0.00%
pass\@8 96.46% 0.00%
#### 5.1.2 AIME 2025 * Dataset: AIME 2025 (30 problems) * Prompt: `generic/math` (boxed answer format) -- sgl-eval's default for this benchmark * Evaluation command: ```bash Command theme={null} sgl-eval run aime25 \ --base-url http://localhost:30000/v1 \ --model MiniMaxAI/MiniMax-M2.7 \ --n-repeats 8 \ --max-tokens 120000 \ --temperature 0.6 \ --top-p 0.95 ``` * Test Results:
Evaluation Mode Accuracy No Answer
pass\@1 (avg-of-8) 92.50% ± 5.56% 2.92%
**majority\@8** **97.08%** 0.00%
pass\@8 100.00% 0.00%
#### 5.1.3 MMLU-Pro * Dataset: [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (12,032 questions, 10-choice) * Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice) -- sgl-eval's default for this benchmark * Evaluation command: ```bash Command theme={null} sgl-eval run mmlu_pro \ --base-url http://localhost:30000/v1 \ --model MiniMaxAI/MiniMax-M2.7 \ --max-tokens 32768 \ --temperature 0.0 ``` * Test Results:
Evaluation Mode Accuracy No Answer
pass\@1 (greedy) 69.41% 18.75%
> **Note**: The high no-answer rate is due to the 32K token limit being insufficient for M2.7's extended thinking on some questions. A rerun with 120K tokens is expected to improve accuracy significantly. #### 5.1.4 GSM8K Benchmark * Benchmark Method: 8-shot Chain-of-Thought, evaluated via OpenAI-compatible API * Test Results: ```text Output theme={null} GSM8K Results (8-shot CoT) Model: MiniMaxAI/MiniMax-M2.7 Total: 1319 Correct: 1218 Accuracy: 92.34% ``` ### 5.2 Speed Benchmark #### 5.2.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 34.33 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.29 Input token throughput (tok/s): 177.71 Output token throughput (tok/s): 122.92 Total token throughput (tok/s): 300.63 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3431.21 Median E2E Latency (ms): 2742.57 ---------------Time to First Token---------------- Mean TTFT (ms): 50.28 Median TTFT (ms): 53.85 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.02 Median TPOT (ms): 8.01 ---------------Inter-Token Latency---------------- Mean ITL (ms): 8.03 Median ITL (ms): 8.02 ================================================== ``` #### 5.2.2 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 100.20 Total input tokens: 249831 Total generated tokens: 252662 Request throughput (req/s): 4.99 Input token throughput (tok/s): 2493.41 Output token throughput (tok/s): 2521.66 Total token throughput (tok/s): 5015.07 Concurrency: 90.19 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 18072.69 Median E2E Latency (ms): 17761.84 ---------------Time to First Token---------------- Mean TTFT (ms): 247.94 Median TTFT (ms): 92.05 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 35.75 Median TPOT (ms): 36.67 ---------------Inter-Token Latency---------------- Mean ITL (ms): 35.34 Median ITL (ms): 30.55 ================================================== ``` # MiniMax-M3 Source: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3 Deploy MiniMax-M3 with SGLang — a ~428B-param (23B activated) multimodal Mixture-of-Experts reasoning model with MiniMax Sparse Attention and 1M context, MXFP8 on NVIDIA Blackwell & AMD Instinct, bf16 on Hopper. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate # MiniMax-M3 ships in SGLang PR #27944, not yet in a tagged release — install from # the PR head. The serving runtime is in the base dependencies, so no extra is needed: git clone https://github.com/sgl-project/sglang.git cd sglang git fetch origin pull/27944/head && git checkout FETCH_HEAD uv pip install --prerelease=allow -e python ``` Then run the **Python** output of the command panel below in that environment. The **Docker** tab is simpler — its image bundles the CUDA-13 runtime and the #27944 code. Once [PR #27944](https://github.com/sgl-project/sglang/pull/27944) is merged and released, `uv pip install --prerelease=allow sglang` will pull M3 support directly. ```bash Command theme={null} # Pull the M3 image the command panel selects for your platform, e.g.: docker pull lmsysorg/sglang:dev-cu13-minimax-m3 ``` The command panel below fills in the right tag per platform: `dev-cu13-minimax-m3` (CUDA 13 — B300, GB200, GB300), `dev-cu12-minimax-m3` (CUDA 12 — Hopper H200; final build, the CUDA 12 lane is retired), or `dev-minimax-m3` (default). On AMD Instinct it uses the matching ROCm image (MI300X/MI325X → `aigmkt/minimax-m3-sglang-rocm700-mi30x`, MI350X/MI355X → `aigmkt/minimax-m3-sglang-rocm720-mi35x`). For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator produces. These M3 dev images now **bundle MiniMax's MSA sparse-attention kernel** (`fmha_sm100`), so Blackwell users get the recommended fast path automatically — no manual install needed (see **§2.1**). On a custom image without it, the same recipe still serves on the built-in Triton sparse path. Pick your hardware + recipe to generate the launch command. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction [MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) is MiniMax's native-multimodal Mixture-of-Experts reasoning model: **\~428B total parameters with \~23B activated per token** (128 experts, 4 active per token), 60 layers, and a **1M-token context** over text, image, and video. Its defining feature is **MiniMax Sparse Attention (MSA)** — a block-sparse "lightning indexer" attention that keeps long-context cost low (MiniMax reports \~9× prefill / \~15× decode speedup over M2 at 1M context). This page serves the **MXFP8** variant (`MiniMaxAI/MiniMax-M3-MXFP8`, \~440 GB) on NVIDIA Blackwell and AMD Instinct; on NVIDIA Hopper (H200), use the full-precision **bfloat16** build [`MiniMaxAI/MiniMax-M3`](https://huggingface.co/MiniMaxAI/MiniMax-M3) (§2.4). Released under the **MiniMax Community License**. Key characteristics as served by SGLang: * **Multimodal (vision + text)**: accepts interleaved text and images through the OpenAI-compatible chat API (loaded as `MiniMaxM3SparseForConditionalGeneration`). Image input via URL and base64 is validated; video input has not been tested here. * **Reasoning model**: emits its chain of thought wrapped in `...`. Always launch with **`--reasoning-parser auto`** — it auto-detects the right parser from the chat template, and SGLang then strips the tags and returns the trace separately in `message.reasoning_content`. * **Native tool calling**: a custom namespace-token XML format, parsed into standard OpenAI `tool_calls`. Always launch with **`--tool-call-parser auto`** — it auto-detects the right parser from the chat template. Single, parallel, and nested (object / array) arguments are supported. * **Sparse attention**: most layers use M3's "lightning indexer" block-sparse attention (top-k 128-token blocks), which keeps decode cost roughly flat in context length. On Blackwell, MiniMax's open-source [MSA kernel](https://github.com/MiniMax-AI/MSA) accelerates this path further (§2.1). * **MXFP8 quantization across vendors**: the MXFP8 MoE weights run natively on NVIDIA Blackwell (B200 / B300 / GB200 / GB300) and on AMD Instinct MI350X/MI355X (gfx950 / CDNA4), both of which have hardware MX-scaled matmul. On AMD MI300X/MI325X (gfx942 / CDNA3) — no hardware MX — SGLang converts the weights to block-fp8 `[128,128]` at load and serves them on the tuned ROCm kernels (§2.3). The vision tower stays unquantized. **Recommended generation**: the model's `generation_config.json` sets `temperature` 1.0 / `top_p` 0.95, which SGLang applies automatically (the default `--sampling-defaults model`). The model card additionally suggests `top_k` 40, but that value is **not** in `generation_config.json`, so SGLang does not apply it by default. `top_k` is a per-request sampling parameter (not a launch flag) — set it per call if you want it, e.g. `extra_body={"top_k": 40}` with the OpenAI client. **Resources:** [HuggingFace](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) · [MSA kernel](https://github.com/MiniMax-AI/MSA) ## 2. Configuration Tips ### 2.1 MSA sparse-attention fast path (recommended for Blackwell users) [MiniMax MSA](https://github.com/MiniMax-AI/MSA) (`fmha_sm100`, MIT-licensed) is the recommended Blackwell kernel for M3's main sparse-attention step — faster and more memory-efficient than the built-in Triton fallback. **It ships pre-installed in the M3 dev image** (`lmsysorg/sglang:dev-minimax-m3`, also published under the `dev-cu13-minimax-m3` tag), so the Blackwell recipe above engages it automatically with no extra setup — `import fmha_sm100` works out of the box and the kernels JIT-compile on first use. It is otherwise purely additive: on a custom image, install it (below) and the recipe engages it automatically; without it the same recipe still serves on the built-in Triton path. The swap is numerically equivalent (cosine ≥ 0.99999 vs Triton), decode stays CUDA-graph-capturable, prefill TTFT drops \~9–12% at 8K–64K context, and the MSA path survives memory configurations where the Triton path OOMs. **Requirements** (from the [MSA README](https://github.com/MiniMax-AI/MSA#requirements)): * **GPU**: NVIDIA SM100 family — sm\_100 (B200 / GB200) and sm\_103 (B300 / GB300). * **Toolchain**: CUDA Toolkit with `nvcc` ≥ 12.x on `PATH` (or `CUDA_HOME` set) — the kernels are JIT-compiled at first import. * **Python**: ≥ 3.10; **OS**: Linux — works on both **x86\_64 and aarch64 (Grace, e.g. GB200 / GB300)**; the aarch64 build needs no source edits. The M3 Blackwell dev images above already bundle MSA, so you can skip straight to the gate check. The `git clone` / `pip install` steps are only needed on a custom image that doesn't have `fmha_sm100`. ```bash Command theme={null} # Only on a custom image: --recursive pulls the CUTLASS submodule required for JIT compilation git clone --recursive https://github.com/MiniMax-AI/MSA.git msa cd msa && pip install . # Verify the SGLang gate (True -> MSA engaged on this device; False -> Triton fallback): python -c "from sglang.srt.layers.attention.minimax_sparse_ops.msa import msa_available; print(msa_available())" ``` The first import JIT-compiles the kernels, which can take 30 s to a few minutes on a cold `nvcc` cache — this is normal, not a hang. Subsequent server starts hit the JIT cache. **Warm the JIT cache before a multi-GPU launch.** On a *cold* cache, several tensor-parallel ranks racing to JIT-compile MSA's plan kernel can leave one rank loading a half-linked module (`AttributeError: Module has no function 'plan'` at CUDA-graph capture). Run the gate-check `python -c "..."` (or any single-process `fmha_sm100_plan` call) once before launching the server — that compiles the kernel single-process, and every rank then hits the warm cache. The gate requires `--attention-backend fa4` (MSA's sparse blocks are 128 tokens, so the page size must be 128). SGLang auto-forces `page_size` to 128 for the `fa4` backend — including the combined `--attention-backend fa4` the M3 recipe uses (#28976) — so `--page-size 128` is omitted from the Blackwell cells below. Force the Triton path at any time with the env var `SGLANG_DISABLE_MSA=1`. MSA is a Blackwell (SM100) kernel and does not apply to the AMD ROCm paths. For multimodal (image) serving, keep the same text recipe above — `--attention-backend fa4` (MSA) is unchanged — and add `--mm-attention-backend flashinfer_cudnn` for the vision tower. The text and vision-tower attention backends are independent knobs; MSA only touches the language-model sparse attention, not image handling. ### 2.2 Memory and workload tuning The NVIDIA Blackwell recipes are validated single-node: **B200 at `--tp 8`** and **B300 / GB300 at `--tp 4`** (4-GPU is also the GB200 / GB300 single-node ceiling). GB200 (sm\_100, aarch64) is inferred-supported — both of its axes are validated above (B200 is sm\_100; GB300 is sm\_103 aarch64) — but not directly benchmarked. The AMD recipes use **8-GPU (`--tp 8`)**. * **Memory**: `--mem-fraction-static` reserves GPU memory for weights + KV pool; the rest is prefill **activation headroom**. The value scales with *free* memory per GPU (card capacity minus per-GPU weight), so it tracks the card more than the TP degree: **`0.65` on B200** (180 GB — less headroom once weights are resident) and **`0.75` on the larger-memory B300 / GB300** (`0.80` on AMD). Lower TP packs more weight per GPU, so a tighter config needs a *lower* value — B200 needs `0.65` even at `--tp 4`. Raising it past the validated value is fine only for low-concurrency single-stream serving; it OOMs under high concurrency or long context. * **Long context (32K+)**: keep `--mem-fraction-static` at the platform default and raise `--chunked-prefill-size` to `16384`. Decode TPOT stays roughly flat in context length thanks to sparse attention; 1K–128K prompts are validated. * **HiSparse for decode capacity**: on NVIDIA CUDA, HiSparse keeps the three dense layers on GPU, moves the 57 sparse-layer K/V caches to pinned host memory, and feeds selected block IDs directly to the swap-in kernel. For the released four-KV-head model, use `--tp 4` or greater, `--disable-radix-cache`, and `device_buffer_size >= 2048`. Enable it with `--enable-hisparse --hisparse-config='{"device_buffer_size":4096,"host_to_device_ratio":2}'` on the Triton launch command. * **Scaling TP**: B200 is documented at `--tp 8`; B300 / GB200 / GB300 at `--tp 4` (the single-node cross-family common denominator). On an 8-GPU B300 host you can also raise to `--tp 8` for more throughput / KV headroom. * **Expert parallelism**: to trade latency for throughput add `--ep` (see [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism)). On AMD, set `--ep` equal to `--tp`. Shared-experts fusion is automatically disabled when EP > 1; on AMD standard EP the server also disables `--enable-aiter-allreduce-fusion` automatically to preserve accuracy. * `--trust-remote-code` is required to load the MiniMax config / processor classes. ### 2.3 AMD Instinct (ROCm) MiniMax-M3 runs on AMD Instinct GPUs through two code paths, by architecture — both selected automatically; you still pass `--quantization mxfp8` either way: * **MI350X / MI355X (gfx950, CDNA4)** has hardware MX-scaled matmul, so the **MXFP8 weights are served natively**. SGLang auto-detects the checkpoint, selects the Triton MiniMax-M3 MoE path with the packaged tuned MXFP8 configs, and enables AITER fused all-reduce for single-node tensor parallelism. The launch command is the NVIDIA recipe minus the Blackwell-only backend flags. * **MI300X / MI325X (gfx942, CDNA3)** has **no** hardware MX matmul. SGLang transparently **converts the MXFP8 weights to block-fp8 `[128,128]` at load time**, then serves them with the tuned ROCm block-fp8 kernels (`--attention-backend aiter`, `--moe-runner-backend triton`; the `aiter` runner also works and scores marginally higher). On a cold start the first generation can JIT-compile AITER configs and exceed the default warmup/HTTP timeout, so the recipe adds `--watchdog-timeout 3600 --skip-server-warmup`. The block-fp8 step adds only a small relative error over MXFP8's native `1×32` scaling — negligible on GSM8K (see the benchmark card). Select an MI300X/MI325X or MI350X/MI355X tile in the command panel above to get the exact launch command for each path. The AMD recipes are validated end-to-end on **text** workloads — chat, reasoning separation, and tool calling. The vision tower was not exercised on ROCm; for image input on AMD, omit the Blackwell `--mm-attention-backend flashinfer_cudnn` flag and let the encoder use the ROCm default backend, and treat vision as unvalidated on that path. ### 2.4 Serving on Hopper (H200) with the bf16 build The MXFP8 kernels are Blackwell-only, so Hopper (H200) serves the full-precision bfloat16 build [`MiniMaxAI/MiniMax-M3`](https://huggingface.co/MiniMaxAI/MiniMax-M3). Select **H200 + BF16** in the Deploy panel above for the exact command — it runs at `--tp 8` (the bf16 weights need a full 8-GPU node). SGLang picks the right backends for Hopper automatically, so the recipe stays minimal: * **MoE runner**: Triton, auto-selected for bf16 weights. * **Attention**: FlashAttention-3 with page size 1. MSA (§2.1) is a Blackwell kernel, so M3's sparse step runs on the built-in Triton path here. * **CUDA graph**: on, with full decode-graph capture. **High-concurrency throughput (optional).** On Hopper the sparse prefill runs on the Triton path as a separate eager forward, which briefly stalls the in-flight decode batch under heavy concurrent load. Adding `--enable-mixed-chunk --chunked-prefill-size 2048` merges the running decodes into the prefill step instead of preempting them, which recovers roughly **+10% output throughput** and **\~10% lower median TPOT** at high concurrency on 8×H200, with no change in accuracy. Leave it off for latency-sensitive low-concurrency serving. Validated on 8×H200 — reasoning and tool-call auto-detection plus long-context generation. For prefill/decode disaggregation on Hopper, see §3.4. ## 3. Advanced Usage ### 3.1 Reasoning Launch with `--reasoning-parser auto` (or toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)). The `` trace then lands in `message.reasoning_content`, separate from the final answer in `message.content` — no client-side tag stripping needed. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M3-MXFP8", messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}], max_tokens=2048, ) message = response.choices[0].message print("=============== Reasoning ===============") print(message.reasoning_content) print("=============== Answer ==================") print(message.content) ``` ```text Output theme={null} =============== Reasoning =============== 15% of 240. 15% = 0.15. 240 * 0.15 = 36. Quick check: 10% is 24, 5% is 12, 24 + 12 = 36. =============== Answer ================== 15% of 240 is **36**. (10% of 240 = 24, and 5% of 240 = 12; 24 + 12 = 36.) ``` When streaming, the trace arrives on `delta.reasoning_content` and the answer on `delta.content`, so the two sections can be rendered separately in real time: ```python Example theme={null} response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M3-MXFP8", messages=[{"role": "user", "content": "Solve step by step: what is 15% of 240?"}], max_tokens=2048, stream=True, ) for chunk in response: if not chunk.choices: continue delta = chunk.choices[0].delta if getattr(delta, "reasoning_content", None): print(delta.reasoning_content, end="", flush=True) # thinking stream if delta.content: print(delta.content, end="", flush=True) # answer stream print() ``` **Output Example:** ```text Output theme={null} [delta.reasoning_content — thinking stream] Let me solve this step by step. 15% of 240 = 0.15 × 240 = 36 Let me verify: 10% of 240 = 24, 5% of 240 = 12, so 15% = 24 + 12 = 36. ✓ [delta.content — answer stream] # Solving 15% of 240 ## Step 1: Convert the percentage to a decimal 15% = 15/100 = 0.15 ## Step 2: Multiply by 240 0.15 × 240 = 36 ## Answer **15% of 240 = 36** ``` ### 3.2 Tool Calling Launch with `--tool-call-parser auto` (or toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) — it auto-detects M3's tool-call parser from the chat template. M3 emits tool calls in a custom namespace-token XML format: ```text Raw model output theme={null} ]<]minimax[>[ ]<]minimax[>[]<]minimax[>[Beijing]<]minimax[>[]<]minimax[>[ ]<]minimax[>[ ``` The parser converts that into the standard OpenAI `tool_calls` structure: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M3-MXFP8", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) message = response.choices[0].message if message.tool_calls: for call in message.tool_calls: print(f"Tool: {call.function.name}") print(f"Args: {call.function.arguments}") ``` ```text Output theme={null} Tool: get_weather Args: {"location": "Beijing"} ``` Beyond a single flat call, the parser also supports: * **Parallel calls** — multiple `` blocks inside the single `` wrapper, surfaced as multiple `message.tool_calls` entries. * **Nested object arguments** — an `object`-typed parameter is emitted as nested XML tags and reconstructed into a JSON object. * **Array arguments** — an `array`-typed parameter uses repeated `` children and is reconstructed into a JSON list. For example, a tool with object and array parameters round-trips cleanly: ```text Output theme={null} create_event {"title": "Design sync", "attendees": ["alice", "bob"], "location": {"room": "R2", "floor": 3}} ``` To return a tool result, append the assistant's `tool_calls` turn plus a matching `tool` message and ask the model to continue — the follow-up answer may place text in `reasoning_content` as well as `content`, so print both. ### 3.3 Multimodal (Vision) Input Images go through the standard OpenAI `image_url` content type. The vision tower is always loaded; for image serving add `--mm-attention-backend flashinfer_cudnn` (the vision-tower backend) to the Blackwell deployment recipe — the text `--attention-backend` is unchanged (§2.1 note). On AMD, omit `--mm-attention-backend` and let the encoder use the ROCm default (vision is unvalidated on ROCm — §2.3). ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M3-MXFP8", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png" }, }, {"type": "text", "text": "Describe this image in detail."}, ], } ], max_tokens=1024, ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} This image captures a striking and unusual urban scene on what appears to be a busy New York City street. **Main Subject:** A man stands on the rear bumper of a yellow taxi cab (an SUV-style cab, likely a Ford Escape hybrid), operating a full-sized ironing board set up across the back of the vehicle. He is wearing a bright yellow long-sleeved shirt and dark pants, and is actively ironing a blue garment, holding an iron in his right hand. **Vehicles:** - The yellow SUV taxi on the right is stationary, its rear hatch serving as the ironing platform. - A second yellow taxi (a sedan) drives past on the left, captured with motion blur. **Setting:** Tall city buildings with classic urban architecture, an American flag, and white lane markings — a bustling downtown area, possibly Midtown Manhattan. ``` Notes: * If the server cannot fetch external URLs, embed the image as a base64 `data:image/png;base64,...` URI — SGLang decodes it server-side. * Multiple images per message are supported; add more `image_url` entries to the `content` list. * Reasoning and tool calling work the same way for multimodal requests — a vision prompt can still produce a `` trace and/or tool calls. ### 3.4 Prefill-Decode (PD) Disaggregation [PD disaggregation](../../../docs/advanced_features/pd_disaggregation) runs prefill and decode on **separate** SGLang servers linked by an RDMA KV-transfer fabric (mooncake or NIXL), fronted by the PD router. M3 needs one thing beyond a dense model: alongside the main KV cache, every sparse "lightning-indexer" layer keeps a **K-only index buffer**, and that buffer must reach the decode server too — otherwise sparse attention reads stale state. SGLang transfers it alongside the main KV — reusing the same page mapping — so M3 disaggregates correctly with no extra flags. **Supported topology** (the released MiniMax-M3, whose sparse layers are all K-only): * **Equal tensor parallelism** — the prefill and decode servers run the same `--tp`. * **Single pipeline stage** — PP = 1 (the default). * **mooncake or NIXL** transfer backend over RDMA / InfiniBand. Launch the prefill server, then the decode server — the same recipe with `--disaggregation-mode decode` and no bootstrap port. Pick your hardware: On Blackwell the MXFP8 recipe — fa4, page size 128, deep\_gemm MoE, and the MSA fast path (§2.1) — is auto-selected, so each role adds only the `--disaggregation-*` flags. This is the validated **2 × 4×B200** setup (TP4 prefill on node A, TP4 decode on node B); point `--disaggregation-ib-device` at your RDMA NIC(s). ```bash Prefill server (node A) theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M3-MXFP8 \ --trust-remote-code \ --reasoning-parser auto \ --tool-call-parser auto \ --tp 4 \ --disaggregation-mode prefill \ --disaggregation-transfer-backend nixl \ --disaggregation-ib-device mlx5_0 \ --host 0.0.0.0 --port 30000 \ --disaggregation-bootstrap-port 8998 ``` ```bash Decode server (node B) theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M3-MXFP8 \ --trust-remote-code \ --reasoning-parser auto \ --tool-call-parser auto \ --tp 4 \ --disaggregation-mode decode \ --disaggregation-transfer-backend nixl \ --disaggregation-ib-device mlx5_0 \ --host 0.0.0.0 --port 30001 ``` On Hopper (H200) M3 runs the bf16 build (§2.4) with Triton MoE and the built-in Triton sparse path, pinned to `--page-size 128` so both roles share the page layout the sparse-index transfer relies on. This is the validated **2 × 8×H200** setup (TP8 each). ```bash Prefill server (node A) theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M3 \ --trust-remote-code \ --reasoning-parser auto \ --tool-call-parser auto \ --tp 8 \ --attention-backend triton \ --moe-runner-backend triton \ --page-size 128 \ --disaggregation-mode prefill \ --disaggregation-transfer-backend mooncake \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \ --host 0.0.0.0 --port 30000 \ --disaggregation-bootstrap-port 8998 ``` ```bash Decode server (node B) theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M3 \ --trust-remote-code \ --reasoning-parser auto \ --tool-call-parser auto \ --tp 8 \ --attention-backend triton \ --moe-runner-backend triton \ --page-size 128 \ --disaggregation-mode decode \ --disaggregation-transfer-backend mooncake \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \ --host 0.0.0.0 --port 30001 ``` Then start the PD router, pointing it at the prefill bootstrap (URL plus its `--disaggregation-bootstrap-port`) and the decode endpoint: ```bash PD router theme={null} python3 -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:30000 8998 \ --decode http://:30001 \ --policy round_robin \ --host 0.0.0.0 --port 8000 ``` Clients hit the router exactly like a single server — it splits each request across the two stages transparently: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M3-MXFP8", messages=[{"role": "user", "content": "What is 2 + 2?"}], max_tokens=64, ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} 2 + 2 = 4 ``` **Validation.** PD disaggregation preserves output quality — the K-only sparse index transfers arrive intact and disaggregated output matches non-disaggregated serving. GSM8K is scored with the single sgl-eval harness used by the benchmark card above (full 1319-question split, chat with `--thinking`); see that card for per-platform single-node accuracy. * **2 × 4×B200** (TP4+TP4, MXFP8, NIXL over InfiniBand) — output matches single-node serving. The 2-node PD serving benchmark (512-token input, 256-token output, 16 concurrent — a different workload from the card's single-node `random` isl=2048 / osl=256 / conc=64 row, so the throughput figures are not directly comparable) measured mean TTFT 1.1 s and TPOT 16.6 ms (≈ 60 tok/s per stream, ≈ 2.3k tok/s aggregate). * **2 × 8×H200** (TP8+TP8, bf16, mooncake) — output matches single-node serving. # Devstral 2 (Mistral) Source: https://docs.sglang.io/cookbook/autoregressive/Mistral/Devstral-2 ## 1. Model Introduction **Devstral 2** is an agentic LLM family for software engineering tasks. It is designed for agentic workflows such as tool use, codebase exploration, and multi-file edits, and achieves strong performance on **SWE-bench**. The **Devstral 2 Instruct** checkpoints are instruction-tuned **FP8** models, making them a good fit for chat, tool-using agents, and instruction-following SWE workloads. **Key Features:** * **Agentic coding**: Optimized for tool-driven coding and software engineering agents * **Improved performance**: A step up compared to earlier Devstral models * **Better generalization**: More robust across diverse prompts and coding environments * **Long context**: Up to a **256K** context window **Use Cases:** AI code assistants, agentic coding, and software engineering tasks that require deep codebase understanding and tool integration. For enterprises requiring specialized capabilities (increased context, domain-specific knowledge, etc.), please reach out to Mistral. **Models:** * **Collection**: [mistralai/devstral-2 (Hugging Face)](https://huggingface.co/collections/mistralai/devstral-2) * **FP8 Instruct**: * **[mistralai/Devstral-2-123B-Instruct-2512](https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512)** * **[mistralai/Devstral-Small-2-24B-Instruct-2512](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512)** *** ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Devstral 2 requires a recent `transformers`. Please verify `transformers >= 5.0.0.rc`: ```shell Command theme={null} python -c "import transformers; print(transformers.__version__)" ``` If your version is lower, upgrade: ```shell Command theme={null} pip install -U --pre "transformers>=5.0.0rc0" ``` *** ## 3. Model Deployment ### 3.1 Basic configuration **Interactive Command Generator**: Use the configuration selector below to generate a launch command for Devstral Small 2 (24B) or Devstral 2 (123B). The TP size is set to the minimum required for the selected model size. ### 3.2 Configuration tips * **Context length vs memory**: Devstral 2 advertises a long context window; if you are memory-constrained, start by lowering `--context-length` (for example `32768`) and increase once things are stable. * **FP8 checkpoints**: Both Devstral Small 2 and Devstral 2 are published as **FP8** weights. If you hit kernel / dtype issues, try a newer SGLang build and recent CUDA drivers. *** ## 4. Model Invocation ### 4.1 Basic Usage (OpenAI-Compatible API) SGLang exposes an OpenAI-compatible endpoint. Example: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="mistralai/Devstral-Small-2-24B-Instruct-2512", messages=[ {"role": "system", "content": "You are a helpful coding assistant."}, {"role": "user", "content": "Write a Python function that retries a request with exponential backoff."}, ], temperature=0.2, max_tokens=512, ) print(resp.choices[0].message.content) ``` **Output Example:** ````text Output theme={null} Here's a Python function that implements exponential backoff for retrying a request. This function uses the `requests` library to make HTTP requests and includes error handling for common HTTP and connection errors. ```python import time import requests from requests.exceptions import RequestException def retry_with_exponential_backoff( url, max_retries=3, initial_delay=1, backoff_factor=2, method="GET", **kwargs ): """ Retry a request with exponential backoff. Parameters: - url: The URL to request. - max_retries: Maximum number of retry attempts (default: 3). - initial_delay: Initial delay in seconds (default: 1). - backoff_factor: Multiplier for the delay between retries (default: 2). - method: HTTP method to use (default: "GET"). - **kwargs: Additional arguments to pass to the request function (e.g., headers, data, etc.). Returns: - Response object if the request succeeds. - Raises an exception if all retries fail. """ retry_count = 0 delay = initial_delay while retry_count < max_retries: try: response = requests.request(method, url, **kwargs) # Check if the response status code indicates success if response.status_code < 400: return response else: raise RequestException(f"HTTP {response.status_code}: {response.text}") except RequestException as e: if retry_count == max_retries - 1: raise Exception(f"All retries failed. Last error: {e}") print(f"Attempt {retry_count + 1} failed. Retrying in {delay} seconds...") time.sleep(delay) ... ```` ### 4.2 Tool calling (optional) Devstral 2 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model mistralai/Devstral-2-123B-Instruct-2512 \ --tp 2 \ --tool-call-parser mistral ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="mistralai/Devstral-2-123B-Instruct-2512", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} 🔧 Tool Call: get_weather Arguments: {"location": "Beijing"} ``` ## AMD GPU Support ## 1. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 1.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 1.2 Advanced Usage ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path mistralai/Devstral-2-123B-Instruct-2512 \ --tp 8 \ --trust-remote-code \ --port 8888 ``` ## 2.Benchmark ### 5.1 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path mistralai/Devstral-2-123B-Instruct-2512 \ --tp 8 \ --trust-remote-code \ --port 8888 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model mistralai/Devstral-2-123B-Instruct-2512 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf \ --port 8888 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 94.30 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4206 Request throughput (req/s): 0.11 Input token throughput (tok/s): 64.70 Output token throughput (tok/s): 44.75 Peak output token throughput (tok/s): 82.00 Peak concurrent requests: 2 Total token throughput (tok/s): 109.44 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 9427.59 Median E2E Latency (ms): 5637.23 ---------------Time to First Token---------------- Mean TTFT (ms): 4253.85 Median TTFT (ms): 116.95 P99 TTFT (ms): 37764.48 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 12.28 Median TPOT (ms): 12.29 P99 TPOT (ms): 12.30 ---------------Inter-Token Latency---------------- Mean ITL (ms): 12.29 Median ITL (ms): 12.29 P95 ITL (ms): 12.38 P99 ITL (ms): 12.42 Max ITL (ms): 12.90 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model mistralai/Devstral-2-123B-Instruct-2512 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf \ --port 8888 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 52.11 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40761 Request throughput (req/s): 1.54 Input token throughput (tok/s): 761.31 Output token throughput (tok/s): 783.13 Peak output token throughput (tok/s): 1120.00 Peak concurrent requests: 20 Total token throughput (tok/s): 1544.44 Concurrency: 13.60 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8856.19 Median E2E Latency (ms): 9314.71 ---------------Time to First Token---------------- Mean TTFT (ms): 398.80 Median TTFT (ms): 127.81 P99 TTFT (ms): 1500.32 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 17.32 Median TPOT (ms): 16.90 P99 TPOT (ms): 32.78 ---------------Inter-Token Latency---------------- Mean ITL (ms): 16.61 Median ITL (ms): 14.26 P95 ITL (ms): 15.07 P99 ITL (ms): 114.46 Max ITL (ms): 1224.45 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model mistralai/Devstral-2-123B-Instruct-2512 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf \ --port 8888 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 116.08 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 252523 Request throughput (req/s): 4.31 Input token throughput (tok/s): 2152.21 Output token throughput (tok/s): 2176.60 Peak output token throughput (tok/s): 3600.00 Peak concurrent requests: 107 Total token throughput (tok/s): 4328.81 Concurrency: 92.42 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 21456.71 Median E2E Latency (ms): 20126.82 ---------------Time to First Token---------------- Mean TTFT (ms): 291.60 Median TTFT (ms): 199.24 P99 TTFT (ms): 866.02 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 42.42 Median TPOT (ms): 45.18 P99 TPOT (ms): 53.32 ---------------Inter-Token Latency---------------- Mean ITL (ms): 41.97 Median ITL (ms): 27.59 P95 ITL (ms): 130.43 P99 ITL (ms): 137.87 Max ITL (ms): 616.73 ================================================== ``` #### 5.2 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.3 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.3.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:8888/v1 \ --num-examples 1319 \ --num-threads 1316 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. **Test Results:** ```text Output theme={null} Accuracy: 0.922 Invalid: 0.000 Latency: 35.800 s Output throughput: 4507.697 token/s ``` # Ministral-3 Source: https://docs.sglang.io/cookbook/autoregressive/Mistral/Ministral-3 ## 1. Model Introduction The largest model in the Ministral 3 family, Ministral 3 14B offers frontier capabilities and performance comparable to its larger Mistral Small 3.2 24B counterpart. A powerful and efficient language model with vision capabilities. The Ministral 3 14B Instruct model offers the following capabilities: Vision: Enables the model to analyze images and provide insights based on visual content, in addition to text. Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Arabic. System Prompt: Maintains strong adherence and support for system prompts. Agentic: Offers best-in-class agentic capabilities with native function calling and JSON outputting. Edge-Optimized: Delivers best-in-class performance at a small scale, deployable anywhere. Apache 2.0 License: Open-source license allowing usage and modification for both commercial and non-commercial purposes. Large Context Window: Supports a 256k context window. For further details, please refer to the [official documentation](https://github.com/mistralai) ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips **Context length vs memory**: Ministral-3 advertises a long context window; if you are memory-constrained, start by lowering --context-length (for example 32768) and increase once things are stable. **Pre-installation steps**: Adding the following steps after launching the docker ```shell Command theme={null} pip install mistral-common --upgrade pip install transformers==5.0.0.rc0 ``` ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Launch the docker ```shell Command theme={null} docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x ``` ```shell Command theme={null} docker run -d -it --ipc=host --network=host --privileged \ --cap-add=CAP_SYS_ADMIN \ --device=/dev/kfd --device=/dev/dri --device=/dev/mem \ --group-add video --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v /:/work \ -e SHELL=/bin/bash \ --name Ministral \ lmsysorg/sglang:v0.5.9-rocm720-mi30x \ /bin/bash ``` #### 4.2.2 Launch the server ```shell Command theme={null} sglang serve \ --model-path mistralai/Ministral-3-14B-Instruct-2512 \ --tp 1 \ --trust-remote-code ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: MI300X GPU (8x) * Model: mistralai/Ministral-3-14B-Instruct-2512 * Tensor Parallelism: 1 * SGLang Version: 0.5.7 * Model Deployment Command: ```bash Command theme={null} sglang serve \ --model-path mistralai/Ministral-3-14B-Instruct-2512 \ --tp 1 \ --trust-remote-code ``` ##### Low Concurrency * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model mistralai/Ministral-3-14B-Instruct-2512 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 65.08 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4218 Request throughput (req/s): 0.15 Input token throughput (tok/s): 93.75 Output token throughput (tok/s): 64.84 Peak output token throughput (tok/s): 151.00 Peak concurrent requests: 2 Total token throughput (tok/s): 158.59 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6505.51 Median E2E Latency (ms): 3037.37 ---------------Time to First Token---------------- Mean TTFT (ms): 3709.33 Median TTFT (ms): 53.72 P99 TTFT (ms): 33320.77 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.63 Median TPOT (ms): 6.64 P99 TPOT (ms): 6.66 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.64 Median ITL (ms): 6.65 P95 ITL (ms): 6.75 P99 ITL (ms): 6.82 Max ITL (ms): 8.45 ================================================== ``` ##### Medium Concurrency * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model mistralai/Ministral-3-14B-Instruct-2512 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 31.20 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40783 Request throughput (req/s): 2.56 Input token throughput (tok/s): 1271.38 Output token throughput (tok/s): 1307.82 Peak output token throughput (tok/s): 1760.00 Peak concurrent requests: 22 Total token throughput (tok/s): 2579.20 Concurrency: 13.72 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5351.07 Median E2E Latency (ms): 5626.45 ---------------Time to First Token---------------- Mean TTFT (ms): 280.87 Median TTFT (ms): 68.16 P99 TTFT (ms): 1194.79 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.47 Median TPOT (ms): 10.10 P99 TPOT (ms): 20.00 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.96 Median ITL (ms): 9.10 P95 ITL (ms): 9.87 P99 ITL (ms): 51.39 Max ITL (ms): 888.63 ================================================== ``` ##### High Concurrency * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model mistralai/Ministral-3-14B-Instruct-2512 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 88.75 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 252547 Request throughput (req/s): 5.63 Input token throughput (tok/s): 2815.01 Output token throughput (tok/s): 2846.91 Peak output token throughput (tok/s): 4271.00 Peak concurrent requests: 110 Total token throughput (tok/s): 5661.93 Concurrency: 93.04 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16514.45 Median E2E Latency (ms): 15834.45 ---------------Time to First Token---------------- Mean TTFT (ms): 148.57 Median TTFT (ms): 99.15 P99 TTFT (ms): 455.86 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 32.93 Median TPOT (ms): 34.73 P99 TPOT (ms): 38.05 ---------------Inter-Token Latency---------------- Mean ITL (ms): 32.45 Median ITL (ms): 27.30 P95 ITL (ms): 71.73 P99 ITL (ms): 73.45 Max ITL (ms): 328.10 ================================================== ``` ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 1319 \ --num-threads 1316 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. **Test Results:** ```text Output theme={null} Accuracy: 0.959 Invalid: 0.000 Latency: 29.185 s Output throughput: 4854.672 token/s ``` # Mistral Medium 3.5 Source: https://docs.sglang.io/cookbook/autoregressive/Mistral/Mistral-Medium-3.5 ## 1. Model Introduction **Mistral Medium 3.5** is Mistral AI's first flagship **merged model** — a single dense 128B checkpoint that handles instruction following, reasoning, and coding in one set of weights. It replaces Mistral Medium 3.1 and Magistral in Le Chat, and replaces Devstral 2 in the Vibe coding agent. Reasoning effort is configurable per request, so the same model can answer a quick chat reply or work through a deep agentic run. The vision encoder was trained from scratch to handle variable image sizes and aspect ratios. **Key Features:** * **Dense 128B parameters** — no MoE, no MLA, plain GQA (96 heads, 8 KV heads, head\_dim=128) * **256K context window** — YARN RoPE scaling on top of the original 4K base * **Hybrid Reasoning**: Toggle between instant reply and deep reasoning per request via `reasoning_effort` (`"none"` or `"high"`) * **Vision**: Accepts text + image input; from-scratch encoder that handles variable image sizes/aspect ratios * **Function Calling**: Native tool calling and JSON output * **FP8 Native**: Released with FP8 e4m3 static-tensor quantization built in * **Multilingual**: 24 supported languages including English, French, German, Spanish, Portuguese, Italian, Japanese, Korean, Russian, Chinese, Arabic, Persian, Indonesian, Malay, Nepali, Polish, Romanian, Serbian, Swedish, Turkish, Ukrainian, Vietnamese, Hindi, and Bengali * **License**: Modified MIT (open for commercial and non-commercial use except for companies with large revenue) **Architecture:** * Mistral 3 backbone with YARN RoPE for 256K context * Dense (no MoE), 128B parameters * Standard GQA attention (not MLA) * Pixtral-style vision encoder (48 layers, patch\_size=14, spatial\_merge=2, image\_size=1540) trained from scratch * Multimodal input: text + image **Models:** * **[mistralai/Mistral-Medium-3.5-128B](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B)** (FP8) The HuggingFace repo ships both the mistral native layout (`params.json` + `consolidated-*.safetensors`) and the HF layout (`config.json` + `model-*.safetensors`). SGLang auto-detects the format — the HF layout is preferred when both are present. *** ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install). **Docker Image:** `lmsysorg/sglang:latest` covers all the GPUs in this cookbook (H100 / H200 / B200 / B300). *** ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Medium 3.5. ### 3.2 Configuration Tips * **Tensor Parallelism**: Mistral Medium 3.5 FP8 (\~130 GB) requires `--tp 4` on Hopper (H100/H200) and `--tp 2` on Blackwell (B200/B300). * **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call. * **Recommended temperature**: `0.7` when `reasoning_effort="high"`. Anywhere from `0.0` to `0.7` when `reasoning_effort="none"`, depending on the task — lower for to-the-point answers, higher for creative output. * **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable. * **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support. * **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content. * **System prompt**: The model ships with a recommended system prompt in `chat_template.jinja` and `SYSTEM_PROMPT.txt`. If you do not pass a system message yourself, the chat template injects Mistral's default (model identity, current date, tool-use guidelines). For full fidelity with Mistral's reference setup, load `SYSTEM_PROMPT.txt` from the HF repo and substitute `{name}`, `{today}`, `{yesterday}` (see Section 4.6). ### 3.3 Speculative Decoding (EAGLE) Mistral ships an EAGLE draft head, [`mistralai/Mistral-Medium-3.5-128B-EAGLE`](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B-EAGLE), that lets you run speculative decoding on top of the dense 128B target. The draft is a 2-layer GQA body sharing the target's vocab/head, FP8-quantized like the target (\~4 GB), and is meant for low-concurrency latency-bound serving. ```bash Command theme={null} python -m sglang.launch_server \ --model-path mistralai/Mistral-Medium-3.5-128B \ --tp 4 \ --dtype bfloat16 \ --tool-call-parser mistral \ --reasoning-parser mistral \ --speculative-algorithm EAGLE \ --speculative-draft-model-path mistralai/Mistral-Medium-3.5-128B-EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --port 30000 ``` * **`--dtype bfloat16` is required.** The draft `params.json` does not carry a `dtype` field, so `--dtype auto` falls back to fp32 and downcasts to fp16, which conflicts with the bf16 target when the embed/head are shared. Setting bf16 explicitly keeps both sides aligned (this is a no-op for the target — it already loads as bf16). * The draft uses the same vocab and lm\_head as the target. Memory overhead on top of the base model is \~4 GB per TP shard. * `(num-steps, eagle-topk, num-draft-tokens) = (3, 1, 4)` is the recommended starting point. Tune for your workload — wider trees (higher `eagle-topk` / `num-draft-tokens`) help high-acceptance (templated) outputs, narrower trees keep latency tight on more diverse text. * EAGLE shines at low concurrency. At high concurrency, throughput is dominated by the target's batched forward pass and the draft's contribution shrinks; consider running without EAGLE for batch-serving workloads. *** ## 4. Model Invocation ### 4.1 Thinking Mode Mistral Medium 3.5 is a hybrid reasoning model. By default it does not produce a reasoning trace — pass `reasoning_effort="high"` to switch on the deep-reasoning path. Mistral recommends `temperature=0.7` for reasoning mode. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="mistralai/Mistral-Medium-3.5-128B", messages=[ {"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"}, ], temperature=0.7, extra_body={"reasoning_effort": "high"}, ) print("Reasoning:", response.choices[0].message.reasoning_content) print("Answer:", response.choices[0].message.content) ``` **Output:** ```text Output theme={null} Reasoning: I need to follow the order of operations (PEMDAS/BODMAS): multiplication and division before addition, evaluated left to right. 17 × 23: I'll break it as 17 × (20 + 3) = 340 + 51 = 391. 144 / 12 = 12. Finally, 391 + 12 = 403. Answer: **17 × 23 + 144 / 12 = 403** Step by step: 1. 17 × 23 = 391 2. 144 / 12 = 12 3. 391 + 12 = 403 ``` ### 4.2 Instruct Mode (Reasoning Off) To skip the reasoning trace and get a fast direct response, set `reasoning_effort="none"`. For instruct mode, Mistral recommends temperature in the `0.0`–`0.7` range depending on how creative the task is: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="mistralai/Mistral-Medium-3.5-128B", messages=[ {"role": "user", "content": "What is the capital of France?"}, ], temperature=0.1, extra_body={"reasoning_effort": "none"}, ) print(response.choices[0].message.content) ``` **Output:** ```text Output theme={null} The capital of France is **Paris**. It is one of the most famous and visited cities in the world, known for its rich history, art, culture, and landmarks like the Eiffel Tower, Louvre Museum, and Notre-Dame Cathedral. ``` ### 4.3 Streaming with Reasoning ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) stream = client.chat.completions.create( model="mistralai/Mistral-Medium-3.5-128B", messages=[ {"role": "user", "content": "Explain the difference between async and threading in Python."}, ], temperature=0.7, extra_body={"reasoning_effort": "high"}, stream=True, ) print("=== Reasoning ===") for chunk in stream: delta = chunk.choices[0].delta if hasattr(delta, "reasoning_content") and delta.reasoning_content: print(delta.reasoning_content, end="", flush=True) elif delta.content: print("\n=== Response ===") print(delta.content, end="", flush=True) print() ``` ### 4.4 Tool Calling Mistral Medium 3.5 supports native function calling. Enable with `--tool-call-parser mistral`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="mistralai/Mistral-Medium-3.5-128B", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=tools, tool_choice="auto", ) tool_calls = response.choices[0].message.tool_calls for tc in tool_calls: print(f"Tool: {tc.function.name}") print(f"Args: {tc.function.arguments}") ``` **Output:** ```text Output theme={null} Tool: get_weather Args: {"location": "Paris"} ``` ### 4.5 Vision (Image Input) Mistral Medium 3.5 accepts image inputs alongside text. The vision encoder was retrained from scratch to handle variable image sizes and aspect ratios: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="mistralai/Mistral-Medium-3.5-128B", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Describe what you see in this image."}, { "type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"}, }, ], } ], temperature=0.7, extra_body={"reasoning_effort": "none"}, ) print(response.choices[0].message.content) ``` **Output:** ```text Output theme={null} The image features a stylized representation of the acronym "SGL." The letters are large, bold, and orange with a brown outline, giving them a three-dimensional effect. To the left of the letters, there is a graphic that resembles a neuron or a node with connections, also in a similar orange and brown color scheme. The node has a code symbol () inside a square, suggesting a connection to programming or technology. ``` ### 4.6 Loading the Reference System Prompt Mistral ships a `SYSTEM_PROMPT.txt` alongside the weights. The reference setup loads it from the HF repo and substitutes `{name}`, `{today}`, and `{yesterday}` at runtime so the model knows its identity and the current date. SGLang's chat template will inject a default system prompt if you omit one, but for full parity with Mistral's reference, load it explicitly: ```python Example theme={null} from datetime import datetime, timedelta from huggingface_hub import hf_hub_download from openai import OpenAI MODEL = "mistralai/Mistral-Medium-3.5-128B" def load_system_prompt(repo_id: str, filename: str = "SYSTEM_PROMPT.txt") -> str: path = hf_hub_download(repo_id=repo_id, filename=filename) today = datetime.today().strftime("%Y-%m-%d") yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d") name = repo_id.split("/")[-1] with open(path) as f: return f.read().format(name=name, today=today, yesterday=yesterday) client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model=MODEL, messages=[ {"role": "system", "content": load_system_prompt(MODEL)}, {"role": "user", "content": "Write me a sentence where every word starts with the next letter in the alphabet — start with 'a' and end with 'z'."}, ], temperature=0.1, extra_body={"reasoning_effort": "none"}, ) print(response.choices[0].message.content) ``` *** ## 5. Benchmarks Validation runs on 4× H200 with `--tp 4`, served via the `/v1/chat/completions` endpoint. ### 5.1 Accuracy Benchmarks #### GSM8K ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. **Results:** ```text Output theme={null} Accuracy: 0.945 Invalid: 0.000 Latency: 13.594 s Output throughput: 1560.660 token/s ``` #### MMMU ```bash Command theme={null} python3 benchmark/mmmu/bench_sglang.py --port 30000 ``` **Results:** ```text Output theme={null} Overall accuracy: 0.586 ``` ### 5.2 Speed Benchmarks #### Latency (Low Concurrency) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 10 \ --max-concurrency 1 \ --random-input-len 1024 \ --random-output-len 512 \ --port 30000 ``` **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Successful requests: 10 Benchmark duration (s): 38.86 Total input tokens: 6101 Total generated tokens: 2684 Output token throughput (tok/s): 69.07 Mean E2E Latency (ms): 3883.80 Median TTFT (ms): 95.90 Median TPOT (ms): 14.19 ================================================== ``` #### Throughput (High Concurrency) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 1000 \ --max-concurrency 100 \ --random-input-len 1024 \ --random-output-len 512 \ --port 30000 ``` **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Successful requests: 1000 Benchmark duration (s): 117.28 Total input tokens: 512842 Total generated tokens: 262023 Output token throughput (tok/s): 2234.18 Total token throughput (tok/s): 6607.01 Mean E2E Latency (ms): 11303.79 Median TTFT (ms): 152.95 Median TPOT (ms): 42.53 ================================================== ``` ### 5.3 EAGLE Speculative Decoding (Latency) Same 4× H200 setup, EAGLE configuration from [Section 3.3](#3-3-speculative-decoding-eagle). Single-stream latency benchmark (`--max-concurrency 1`). ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 10 \ --max-concurrency 1 \ --random-input-len 1024 \ --random-output-len 512 \ --port 30000 ``` **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Successful requests: 10 Benchmark duration (s): 27.64 Total input tokens: 6101 Total generated tokens: 2684 Output token throughput (tok/s): 97.10 Mean E2E Latency (ms): 2762.99 Median TTFT (ms): 90.69 Median TPOT (ms): 9.73 Accept length: 1.72 ================================================== ``` EAGLE delivers **\~1.41× output throughput and \~29% lower E2E latency** vs. the baseline in [Section 5.2](#5-2-speed-benchmarks) on the same workload. Acceptance length of 1.72 means each draft cycle averages roughly 1.7 accepted tokens. # Mistral Small 4 Source: https://docs.sglang.io/cookbook/autoregressive/Mistral/Mistral-Small-4 ## 1. Model Introduction **Mistral Small 4** is a powerful hybrid model from Mistral AI that unifies the capabilities of three different model families — **Instruct**, **Reasoning** (formerly called Magistral), and **Agentic (formerly called Devstral)** — into a single, unified model. With its multimodal capabilities, efficient MoE architecture, and flexible mode switching, Mistral Small 4 is a versatile general-purpose model for virtually any task. In a latency-optimized setup, it achieves a 40% reduction in end-to-end completion time; in a throughput-optimized setup, it delivers 3× more requests per second compared to Mistral Small 3. **Key Features:** * **Hybrid Reasoning**: Switch between instant reply mode and deep reasoning/thinking mode — reasoning effort is configurable per request * **Vision**: Accepts both text and image inputs, providing insights based on visual content * **Function Calling**: Native tool calling and JSON output support with best-in-class agentic capabilities * **Multilingual**: Supports dozens of languages including English, French, Spanish, German, Chinese, Japanese, Korean, Arabic, and more * **Context Window**: 256K context window * **Efficient MoE**: 119B total parameters, 128 experts, 4 active per token (6.5B activated parameters) * **Apache 2.0 License**: Open-source, usable and modifiable for commercial and non-commercial purposes * Reasoning effort supported are only **"none" and "high"** **Architecture:** * Same general architecture as Mistral 3 * MoE: 128 experts, 4 active per token * 119B total parameters, 6.5B activated per token * Multimodal input: text + image **Models:** * **[mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603)** (FP8) * **[mistralai/Mistral-Small-4-119B-2603-NVFP4](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4)** * **[mistralai/Leanstral-2603](https://huggingface.co/mistralai/Leanstral-2603)** — same architecture, use the same launch commands as Mistral-Small-4-119B-2603 * **[mistralai/Mistral-Small-4-119B-2603-eagle](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle)** — EAGLE speculative decoding weights for faster inference *** ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Mistral Small 4 support landed in [sgl-project/sglang#20708](https://github.com/sgl-project/sglang/pull/20708) and has been merged into `main`. A model-specific Docker image is no longer required. Use the standard SGLang installation methods from the [official installation guide](../../../docs/get-started/install). *** ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Small 4. ### 3.2 Configuration Tips * **Tensor Parallelism**: Mistral Small 4 FP8 (\~119 GB) requires tp=2 on Hopper (H100/H200), tp=1 on Blackwell (B200/B300). NVFP4 (\~60 GB, Blackwell only) runs with tp=1. * **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call. * **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable. * **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support. * **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content. * **Speculative decoding (EAGLE)**: Enable with `--speculative-algorithm EAGLE --speculative-draft-model-path mistralai/Mistral-Small-4-119B-2603-eagle` using the [EAGLE weights](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle) for lower latency. *** ## 4. Model Invocation ### 4.1 Thinking Mode Mistral Small 4 is a hybrid reasoning model. By default, it does not produce a default reasoning response. Use `--reasoning_effort high` to toggle reasoning on. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="mistralai/Mistral-Small-4-119B-2603", messages=[ {"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"}, ], extra_body={"reasoning_effort": "high"}, ) print("Reasoning:", response.choices[0].message.reasoning_content) print("Answer:", response.choices[0].message.content) ``` **Output:** ```text Output theme={null} Reasoning: First, I'll break down the problem into two parts: the multiplication and the division. According to the order of operations (PEMDAS/BODMAS), multiplication and division are performed from left to right before addition. 17 × 23 = 17 × (20 + 3) = (17 × 20) + (17 × 3) = 340 + 51 = 391 144 / 12 = 12 Finally, add the results: 391 + 12 = 403 Answer: The solution to the problem is as follows: 1. First, perform the multiplication: 17 × 23. - 17 × 20 = 340 - 17 × 3 = 51 - 340 + 51 = 391 2. Then, perform the division: 144 / 12 = 12. 3. Finally, add the results: - 391 + 12 = 403 **Answer:** \boxed{403} ``` ### 4.2 Instruct Mode (Reasoning Off) To skip the reasoning trace and get a fast direct response, set `reasoning_effort` to `"none"`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="mistralai/Mistral-Small-4-119B-2603", messages=[ {"role": "user", "content": "Write a Python function to reverse a string."}, ], extra_body={"reasoning_effort": "none"}, ) print(response.choices[0].message.content) ``` **Output:** ````text Output theme={null} # Python Function to Reverse a String Here are several ways to write a Python function to reverse a string: ## Method 1: Using String Slicing (Most Pythonic) ```python def reverse_string(s): """Reverse a string using slicing.""" return s[::-1] ``` ## Method 2: Using a Loop ```python Example def reverse_string(s): """Reverse a string using a loop.""" reversed_str = "" for char in s: reversed_str = char + reversed_str return reversed_str ``` ## Method 3: Using reversed() function ```python Example def reverse_string(s): """Reverse a string using reversed() function.""" return ''.join(reversed(s)) ``` The first method using string slicing (`s[::-1]`) is generally the most efficient and recommended approach in Python. Example usage: ```python Example original = "Hello, World!" reversed_str = reverse_string(original) print(reversed_str) # Output: "!dlroW ,olleH" ``` ```` ### 4.3 Streaming with Reasoning ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) stream = client.chat.completions.create( model="mistralai/Mistral-Small-4-119B-2603", messages=[ {"role": "user", "content": "Explain the difference between async and threading in Python."}, ], extra_body={"reasoning_effort": "high"}, stream=True, ) print("=== Reasoning ===") for chunk in stream: delta = chunk.choices[0].delta if hasattr(delta, "reasoning_content") and delta.reasoning_content: print(delta.reasoning_content, end="", flush=True) elif delta.content: print("\n=== Response ===") print(delta.content, end="", flush=True) print() ``` **Output:** ```text Output theme={null} === Reasoning === Okay, the user is asking about the difference between async and threading in Python. I need to break this down clearly, covering the key aspects of both, like their purposes, performance characteristics, and use cases... === Response === In Python, **`async`/`asyncio`** and **`threading`** are two different concurrency models, each suited for specific use cases. Here's a breakdown of their key differences: ### 1. Model of Concurrency - **Threading**: Based on preemptive multitasking using OS threads. - **Async** (`asyncio`): Based on cooperative multitasking. Tasks voluntarily yield... ``` ### 4.4 Tool Calling Mistral Small 4 supports native function calling. Enable with `--tool-call-parser mistral`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="mistralai/Mistral-Small-4-119B-2603", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=tools, tool_choice="auto", ) tool_calls = response.choices[0].message.tool_calls for tc in tool_calls: print(f"Tool: {tc.function.name}") print(f"Args: {tc.function.arguments}") ``` **Output:** ```text Output theme={null} Tool: get_weather Args: {"location": "Paris"} ``` ### 4.5 Vision (Image Input) Mistral Small 4 accepts image inputs alongside text: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="mistralai/Mistral-Small-4-119B-2603", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Describe what you see in this image."}, { "type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"}, }, ], } ], ) print(response.choices[0].message.content) ``` **Output:** ```text Output theme={null} The image is a copyright symbol, represented by a stylized version of the lowercase letter "c" inside a circle. The "c" is depicted in a white or light-colored font, and the circle is orange. The design is simple yet striking, using oval and elliptical shapes to create a distinct symbol which signifies copyright protection. ``` *** ## 5. Benchmarks ### 5.1 Accuracy Benchmarks #### GSM8K ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. **Results:** ```text Output theme={null} TODO ``` #### MMLU ```bash Command theme={null} sgl-eval run mmlu \ --base-url http://127.0.0.1:30000/v1 \ --num-threads 64 --max-tokens 2048 ``` **Results:** ```text Output theme={null} TODO ``` ### 5.2 Speed Benchmarks #### Latency (Low Concurrency) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --num-prompts 10 \ --max-concurrency 1 \ --random-input-len 1024 \ --random-output-len 512 \ --port 30000 ``` **Results:** ```text Output theme={null} TODO ``` #### Throughput (High Concurrency) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --num-prompts 1000 \ --max-concurrency 100 \ --random-input-len 1024 \ --random-output-len 512 \ --port 30000 ``` **Results:** ```text Output theme={null} TODO ``` # Kimi-K2 Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K2 ## 1. Model Introduction [Kimi-K2](https://moonshotai.github.io/Kimi-K2/) is a state-of-the-art MoE language model by Moonshot AI with 32B activated parameters and 1T total parameters. **Model Variants:** * **[Kimi-K2-Instruct](https://huggingface.co/moonshotai/Kimi-K2-Instruct)**: Post-trained model optimized for general-purpose chat and agentic tasks. Compatible with vLLM, SGLang, KTransformers, and TensorRT-LLM. * **[Kimi-K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking)**: Advanced thinking model with step-by-step reasoning and tool calling. Native INT4 quantization with 256k context window. Ideal for complex reasoning and multi-step tool use. * **ROCm Support**: Compatible with AMD MI300X GPUs via SGLang (verified). For details, see [official documentation](https://github.com/MoonshotAI/Kimi-K2) and [technical report](https://www.arxiv.org/abs/2507.20534). ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and capabilities. ### 3.2 Configuration Tips * **Memory**: Requires 8 GPUs with ≥140GB each (H200/B200). Use `--context-length 128000` to conserve memory. * **Expert Parallelism (EP)**: Use `--ep` for better MoE throughput. See [EP docs](../../../docs/advanced_features/expert_parallelism). * **Data Parallel (DP)**: Enable with `--attn-dp-size 4` for production throughput. * **KV Cache**: Use `--kv-cache-dtype fp8_e4m3` to reduce memory by 50% (CUDA 11.8+). * **Reasoning Parser**: Add `--reasoning-parser kimi_k2` for Kimi-K2-Thinking to separate thinking and content. * **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls. * **AMD GPU**: Set `SGLANG_ROCM_FUSED_DECODE_MLA=0` before launching AMD GPU. ## 4. Model Invocation ### 4.1 Basic Usage See [Basic API Usage](../../../docs/get-started/quickstart). ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser Enable reasoning parser for Kimi-K2-Thinking: ```shell Command theme={null} python -m sglang.launch_server \ --model moonshotai/Kimi-K2-Thinking \ --reasoning-parser kimi_k2 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="moonshotai/Kimi-K2-Thinking", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.6, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user asks: "What is 15% of 240?" This is a straightforward percentage calculation problem. I need to solve it step by step. Step 1: Understand what "percent" means. - "Percent" means "per hundred". So 15% means 15 per 100, or 15/100, or 0.15. Step 2: Convert the percentage to a decimal. - 15% = 15 / 100 = 0.15 Step 3: Multiply the decimal by the number. - 0.15 * 240 Step 4: Perform the multiplication. - 0.15 * 240 = (15/100) * 240 - = 15 * 240 / 100 - = 3600 / 100 - = 36 Alternatively, I can calculate it directly: - 0.15 * 240 - 15 * 240 = 3600 - 3600 / 100 = 36 Or, break it down: - 10% of 240 = 24 - 5% of 240 = half of 10% = 12 - 15% of 240 = 10% + 5% = 24 + 12 = 36 I should present the solution clearly with steps. The most standard method is converting to decimal and multiplying. Let me structure the answer: 1. Convert the percentage to a decimal. 2. Multiply the decimal by the number. 3. Show the calculation. 4. State the final answer. This is simple and easy to follow. =============== Content ================= Here is the step-by-step solution: **Step 1: Convert the percentage to a decimal** 15% means 15 per 100, which is 15 ÷ 100 = **0.15** **Step 2: Multiply the decimal by the number** 0.15 × 240 **Step 3: Calculate the result** 0.15 × 240 = **36** **Answer:** 15% of 240 is **36**. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling Kimi-K2-Instruct and Kimi-K2-Thinking support tool calling capabilities. Enable the tool call parser during deployment: **Deployment Command:** ```shell Command theme={null} python -m sglang.launch_server \ --model moonshotai/Kimi-K2-Instruct \ --tool-call-parser kimi_k2 \ --tp 8 \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="moonshotai/Kimi-K2-Thinking", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. Beijing is a major city in China, so I should be able to get weather data for it. The location parameter is required, but the unit parameter is optional. Since the user didn't specify a temperature unit, I can just provide the location and let the function use its default. I'll check the weather in Beijing for you. =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location":"Beijing"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="moonshotai/Kimi-K2-Thinking", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (8x) * Model: Kimi-K2-Instruct * sglang version: 0.5.6.post1 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path moonshotai/Kimi-K2-Instruct \ --tp 8 \ --attn-dp-size 4 \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 8000 \ --model moonshotai/Kimi-K2-Instruct\ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 44.93 Total input tokens: 1951 Total input text tokens: 1951 Total input vision tokens: 0 Total generated tokens: 2755 Total generated tokens (retokenized): 2748 Request throughput (req/s): 0.22 Input token throughput (tok/s): 43.42 Output token throughput (tok/s): 61.32 Peak output token throughput (tok/s): 64.00 Peak concurrent requests: 3 Total token throughput (tok/s): 104.74 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4489.56 Median E2E Latency (ms): 4994.53 ---------------Time to First Token---------------- Mean TTFT (ms): 141.22 Median TTFT (ms): 158.28 P99 TTFT (ms): 166.90 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 18.40 Median TPOT (ms): 15.63 P99 TPOT (ms): 39.88 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.78 Median ITL (ms): 15.76 P95 ITL (ms): 16.36 P99 ITL (ms): 16.59 Max ITL (ms): 19.94 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path moonshotai/Kimi-K2-Instruct \ --tp 8 \ --attn-dp-size 4 \ --ep 4 \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 8000 \ --model moonshotai/Kimi-K2-Instruct\ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 174.11 Total input tokens: 296642 Total input text tokens: 296642 Total input vision tokens: 0 Total generated tokens: 193831 Total generated tokens (retokenized): 168687 Request throughput (req/s): 5.74 Input token throughput (tok/s): 1703.73 Output token throughput (tok/s): 1113.25 Peak output token throughput (tok/s): 2383.00 Peak concurrent requests: 112 Total token throughput (tok/s): 2816.97 Concurrency: 89.60 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 15601.09 Median E2E Latency (ms): 10780.52 ---------------Time to First Token---------------- Mean TTFT (ms): 457.42 Median TTFT (ms): 221.62 P99 TTFT (ms): 2475.32 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 97.23 Median TPOT (ms): 85.61 P99 TPOT (ms): 435.95 ---------------Inter-Token Latency---------------- Mean ITL (ms): 78.61 Median ITL (ms): 43.66 P95 ITL (ms): 169.53 P99 ITL (ms): 260.91 Max ITL (ms): 1703.21 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * Server Command ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path moonshotai/Kimi-K2-Instruct \ --tp 8 \ --dp 4 \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:8000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * **Result**: ```text Output theme={null} Accuracy: 0.960 Invalid: 0.000 Latency: 15.956 s Output throughput: 1231.699 token/s ``` # Kimi-K2.5 Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K2.5 ## 1. Model Introduction [Kimi-K2.5](https://huggingface.co/moonshotai/Kimi-K2.5) is an open-source, native multimodal agentic model by Moonshot AI, built through continual pretraining on approximately 15 trillion mixed visual and text tokens atop Kimi-K2-Base. It seamlessly integrates vision and language understanding with advanced agentic capabilities, instant and thinking modes. **Key Features:** * **Native Multimodality**: Pre-trained on vision-language tokens, K2.5 excels in visual knowledge, cross-modal reasoning, and agentic tool use grounded in visual inputs. * **Coding with Vision**: K2.5 generates code from visual specifications (UI designs, video workflows) and autonomously orchestrates tools for visual data processing. * **Agent Swarm**: K2.5 transitions from single-agent scaling to a self-directed, coordinated swarm-like execution scheme. It decomposes complex tasks into parallel sub-tasks executed by dynamically instantiated, domain-specific agents. * **Speculative Decoding**: EAGLE-based speculative decoding support for lower latency. **Available Models**: * INT4 (Initial Released): [moonshotai/Kimi-K2.5](https://huggingface.co/moonshotai/Kimi-K2.5) * NVFP4 (4-bit quantized): [nvidia/Kimi-K2.5-NVFP4](https://huggingface.co/nvidia/Kimi-K2.5-NVFP4) For details, see [official documentation](https://huggingface.co/moonshotai/Kimi-K2.5) and [deployment guidance](https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/docs/deploy_guidance.md). ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and capabilities. ### 3.2 Configuration Tips * **Memory**: Requires GPUs with >=140GB each. Supported platforms: H200 (8x, TP=8), B300 (8x, TP=8), GB300 (4x, TP=4), MI300X/MI325X (4x, TP=4), MI350X/MI355X (4x, TP=4). Use `--context-length 128000` to conserve memory. * **AMD GPU TP Constraint**: On AMD GPUs, TP must be \<= 4 (not 8). Kimi-K2.5 has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid). * **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X. The ROCm 7.2 images (`rocm720`) have an AITER compatibility issue. * **DP Attention**: Enable with `--attn-dp-size ` for production throughput. A common choice is to set `--attn-dp-size` equal to `--tp`, but this is not required. * **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs. * **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls. ## 4. Model Invocation ### 4.1 Basic Usage See [Basic API Usage](../../../docs/basic_usage/send_request). ### 4.2 Advanced Usage #### 4.2.1 Multimodal (Vision + Text) Input Kimi-K2.5 supports native multimodal input with images: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "What is in this image? Describe it in detail." } ] } ] ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} This image shows a **receipt from Auntie Anne's** (a pretzel franchise restaurant). ## Key Details: **Item Purchased:** - **CINNAMON SUGAR** - 1 unit x 17,000 = **17,000** **Payment Summary:** - **SUB TOTAL:** 17,000 - **GRAND TOTAL:** 17,000 - **CASH IDR:** 20,000 (Indonesian Rupiah) - **CHANGE DUE:** 3,000 ## Context: The receipt indicates a transaction in **Indonesian Rupiah (IDR)**. A customer purchased one Cinnamon Sugar pretzel for 17,000 IDR, paid with a 20,000 IDR note, and received 3,000 IDR in change. The top of the receipt shows the Auntie Anne's logo (a heart-shaped pretzel with a halo), and some text appears blurred for privacy, likely obscuring the store location, date, and transaction number. The receipt is printed on white thermal paper. ``` #### 4.2.2 Reasoning Output Kimi-K2.5 supports both thinking mode (default) and instant mode. **Thinking Mode (default)** -- reasoning content is automatically separated: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=[ {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."} ] ) print("====== Reasoning Content (Thinking Mode) ======") print(response.choices[0].message.reasoning_content) print("====== Response (Thinking Mode) ======") print(response.choices[0].message.content) ``` **Instant Mode (thinking off)** -- disable thinking for faster responses: ```python Example theme={null} response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=[ {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."} ], extra_body={"chat_template_kwargs": {"thinking": False}} ) print("====== Response (Instant Mode) ======") print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} ====== Reasoning Content (Thinking Mode) ====== The user is asking which number is bigger: 9.11 or 9.9. At first glance, someone might think 9.11 is bigger because 11 > 9, but that's incorrect because we're dealing with decimal numbers, not whole numbers. Let me compare them properly: - 9.9 = 9.90 - 9.11 When comparing decimals, we look at each place value from left to right: - Units place: 9 = 9 (tie) - Tenths place: 9 vs 1 Since 9 > 1, we have 9.9 > 9.11. Alternatively, we can think of it as: - 9.9 = 9 + 9/10 = 9 + 0.9 = 9.90 - 9.11 = 9 + 11/100 = 9 + 0.11 Since 0.90 > 0.11, then 9.9 > 9.11. So the answer is clearly 9.9 is bigger. The "think carefully" hint suggests the user is trying to catch the common error where people compare 11 and 9 as whole numbers rather than understanding decimal place value (tenths vs hundredths). I should explain this clearly to avoid confusion. ====== Response (Thinking Mode) ====== **9.9 is bigger.** Here's why this can be tricky: Many people instinctively compare 11 and 9 and think "11 is bigger than 9," but that's comparing the wrong place values. When comparing decimals, align them by place value: - 9.9 = 9.**90** - 9.11 = 9.**11** After the decimal point: - The first digit (tenths place): **9** vs **1** - Since 9 > 1, we stop there. **9.9 is larger.** Think of it as money: - $9.90 (nine dollars and ninety cents) - $9.11 (nine dollars and eleven cents) $9.90 is clearly more than $9.11. ====== Response (Instant Mode) ====== Let me think through this carefully. **9.9 is bigger than 9.11** Here's why: When comparing decimals, we need to align them by their decimal places: - 9.9 = 9.90 - 9.11 = 9.11 Now comparing: - The whole number parts are equal (9 = 9) - Comparing tenths: **9 > 1** So 9.90 > 9.11 A common mistake is thinking 11 hundredths is larger than 9 tenths, but 9 tenths = 90 hundredths, which is clearly larger than 11 hundredths. ``` #### 4.2.3 Tool Calling Kimi-K2.5 supports tool calling capabilities for agentic tasks: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) # Process streaming response tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'tool_calls') and delta.tool_calls: for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = {'name': None, 'arguments': ''} if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments if delta.content: print(delta.content, end="", flush=True) for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") ``` **Output Example:** ```text Output theme={null} Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Handling Tool Call Results:** ```python Example theme={null} # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": "The weather in Beijing is 22°C and sunny." } ] final_response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=messages ) print(final_response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} The weather in Beijing is **22°C and sunny**. ☀️ It's a nice day there with comfortable temperatures and clear skies! ``` #### 4.2.4 Multimodal + Tool Calling (Agentic Vision) Combine vision understanding with tool calling for advanced agentic tasks: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "search_product", "description": "Search for a product by name or description", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "The product name or description to search for" } }, "required": ["query"] } } } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Can you identify this product and search for similar items?" } ] } ], tools=tools ) msg = response.choices[0].message # Print reasoning process if msg.reasoning_content: print("=== Reasoning ===") print(msg.reasoning_content) # Print response content if msg.content: print("=== Content ===") print(msg.content) # Print tool calls if msg.tool_calls: print("=== Tool Calls ===") for tc in msg.tool_calls: print(f" Function: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` **Output Example:** ```text Output theme={null} === Reasoning === The user is asking me to identify a product from a receipt and search for similar items. Looking at the receipt, I can see: 1. The store is "Auntie Anne's" - which is a popular pretzel chain 2. The product purchased is "CINNAMON SUGAR" 3. Price is 17,000 (likely Indonesian Rupiah based on "CASH IDR") 4. Quantity is 1 So the product is a Cinnamon Sugar pretzel from Auntie Anne's. Now I need to search for this product or similar items using the search_product function. === Content === I can see from the receipt that the product is a **Cinnamon Sugar** item from **Auntie Anne's** (the famous pretzel chain). This appears to be a Cinnamon Sugar Pretzel purchased for 17,000 IDR (Indonesian Rupiah). Let me search for this product and similar items: === Tool Calls === Function: search_product Arguments: {"query": "Auntie Anne's Cinnamon Sugar Pretzel"} ``` #### 4.2.5 Speculative Decoding **Nvidia** Deploy Kimi-K2.5 with the following command (H200/B300, all features enabled): ```shell Command theme={null} sglang serve \ --model-path moonshotai/Kimi-K2.5 \ --tp 8 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm=EAGLE3 \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-path lightseekorg/kimi-k2.5-eagle3-mla \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` Deploy Kimi-K2.5-NVFP4 with the following command (B300, all features enabled): ```shell Command theme={null} sglang serve \ --model-path nvidia/Kimi-K2.5-NVFP4 \ --tp 8 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --kv-cache-dtype fp8_e4m3 \ --speculative-algorithm=EAGLE3 \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-path lightseekorg/kimi-k2.5-eagle3-mla \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` For GB300, use `--tp 4`. ## 5. Benchmark ### 5.1 Accuracy Benchmark #### 5.1.1 MMMU Benchmark You can evaluate the model's accuracy using the MMMU benchmark, which tests multimodal understanding and reasoning across various subjects: * **Benchmark Command:** ```shell Command theme={null} python3 benchmark/mmmu/bench_sglang.py \ --response-answer-regex "(?i)(?:answer|ans)[:\s]*(?:\*\*)?[\(\[]?([A-Za-z])[\)\]]?(?:\*\*)?" \ --port 30000 \ --concurrency 64 ``` * **Result:** ```text Output theme={null} Benchmark time: 2785.4322692090645 answers saved to: ./answer_sglang.json Evaluating... answers saved to: ./answer_sglang.json {'Accounting': {'acc': 0.667, 'num': 30}, 'Agriculture': {'acc': 0.567, 'num': 30}, 'Architecture_and_Engineering': {'acc': 0.733, 'num': 30}, 'Art': {'acc': 0.833, 'num': 30}, 'Art_Theory': {'acc': 0.8, 'num': 30}, 'Basic_Medical_Science': {'acc': 0.833, 'num': 30}, 'Biology': {'acc': 0.6, 'num': 30}, 'Chemistry': {'acc': 0.633, 'num': 30}, 'Clinical_Medicine': {'acc': 0.733, 'num': 30}, 'Computer_Science': {'acc': 0.667, 'num': 30}, 'Design': {'acc': 0.7, 'num': 30}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.5, 'num': 30}, 'Economics': {'acc': 0.867, 'num': 30}, 'Electronics': {'acc': 0.3, 'num': 30}, 'Energy_and_Power': {'acc': 0.767, 'num': 30}, 'Finance': {'acc': 0.833, 'num': 30}, 'Geography': {'acc': 0.667, 'num': 30}, 'History': {'acc': 0.767, 'num': 30}, 'Literature': {'acc': 0.767, 'num': 30}, 'Manage': {'acc': 0.733, 'num': 30}, 'Marketing': {'acc': 0.833, 'num': 30}, 'Materials': {'acc': 0.567, 'num': 30}, 'Math': {'acc': 0.633, 'num': 30}, 'Mechanical_Engineering': {'acc': 0.567, 'num': 30}, 'Music': {'acc': 0.5, 'num': 30}, 'Overall': {'acc': 0.698, 'num': 900}, 'Overall-Art and Design': {'acc': 0.708, 'num': 120}, 'Overall-Business': {'acc': 0.787, 'num': 150}, 'Overall-Health and Medicine': {'acc': 0.74, 'num': 150}, 'Overall-Humanities and Social Science': {'acc': 0.75, 'num': 120}, 'Overall-Science': {'acc': 0.66, 'num': 150}, 'Overall-Tech and Engineering': {'acc': 0.595, 'num': 210}, 'Pharmacy': {'acc': 0.767, 'num': 30}, 'Physics': {'acc': 0.767, 'num': 30}, 'Psychology': {'acc': 0.667, 'num': 30}, 'Public_Health': {'acc': 0.867, 'num': 30}, 'Sociology': {'acc': 0.8, 'num': 30}} eval out saved to ./val_sglang.json Overall accuracy: 0.698 ``` ### 5.2 Speed Benchmark **Test Environment:** * Hardware: NVIDIA H200 GPU (8x) * Model: Kimi-K2.5 * Tensor Parallelism: 8 * SGLang Version: 0.5.6.post2 We use SGLang's built-in benchmarking tool with the `random` dataset for standardized performance evaluation. #### 5.2.1 Latency Benchmark * **Model Deployment:** ```bash Command theme={null} sglang serve \ --model-path moonshotai/Kimi-K2.5 \ --tp 8 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` * **Benchmark Command:** ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 39.77 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4221 Request throughput (req/s): 0.25 Input token throughput (tok/s): 153.40 Output token throughput (tok/s): 106.10 Peak output token throughput (tok/s): 156.00 Peak concurrent requests: 2 Total token throughput (tok/s): 259.50 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3972.87 Median E2E Latency (ms): 4044.55 P90 E2E Latency (ms): 7046.30 P99 E2E Latency (ms): 7441.13 ---------------Time to First Token---------------- Mean TTFT (ms): 176.89 Median TTFT (ms): 154.24 P99 TTFT (ms): 285.75 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.22 Median TPOT (ms): 9.32 P99 TPOT (ms): 12.72 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.02 Median ITL (ms): 8.80 P95 ITL (ms): 13.23 P99 ITL (ms): 14.17 Max ITL (ms): 29.38 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 158.05 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40775 Request throughput (req/s): 0.51 Input token throughput (tok/s): 250.99 Output token throughput (tok/s): 258.18 Peak output token throughput (tok/s): 1103.00 Peak concurrent requests: 19 Total token throughput (tok/s): 509.17 Concurrency: 14.09 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 27837.05 Median E2E Latency (ms): 23508.00 P90 E2E Latency (ms): 57126.31 P99 E2E Latency (ms): 66044.35 ---------------Time to First Token---------------- Mean TTFT (ms): 374.30 Median TTFT (ms): 375.51 P99 TTFT (ms): 695.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 53.25 Median TPOT (ms): 57.93 P99 TPOT (ms): 85.45 ---------------Inter-Token Latency---------------- Mean ITL (ms): 53.95 Median ITL (ms): 53.97 P95 ITL (ms): 84.74 P99 ITL (ms): 244.84 Max ITL (ms): 655.61 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 996.64 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 252588 Request throughput (req/s): 0.50 Input token throughput (tok/s): 250.67 Output token throughput (tok/s): 253.51 Peak output token throughput (tok/s): 1199.00 Peak concurrent requests: 104 Total token throughput (tok/s): 504.18 Concurrency: 92.70 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 184773.75 Median E2E Latency (ms): 174183.65 P90 E2E Latency (ms): 343625.28 P99 E2E Latency (ms): 404284.53 ---------------Time to First Token---------------- Mean TTFT (ms): 1289.59 Median TTFT (ms): 1313.35 P99 TTFT (ms): 2346.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 364.70 Median TPOT (ms): 403.32 P99 TPOT (ms): 452.34 ---------------Inter-Token Latency---------------- Mean ITL (ms): 363.82 Median ITL (ms): 316.21 P95 ITL (ms): 745.91 P99 ITL (ms): 1345.88 Max ITL (ms): 3118.59 ================================================== ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 680.26 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 44462 Total generated tokens (retokenized): 44455 Request throughput (req/s): 0.01 Input token throughput (tok/s): 8.97 Output token throughput (tok/s): 65.36 Peak output token throughput (tok/s): 151.00 Peak concurrent requests: 2 Total token throughput (tok/s): 74.33 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 68019.29 Median E2E Latency (ms): 70568.85 P90 E2E Latency (ms): 113237.40 P99 E2E Latency (ms): 121682.34 ---------------Time to First Token---------------- Mean TTFT (ms): 206.17 Median TTFT (ms): 177.28 P99 TTFT (ms): 445.37 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.36 Median TPOT (ms): 15.89 P99 TPOT (ms): 16.43 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.26 Median ITL (ms): 15.85 P95 ITL (ms): 17.50 P99 ITL (ms): 23.21 Max ITL (ms): 45.22 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 2475.98 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 318306 Total generated tokens (retokenized): 318166 Request throughput (req/s): 0.03 Input token throughput (tok/s): 16.02 Output token throughput (tok/s): 128.56 Peak output token throughput (tok/s): 847.00 Peak concurrent requests: 18 Total token throughput (tok/s): 144.58 Concurrency: 14.62 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 452592.46 Median E2E Latency (ms): 486002.05 P90 E2E Latency (ms): 833197.57 P99 E2E Latency (ms): 957399.48 ---------------Time to First Token---------------- Mean TTFT (ms): 359.38 Median TTFT (ms): 350.78 P99 TTFT (ms): 500.36 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 111.18 Median TPOT (ms): 122.76 P99 TPOT (ms): 145.90 ---------------Inter-Token Latency---------------- Mean ITL (ms): 113.69 Median ITL (ms): 122.81 P95 ITL (ms): 147.87 P99 ITL (ms): 151.03 Max ITL (ms): 272.05 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} Waiting for completion... ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 120.73 Total input tokens: 41941 Total input text tokens: 41941 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.08 Input token throughput (tok/s): 347.41 Output token throughput (tok/s): 34.96 Peak output token throughput (tok/s): 73.00 Peak concurrent requests: 2 Total token throughput (tok/s): 382.36 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 12068.56 Median E2E Latency (ms): 10211.36 P90 E2E Latency (ms): 23203.32 P99 E2E Latency (ms): 30677.66 ---------------Time to First Token---------------- Mean TTFT (ms): 1625.64 Median TTFT (ms): 1526.63 P99 TTFT (ms): 3743.51 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 24.95 Median TPOT (ms): 23.95 P99 TPOT (ms): 35.40 ---------------Inter-Token Latency---------------- Mean ITL (ms): 24.80 Median ITL (ms): 21.73 P95 ITL (ms): 59.56 P99 ITL (ms): 61.10 Max ITL (ms): 62.70 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 389.96 Total input tokens: 300020 Total input text tokens: 300020 Total generated tokens: 41669 Total generated tokens (retokenized): 41670 Request throughput (req/s): 0.21 Input token throughput (tok/s): 769.36 Output token throughput (tok/s): 106.86 Peak output token throughput (tok/s): 304.00 Peak concurrent requests: 19 Total token throughput (tok/s): 876.22 Concurrency: 14.95 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 72870.97 Median E2E Latency (ms): 70495.88 P90 E2E Latency (ms): 121820.46 P99 E2E Latency (ms): 148933.09 ---------------Time to First Token---------------- Mean TTFT (ms): 2460.45 Median TTFT (ms): 1976.29 P99 TTFT (ms): 7305.53 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 140.57 Median TPOT (ms): 142.31 P99 TPOT (ms): 273.40 ---------------Inter-Token Latency---------------- Mean ITL (ms): 135.44 Median ITL (ms): 95.96 P95 ITL (ms): 152.93 P99 ITL (ms): 1488.37 Max ITL (ms): 6540.24 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 1279.50 Total input tokens: 1273893 Total input text tokens: 1273893 Total generated tokens: 170000 Total generated tokens (retokenized): 169981 Request throughput (req/s): 0.25 Input token throughput (tok/s): 995.62 Output token throughput (tok/s): 132.86 Peak output token throughput (tok/s): 703.00 Peak concurrent requests: 67 Total token throughput (tok/s): 1128.49 Concurrency: 60.12 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 240385.63 Median E2E Latency (ms): 236266.30 P90 E2E Latency (ms): 429882.12 P99 E2E Latency (ms): 515158.36 ---------------Time to First Token---------------- Mean TTFT (ms): 2710.44 Median TTFT (ms): 2345.63 P99 TTFT (ms): 7144.20 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 443.84 Median TPOT (ms): 493.29 P99 TPOT (ms): 606.19 ---------------Inter-Token Latency---------------- Mean ITL (ms): 448.23 Median ITL (ms): 296.17 P95 ITL (ms): 1869.15 P99 ITL (ms): 2708.95 Max ITL (ms): 7778.47 ================================================== ``` #### 5.2.2 Speculative Decoding Benchmark * **Model Deployment:** ```bash Command theme={null} sglang serve \ --model-path moonshotai/Kimi-K2.5 \ --tp 8 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm=EAGLE3 \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-path lightseekorg/kimi-k2.5-eagle3-mla \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` * **Benchmark Command:** ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Results:** ```text Output theme={null} Pending update... ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} Pending update... ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} Pending update... ``` ### 5.3 Speed Benchmark (AMD MI350X) **Test Environment:** * Hardware: AMD Instinct MI350X GPU (4x) * Model: Kimi-K2.5 (BF16) * Tensor Parallelism: 4 * SGLang Version: 0.5.9 * Docker Image: `lmsysorg/sglang:v0.5.9-rocm700-mi35x` * ROCm: 7.0 We use SGLang's built-in benchmarking tool with the `random` dataset for standardized performance evaluation. :::info AMD GPU TP Constraint Kimi-K2.5 requires TP \<= 4 on AMD GPUs. The model has 64 attention heads, and the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid). ::: #### 5.3.1 Latency Benchmark * **Model Deployment:** ```bash Command theme={null} SGLANG_USE_AITER=1 SGLANG_ROCM_FUSED_DECODE_MLA=0 \ sglang serve \ --model-path moonshotai/Kimi-K2.5 \ --tp 4 \ --mem-fraction-static 0.8 \ --trust-remote-code \ --reasoning-parser kimi_k2 \ --host 0.0.0.0 \ --port 30000 ``` * **Benchmark Command:** ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 155.81 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4222 Request throughput (req/s): 0.06 Input token throughput (tok/s): 39.16 Output token throughput (tok/s): 27.09 Peak output token throughput (tok/s): 29.00 Peak concurrent requests: 2 Total token throughput (tok/s): 66.24 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 15576.22 Median E2E Latency (ms): 12539.80 P90 E2E Latency (ms): 28150.56 P99 E2E Latency (ms): 34873.51 ---------------Time to First Token---------------- Mean TTFT (ms): 563.50 Median TTFT (ms): 594.92 P99 TTFT (ms): 830.31 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 35.61 Median TPOT (ms): 35.66 P99 TPOT (ms): 35.77 ---------------Inter-Token Latency---------------- Mean ITL (ms): 35.66 Median ITL (ms): 35.69 P95 ITL (ms): 35.96 P99 ITL (ms): 36.13 Max ITL (ms): 36.92 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 526.66 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40798 Request throughput (req/s): 0.15 Input token throughput (tok/s): 75.32 Output token throughput (tok/s): 77.48 Peak output token throughput (tok/s): 96.00 Peak concurrent requests: 18 Total token throughput (tok/s): 152.80 Concurrency: 14.59 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 96023.27 Median E2E Latency (ms): 93940.20 P90 E2E Latency (ms): 159449.54 P99 E2E Latency (ms): 194706.61 ---------------Time to First Token---------------- Mean TTFT (ms): 989.08 Median TTFT (ms): 886.42 P99 TTFT (ms): 1543.60 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 191.04 Median TPOT (ms): 195.20 P99 TPOT (ms): 238.84 ---------------Inter-Token Latency---------------- Mean ITL (ms): 186.68 Median ITL (ms): 183.82 P95 ITL (ms): 189.90 P99 ITL (ms): 673.64 Max ITL (ms): 1633.20 ================================================== ``` # Kimi-K2.6 Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K2.6 ## 1. Model Introduction [Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) is an open-source, native multimodal agentic model by Moonshot AI, delivering industry-leading coding, long-horizon execution, and agent swarm capabilities. It matches or surpasses GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro across key benchmarks. **Key Features:** * **Long-Horizon Coding**: Excels at complex, end-to-end coding tasks with 13+ hours of continuous execution and 4,000+ lines of code modification, generalizing across languages (Rust, Go, Python) and tasks (frontend, devops, performance optimization). * **Coding-Driven Design**: Transforms prompts and visual inputs into production-ready interfaces with motion-rich elements including WebGL shaders, GSAP + Framer Motion, and Three.js 3D. * **Agent Swarms Elevated**: Scales to 300 parallel sub-agents executing 4,000 coordinated steps per run. One prompt, 100+ files. * **Proactive Agents**: Powers OpenClaw, Hermes Agent, and other autonomous frameworks for 5-day continuous operation. * **Native Multimodality**: Pre-trained on vision–language tokens with MoonViT (400M parameters) for visual understanding, cross-modal reasoning, and agentic tool use grounded in visual inputs. **Benchmarks (Open-Source SOTA):**
Benchmark Score
HLE w/ tools 54.0
SWE-Bench Pro 58.6
SWE-bench Multilingual 76.7
BrowseComp 83.2
Toolathlon 50.0
AIME 2026 96.4
GPQA-Diamond 90.5
LiveCodeBench 89.6
**Recommended Generation Parameters:** * Thinking Mode: `temperature=1.0`, `top_p=0.95` * Instant Mode: `temperature=0.6`, `top_p=0.95` **Available Models:** * **INT4 (native checkpoint)**: [moonshotai/Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) * **NVFP4 (4-bit quantized, NVIDIA Blackwell)**: [nvidia/Kimi-K2.6-NVFP4](https://huggingface.co/nvidia/Kimi-K2.6-NVFP4) **License:** Modified MIT for the native checkpoint. The NVIDIA NVFP4 checkpoint is governed by the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). For details, see [official documentation](https://huggingface.co/moonshotai/Kimi-K2.6) and [tech blog](https://kimi.com/blog/kimi-k2-6). ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and capabilities. ### 3.2 Configuration Tips * **Memory**: Requires GPUs with ≥140GB each. The native INT4 checkpoint supports H200 (8×, TP=8), B300 (8×, TP=8), GB300 (4×, TP=4), MI300X/MI325X (4×, TP=4), and MI350X/MI355X (4×, TP=4). Use `--context-length 128000` to conserve memory. * **NVFP4**: Use `nvidia/Kimi-K2.6-NVFP4` with `--quantization modelopt_fp4` on NVIDIA Blackwell. Use `tp=8` on B300 and `tp=4` on GB300. * **AMD GPU TP Constraint**: On AMD GPUs, TP must be ≤ 4 (not 8). Kimi-K2.6 has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid). * **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X. * **DP Attention**: Enable with `--attn-dp-size ` for production throughput. A common choice is to set `--attn-dp-size` equal to `--tp`, but this is not required. * **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs. * **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls. * **AMD FP8 KV Cache**: On AMD platforms the generator adds `--kv-cache-dtype fp8_e4m3` by default and sets `--mem-fraction-static 0.8` to fit the INT4 weights plus KV cache. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload. ## 4. Model Invocation ### 4.1 Basic Usage See [Basic API Usage](../../../docs/basic_usage/send_request). ### 4.2 Advanced Usage #### 4.2.1 Multimodal (Vision + Text) Input Kimi-K2.6 supports native multimodal input with images: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="moonshotai/Kimi-K2.6", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "What is in this image? Describe it in detail." } ] } ] ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} This image shows a **paper receipt from Auntie Anne's**, the pretzel chain restaurant. Here's a detailed breakdown: ## Header - At the top left is the Auntie Anne's logo (a pretzel with a halo) - The store name "**Auntie Anne's**" is printed prominently at the top - Some text below the store name appears blurred/redacted (likely store location, address, or transaction details) ## Purchase Details - **Item**: CINNAMON SUGAR - **Quantity & Price**: 1 × 17,000 - **Item Total**: 17,000 ## Financial Summary - **SUB TOTAL**: 17,000 - **GRAND TOTAL**: 17,000 - **CASH IDR**: 20,000 (customer paid 20,000 Indonesian Rupiah) - **CHANGE DUE**: 3,000 ## Physical Description - The receipt is printed on white thermal paper - Some information in the middle section and toward the bottom is intentionally blurred/obscured - The paper appears slightly curved/wrinkled and is placed on a dark brown surface (likely a table or counter) The transaction is in **Indonesian Rupiah (IDR)**, indicating this purchase was made at an Auntie Anne's location in Indonesia. The customer bought one Cinnamon Sugar pretzel for 17,000 IDR and received 3,000 IDR in change after paying with 20,000 IDR cash. ``` #### 4.2.2 Reasoning Output Kimi-K2.6 supports both thinking mode (default) and instant mode. **Thinking Mode (default)** — reasoning content is automatically separated: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="moonshotai/Kimi-K2.6", messages=[ {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."} ] ) print("====== Reasoning Content (Thinking Mode) ======") print(response.choices[0].message.reasoning_content) print("====== Response (Thinking Mode) ======") print(response.choices[0].message.content) ``` **Instant Mode (thinking off)** — disable thinking for faster responses: ```python Example theme={null} response = client.chat.completions.create( model="moonshotai/Kimi-K2.6", messages=[ {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."} ], extra_body={"chat_template_kwargs": {"thinking": False}} ) print("====== Response (Instant Mode) ======") print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} ====== Reasoning Content (Thinking Mode) ====== The user is asking which number is bigger: 9.11 or 9.9. This seems straightforward, but there's a viral internet debate about this due to decimal confusion. Let me think carefully: - 9.11 means 9 + 11/100 = 9.11 - 9.9 means 9 + 9/10 = 9.90 So 9.9 = 9.90, and 9.90 > 9.11 because 0.90 > 0.11. The confusion often comes from people thinking of software versioning (where 9.11 comes after 9.9) or comparing the numbers after the decimal as whole numbers (11 vs 9, thinking 11 > 9). So mathematically, 9.9 is clearly bigger. 9.9 - 9.11 = 0.79. I should explain this clearly and address the common misconception. ====== Response (Thinking Mode) ====== Mathematically, **9.9 is bigger**. Here's why: **9.9 = 9.90** When comparing decimals, you need to look at the same place values: - 9.11 = 9 ones, 1 tenth, and 1 hundredth - 9.9 = 9 ones, 9 tenths, and 0 hundredths (9.90) Since **0.90 > 0.11**, it follows that **9.9 > 9.11**. The difference is: 9.9 - 9.11 = 0.79 **Why people get confused:** Many mistakenly treat the decimals like whole numbers (thinking "11 is bigger than 9") or confuse this with software version numbering (where version 9.11 comes after version 9.9). But in standard mathematics, 9.9 is definitively larger. ====== Response (Instant Mode) ====== I need to compare 9.11 and 9.9. Let me think carefully by aligning the decimal places: - 9.11 = 9 and 11/100 = 9.11 - 9.9 = 9 and 9/10 = 9.90 Since 0.90 > 0.11 **9.9 is bigger.** This is a common trick question because people sometimes mistakenly compare 11 and 9 as whole numbers after the decimal point, forgetting that 9.9 = 9.90, which is greater than 9.11. ``` #### 4.2.3 Tool Calling Kimi-K2.6 supports tool calling capabilities for agentic tasks: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.6", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) # Process streaming response tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'tool_calls') and delta.tool_calls: for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = {'name': None, 'arguments': ''} if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments if delta.content: print(delta.content, end="", flush=True) for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") ``` **Output Example:** ```text Output theme={null} Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Handling Tool Call Results:** ```python Example theme={null} # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": "The weather in Beijing is 22°C and sunny." } ] final_response = client.chat.completions.create( model="moonshotai/Kimi-K2.6", messages=messages ) print(final_response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} The weather in Beijing is currently **22°C and sunny**. ☀️ It's a nice, warm day there—great for being outdoors! ``` #### 4.2.4 Multimodal + Tool Calling (Agentic Vision) Combine vision understanding with tool calling for advanced agentic tasks: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "search_product", "description": "Search for a product by name or description", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "The product name or description to search for" } }, "required": ["query"] } } } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.6", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Can you identify this product and search for similar items?" } ] } ], tools=tools ) msg = response.choices[0].message # Print reasoning process if msg.reasoning_content: print("=== Reasoning ===") print(msg.reasoning_content) # Print response content if msg.content: print("=== Content ===") print(msg.content) # Print tool calls if msg.tool_calls: print("=== Tool Calls ===") for tc in msg.tool_calls: print(f" Function: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` **Output Example:** ```text Output theme={null} === Reasoning === The user wants me to identify the product from the receipt and search for similar items. Looking at the receipt, it's from Auntie Anne's and the item purchased is "CINNAMON SUGAR" for 17,000 IDR. This is likely a Cinnamon Sugar Pretzel from Auntie Anne's, which is a popular pretzel chain. I should search for this product using the search_product function. The query should be something like "Auntie Anne's Cinnamon Sugar Pretzel" or just "Cinnamon Sugar Pretzel" to find similar items. === Content === Based on the receipt, the product is a **Cinnamon Sugar Pretzel** from **Auntie Anne's** (a popular pretzel bakery chain). The receipt shows it was purchased for 17,000 Indonesian Rupiah (IDR). Let me search for this product and similar items for you. === Tool Calls === Function: search_product Arguments: {"query":"Auntie Anne's Cinnamon Sugar Pretzel"} ``` #### 4.2.5 Speculative Decoding **NVIDIA** Deploy Kimi-K2.6 with the following command (H200/B300, all features enabled): ```shell Command theme={null} sglang serve \ --model-path moonshotai/Kimi-K2.6 \ --tp 8 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm EAGLE3 \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-path lightseekorg/kimi-k2.6-eagle3.1-mla \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` Deploy Kimi-K2.6-NVFP4 with the following command (B300, all features enabled): ```shell Command theme={null} sglang serve \ --model-path nvidia/Kimi-K2.6-NVFP4 \ --tp 8 \ --quantization modelopt_fp4 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm EAGLE3 \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-path lightseekorg/kimi-k2.6-eagle3.1-mla \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` For GB300, use `--tp 4`. ## 5. Benchmark ### 5.1 Accuracy Benchmark **Test Environment:** * Hardware: 8× NVIDIA H200 * Model: moonshotai/Kimi-K2.6 (INT4) * Tensor Parallelism: 8 * SGLang version: 0.5.9 * Reasoning Parser: `kimi_k2` * Tool Call Parser: `kimi_k2` #### NVIDIA NVFP4 Accuracy Reference NVIDIA reports the following accuracy results for `nvidia/Kimi-K2.6-NVFP4` compared with the native INT4 baseline, using `temperature=1.0`, `top_p=0.95`, and max tokens 128,000:
Precision GPQA Diamond SciCode τ²-Bench Telecom MMMU Pro AA-LCR IFBench
Baseline (INT4) 90.9 52.6 98.2 75.6 71.0 73.9
NVFP4 90.4 54.4 98.0 76.5 71.8 73.9
#### 5.1.1 K2-Vendor-Verifier (Tool Calling) * Dataset: [K2-Vendor-Verifier](https://github.com/MoonshotAI/K2-Vendor-Verifier) tool-calls dataset (2,000 requests) * Evaluation Tool: K2-Vendor-Verifier `tool_calls_eval.py` * Settings: temperature=1.0, max\_tokens=64,000, concurrency=256 **Evaluation Command:** ```shell Command theme={null} cd K2-Vendor-Verifier python tool_calls_eval.py tool-calls/samples.jsonl \ --model "moonshotai/Kimi-K2.6" \ --base-url "http://localhost:30000/v1" \ --api-key "placeholder" \ --concurrency 256 \ --temperature 1.0 \ --max-tokens 64000 \ --output kimi-k26-results.jsonl ``` **Results:**
Metric Value
Success Rate 99.95% (1999/2000)
Tool Call Triggered 970
Tool Call Valid 89.6% (869/970)
Tool Call Invalid (schema error) 10.4% (101/970)
#### 5.1.2 AIME 2025 * Dataset: [AIME 2025](https://huggingface.co/datasets/nvidia/aime25) (30 problems) * Evaluation Tool: [sgl-eval](https://github.com/sgl-project/sgl-eval) * Prompt: `eval/matharena/aime` (MathArena format with `\boxed{}` answers) * Settings: temperature=1.0, top\_p=0.95, max\_tokens=131,072, 32 repeats **Evaluation Command:** ```shell Command theme={null} pip install sgl-eval # The dataset ships with sgl-eval; --n-repeats runs all 32 samples per problem. sgl-eval run aime25 \ --base-url http://localhost:30000/v1 \ --model moonshotai/Kimi-K2.6 \ --n-repeats 32 \ --temperature 1.0 \ --top-p 0.95 \ --max-tokens 131072 \ --prompt matharena-aime \ --num-threads 512 ``` **Results:**
Evaluation Mode Accuracy
pass\@1 (avg-of-32) 98.9% (29.7/30)
majority\@32 100.0% (30/30)
pass\@32 100.0%
> 22 out of 32 seeds achieved a perfect score of 30/30. The remaining 10 seeds each missed exactly 1 problem (29/30). #### 5.1.3 GPQA Diamond * Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions, 4-choice multiple choice) * Evaluation Tool: [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai) with `inspect_evals/gpqa_diamond` * Settings: temperature=1.0, top\_p=0.95, max\_tokens=131,072, 4 epochs, cot=True **Evaluation Command:** ```shell Command theme={null} OPENAI_BASE_URL=http://localhost:30000/v1 OPENAI_API_KEY=placeholder \ inspect eval inspect_evals/gpqa_diamond \ --model openai/moonshotai/Kimi-K2.6 \ --max-tokens 131072 \ --temperature 1.0 \ --top-p 0.95 \ --max-connections 128 \ -T cot=True ``` **Results (partial — 553/792 samples across 4 epochs):**
Evaluation Mode Accuracy
pass\@1 (avg across epochs) 96.9%
Epoch Accuracy
1 96.4% (160/166)
2 96.9% (156/161)
3 96.9% (155/160)
4 98.5% (65/66)
#### 5.1.4 OCRBench * Dataset: [OCRBench](https://huggingface.co/datasets/echo840/OCRBench) (1,000 questions with images) * Evaluation Tool: [Kimi-Vendor-Verifier](https://github.com/MoonshotAI/Kimi-Vendor-Verifier) (inspect-ai based) * Settings: max\_tokens=4,096, thinking mode enabled (opensource) **Evaluation Command:** ```shell Command theme={null} cd Kimi-Vendor-Verifier OPENAI_BASE_URL=http://localhost:30000/v1 OPENAI_API_KEY=placeholder \ python3 eval.py ocrbench \ --model openai/moonshotai/Kimi-K2.6 \ --max-tokens 4096 \ --think-mode opensource \ --thinking \ --max-connections 256 ``` **Results:**
Evaluation Mode Accuracy
pass\@1 90.8%
#### 5.1.5 MMMU Pro Vision * Dataset: [MMMU Pro](https://huggingface.co/datasets/MMMU/MMMU_Pro) standard 10-option subset (1,730 questions with images) * Evaluation Tool: [Kimi-Vendor-Verifier](https://github.com/MoonshotAI/Kimi-Vendor-Verifier) (inspect-ai based) * Settings: max\_tokens=32,768, thinking mode (default), max\_connections=256 > **Important**: Kimi-K2.6 is a reasoning model. Setting `max_tokens` too low (e.g., 4096) causes the thinking process to consume the entire token budget, leaving no tokens for the final answer. Use `max_tokens=32768` or higher. **Evaluation Command:** ```shell Command theme={null} cd Kimi-Vendor-Verifier OPENAI_BASE_URL=http://localhost:30000/v1 OPENAI_API_KEY=placeholder \ python3 eval.py mmmu \ --model openai/moonshotai/Kimi-K2.6 \ --max-tokens 32768 \ --think-mode none \ --max-connections 256 ``` **Results (1,481/1,730 samples completed):**
Evaluation Mode Accuracy
pass\@1 82.2%
### 5.2 Speed Benchmark **Test Environment:** * Hardware: NVIDIA H200 GPU (8x) * Model: Kimi-K2.6 * Tensor Parallelism: 8 * SGLang Version: 0.5.9 Kimi-K2.6 shares the same architecture as K2.5. Speed benchmarks are expected to be equivalent. The results below are measured with K2.5 and serve as a reference. We use SGLang's built-in benchmarking tool with the `random` dataset for standardized performance evaluation. #### 5.2.1 Latency Benchmark * **Model Deployment:** ```shell Command theme={null} sglang serve \ --model-path moonshotai/Kimi-K2.6 \ --tp 8 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` **Scenario 1: Chat (1K/1K)** * Low Concurrency ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 39.77 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4221 Request throughput (req/s): 0.25 Input token throughput (tok/s): 153.40 Output token throughput (tok/s): 106.10 Peak output token throughput (tok/s): 156.00 Peak concurrent requests: 2 Total token throughput (tok/s): 259.50 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3972.87 Median E2E Latency (ms): 4044.55 P90 E2E Latency (ms): 7046.30 P99 E2E Latency (ms): 7441.13 ---------------Time to First Token---------------- Mean TTFT (ms): 176.89 Median TTFT (ms): 154.24 P99 TTFT (ms): 285.75 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.22 Median TPOT (ms): 9.32 P99 TPOT (ms): 12.72 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.02 Median ITL (ms): 8.80 P95 ITL (ms): 13.23 P99 ITL (ms): 14.17 Max ITL (ms): 29.38 ================================================== ``` * Medium Concurrency (Balanced) ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 158.05 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40775 Request throughput (req/s): 0.51 Input token throughput (tok/s): 250.99 Output token throughput (tok/s): 258.18 Peak output token throughput (tok/s): 1103.00 Peak concurrent requests: 19 Total token throughput (tok/s): 509.17 Concurrency: 14.09 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 27837.05 Median E2E Latency (ms): 23508.00 P90 E2E Latency (ms): 57126.31 P99 E2E Latency (ms): 66044.35 ---------------Time to First Token---------------- Mean TTFT (ms): 374.30 Median TTFT (ms): 375.51 P99 TTFT (ms): 695.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 53.25 Median TPOT (ms): 57.93 P99 TPOT (ms): 85.45 ---------------Inter-Token Latency---------------- Mean ITL (ms): 53.95 Median ITL (ms): 53.97 P95 ITL (ms): 84.74 P99 ITL (ms): 244.84 Max ITL (ms): 655.61 ================================================== ``` * High Concurrency (Throughput-Optimized) ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 996.64 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 252588 Request throughput (req/s): 0.50 Input token throughput (tok/s): 250.67 Output token throughput (tok/s): 253.51 Peak output token throughput (tok/s): 1199.00 Peak concurrent requests: 104 Total token throughput (tok/s): 504.18 Concurrency: 92.70 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 184773.75 Median E2E Latency (ms): 174183.65 P90 E2E Latency (ms): 343625.28 P99 E2E Latency (ms): 404284.53 ---------------Time to First Token---------------- Mean TTFT (ms): 1289.59 Median TTFT (ms): 1313.35 P99 TTFT (ms): 2346.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 364.70 Median TPOT (ms): 403.32 P99 TPOT (ms): 452.34 ---------------Inter-Token Latency---------------- Mean ITL (ms): 363.82 Median ITL (ms): 316.21 P95 ITL (ms): 745.91 P99 ITL (ms): 1345.88 Max ITL (ms): 3118.59 ================================================== ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 680.26 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 44462 Total generated tokens (retokenized): 44455 Request throughput (req/s): 0.01 Input token throughput (tok/s): 8.97 Output token throughput (tok/s): 65.36 Peak output token throughput (tok/s): 151.00 Peak concurrent requests: 2 Total token throughput (tok/s): 74.33 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 68019.29 Median E2E Latency (ms): 70568.85 P90 E2E Latency (ms): 113237.40 P99 E2E Latency (ms): 121682.34 ---------------Time to First Token---------------- Mean TTFT (ms): 206.17 Median TTFT (ms): 177.28 P99 TTFT (ms): 445.37 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.36 Median TPOT (ms): 15.89 P99 TPOT (ms): 16.43 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.26 Median ITL (ms): 15.85 P95 ITL (ms): 17.50 P99 ITL (ms): 23.21 Max ITL (ms): 45.22 ================================================== ``` * Medium Concurrency ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 2475.98 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 318306 Total generated tokens (retokenized): 318166 Request throughput (req/s): 0.03 Input token throughput (tok/s): 16.02 Output token throughput (tok/s): 128.56 Peak output token throughput (tok/s): 847.00 Peak concurrent requests: 18 Total token throughput (tok/s): 144.58 Concurrency: 14.62 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 452592.46 Median E2E Latency (ms): 486002.05 P90 E2E Latency (ms): 833197.57 P99 E2E Latency (ms): 957399.48 ---------------Time to First Token---------------- Mean TTFT (ms): 359.38 Median TTFT (ms): 350.78 P99 TTFT (ms): 500.36 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 111.18 Median TPOT (ms): 122.76 P99 TPOT (ms): 145.90 ---------------Inter-Token Latency---------------- Mean ITL (ms): 113.69 Median ITL (ms): 122.81 P95 ITL (ms): 147.87 P99 ITL (ms): 151.03 Max ITL (ms): 272.05 ================================================== ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 120.73 Total input tokens: 41941 Total input text tokens: 41941 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.08 Input token throughput (tok/s): 347.41 Output token throughput (tok/s): 34.96 Peak output token throughput (tok/s): 73.00 Peak concurrent requests: 2 Total token throughput (tok/s): 382.36 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 12068.56 Median E2E Latency (ms): 10211.36 P90 E2E Latency (ms): 23203.32 P99 E2E Latency (ms): 30677.66 ---------------Time to First Token---------------- Mean TTFT (ms): 1625.64 Median TTFT (ms): 1526.63 P99 TTFT (ms): 3743.51 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 24.95 Median TPOT (ms): 23.95 P99 TPOT (ms): 35.40 ---------------Inter-Token Latency---------------- Mean ITL (ms): 24.80 Median ITL (ms): 21.73 P95 ITL (ms): 59.56 P99 ITL (ms): 61.10 Max ITL (ms): 62.70 ================================================== ``` * Medium Concurrency ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 389.96 Total input tokens: 300020 Total input text tokens: 300020 Total generated tokens: 41669 Total generated tokens (retokenized): 41670 Request throughput (req/s): 0.21 Input token throughput (tok/s): 769.36 Output token throughput (tok/s): 106.86 Peak output token throughput (tok/s): 304.00 Peak concurrent requests: 19 Total token throughput (tok/s): 876.22 Concurrency: 14.95 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 72870.97 Median E2E Latency (ms): 70495.88 P90 E2E Latency (ms): 121820.46 P99 E2E Latency (ms): 148933.09 ---------------Time to First Token---------------- Mean TTFT (ms): 2460.45 Median TTFT (ms): 1976.29 P99 TTFT (ms): 7305.53 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 140.57 Median TPOT (ms): 142.31 P99 TPOT (ms): 273.40 ---------------Inter-Token Latency---------------- Mean ITL (ms): 135.44 Median ITL (ms): 95.96 P95 ITL (ms): 152.93 P99 ITL (ms): 1488.37 Max ITL (ms): 6540.24 ================================================== ``` * High Concurrency ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 1279.50 Total input tokens: 1273893 Total input text tokens: 1273893 Total generated tokens: 170000 Total generated tokens (retokenized): 169981 Request throughput (req/s): 0.25 Input token throughput (tok/s): 995.62 Output token throughput (tok/s): 132.86 Peak output token throughput (tok/s): 703.00 Peak concurrent requests: 67 Total token throughput (tok/s): 1128.49 Concurrency: 60.12 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 240385.63 Median E2E Latency (ms): 236266.30 P90 E2E Latency (ms): 429882.12 P99 E2E Latency (ms): 515158.36 ---------------Time to First Token---------------- Mean TTFT (ms): 2710.44 Median TTFT (ms): 2345.63 P99 TTFT (ms): 7144.20 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 443.84 Median TPOT (ms): 493.29 P99 TPOT (ms): 606.19 ---------------Inter-Token Latency---------------- Mean ITL (ms): 448.23 Median ITL (ms): 296.17 P95 ITL (ms): 1869.15 P99 ITL (ms): 2708.95 Max ITL (ms): 7778.47 ================================================== ``` ### 5.3 Speed Benchmark (AMD MI350X) **Test Environment:** * Hardware: AMD Instinct MI350X GPU (4x) * Model: Kimi-K2.6 (INT4) * Tensor Parallelism: 4 * SGLang Version: 0.5.9 * Docker Image: `lmsysorg/sglang:v0.5.9-rocm700-mi35x` * ROCm: 7.0 We use SGLang's built-in benchmarking tool with the `random` dataset for standardized performance evaluation. **AMD GPU TP Constraint**: Kimi-K2.6 requires TP ≤ 4 on AMD GPUs. The model has 64 attention heads, and the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid). #### 5.3.1 Latency Benchmark * **Model Deployment:** ```shell Command theme={null} SGLANG_USE_AITER=1 SGLANG_ROCM_FUSED_DECODE_MLA=0 \ sglang serve \ --model-path moonshotai/Kimi-K2.6 \ --tp 4 \ --mem-fraction-static 0.8 \ --trust-remote-code \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --kv-cache-dtype fp8_e4m3 \ --host 0.0.0.0 \ --port 30000 ``` * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 155.81 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4222 Request throughput (req/s): 0.06 Input token throughput (tok/s): 39.16 Output token throughput (tok/s): 27.09 Peak output token throughput (tok/s): 29.00 Peak concurrent requests: 2 Total token throughput (tok/s): 66.24 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 15576.22 Median E2E Latency (ms): 12539.80 P90 E2E Latency (ms): 28150.56 P99 E2E Latency (ms): 34873.51 ---------------Time to First Token---------------- Mean TTFT (ms): 563.50 Median TTFT (ms): 594.92 P99 TTFT (ms): 830.31 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 35.61 Median TPOT (ms): 35.66 P99 TPOT (ms): 35.77 ---------------Inter-Token Latency---------------- Mean ITL (ms): 35.66 Median ITL (ms): 35.69 P95 ITL (ms): 35.96 P99 ITL (ms): 36.13 Max ITL (ms): 36.92 ================================================== ``` * Medium Concurrency (Balanced) ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 526.66 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40798 Request throughput (req/s): 0.15 Input token throughput (tok/s): 75.32 Output token throughput (tok/s): 77.48 Peak output token throughput (tok/s): 96.00 Peak concurrent requests: 18 Total token throughput (tok/s): 152.80 Concurrency: 14.59 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 96023.27 Median E2E Latency (ms): 93940.20 P90 E2E Latency (ms): 159449.54 P99 E2E Latency (ms): 194706.61 ---------------Time to First Token---------------- Mean TTFT (ms): 989.08 Median TTFT (ms): 886.42 P99 TTFT (ms): 1543.60 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 191.04 Median TPOT (ms): 195.20 P99 TPOT (ms): 238.84 ---------------Inter-Token Latency---------------- Mean ITL (ms): 186.68 Median ITL (ms): 183.82 P95 ITL (ms): 189.90 P99 ITL (ms): 673.64 Max ITL (ms): 1633.20 ================================================== ``` # Kimi-K2.7-Code Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K2.7-Code Deploy Kimi-K2.7-Code with SGLang for coding-focused agentic workflows, thinking output, tool calling, and multimodal input. ## 1. Model Introduction [Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code) is a coding-focused agentic model by Moonshot AI, built on top of Kimi-K2.6. It improves real-world long-horizon coding task completion while reducing thinking-token usage by approximately 30% compared with Kimi-K2.6. **Key Features:** * **Coding-Focused Agentic Model**: Optimized for end-to-end coding workflows and complex software engineering tasks. * **Token Efficiency**: Reduces thinking-token usage by approximately 30% versus Kimi-K2.6. * **K2.6-Compatible Deployment**: Shares the same architecture as Kimi-K2.5/Kimi-K2.6, so the SGLang deployment method can be reused with the new model ID. * **Native Multimodality**: Shares Kimi-K2.6's native multimodal architecture with a MoonViT vision encoder (400M parameters) and supports image and video (experimental) input. **Benchmarks:**
Benchmark Kimi-K2.6 Kimi-K2.7-Code
Kimi Code Bench v2 50.9 62.0
Program Bench 48.3 53.6
MLS Bench Lite 26.7 35.1
Kimi Claw 24/7 Bench 42.9 46.9
MCP Atlas 69.4 76.0
MCP Mark Verified 72.8 81.1
**Recommended Generation Parameters:** * Thinking Mode: `temperature=1.0`, `top_p=0.95` * Kimi-K2.7-Code forces thinking and preserve-thinking behavior; instant mode is not supported. **Available Models:** * **INT4 (native checkpoint)**: [moonshotai/Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code) * **MXFP4**: [amd/Kimi-K2.7-Code-MXFP4](https://huggingface.co/amd/Kimi-K2.7-Code-MXFP4) — validated on AMD MI350X/MI355X. **License:** Modified MIT for the native checkpoint. For details, see the [official model card](https://huggingface.co/moonshotai/Kimi-K2.7-Code). ## 2. SGLang Installation Refer to the [official SGLang installation guide](/docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and capabilities. ### 3.2 Configuration Tips * **Memory**: Requires GPUs with ≥140GB each. The native INT4 checkpoint supports H200 (8×, TP=8), B300 (8×, TP=8), GB300 (4×, TP=4), MI300X/MI325X (4×, TP=4), and MI350X/MI355X (4×, TP=4). Use `--context-length 128000` to conserve memory. * **Context Length**: The model supports a 256K context length. Use a shorter `--context-length` when you need to reserve memory for larger batches. * **Transformers Version**: The model card requires `transformers>=4.57.1,<5.0.0`. * **AMD GPU TP Constraint**: On AMD GPUs, TP must be ≤ 4 (not 8). Kimi-K2.7-Code has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid). * **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X. * **DP Attention**: Enable with `--attn-dp-size ` for production throughput. A common choice is to set `--attn-dp-size` equal to `--tp`, but this is not required. * **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs. * **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls. * **AMD FP8 KV Cache**: On AMD platforms the generator adds `--kv-cache-dtype fp8_e4m3` by default and sets `--mem-fraction-static 0.8` to fit the INT4 weights plus KV cache. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload. * **MXFP4 Checkpoint**: The [MXFP4 checkpoint](https://huggingface.co/amd/Kimi-K2.7-Code-MXFP4) is validated on MI350X/MI355X (TP=4) with the `rocm/sgl-dev:v0.5.17-rocm720-mi35x-20260812` image (ROCm 7.2). Select `MXFP4` in the Quantization control to target `amd/Kimi-K2.7-Code-MXFP4`. The MXFP4 quantization is auto-detected from the checkpoint, so no `--quantization` flag is required; the generator emits the AITER/ROCm environment block plus `--attention-backend aiter`, `--mem-fraction-static 0.90`, FP8 KV cache, `--disable-radix-cache`, and `--enable-aiter-allreduce-fusion`. The reasoning and tool-call parsers are auto-detected, so they are not passed explicitly. ## 4. Model Invocation ### 4.1 Basic Usage See [Basic API Usage](/docs/basic_usage/send_request). ### 4.2 Advanced Usage #### 4.2.1 Multimodal (Vision + Text) Input Kimi-K2.7-Code supports native multimodal input with images: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="moonshotai/Kimi-K2.7-Code", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "What is in this image? Describe it in detail." } ] } ] ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} This image shows a **paper receipt from Auntie Anne's**, the pretzel chain restaurant. Here's a detailed breakdown: ## Header - At the top left is the Auntie Anne's logo (a pretzel with a halo) - The store name "**Auntie Anne's**" is printed prominently at the top - Some text below the store name appears blurred/redacted (likely store location, address, or transaction details) ## Purchase Details - **Item**: CINNAMON SUGAR - **Quantity & Price**: 1 × 17,000 - **Item Total**: 17,000 ## Financial Summary - **SUB TOTAL**: 17,000 - **GRAND TOTAL**: 17,000 - **CASH IDR**: 20,000 (customer paid 20,000 Indonesian Rupiah) - **CHANGE DUE**: 3,000 ## Physical Description - The receipt is printed on white thermal paper - Some information in the middle section and toward the bottom is intentionally blurred/obscured - The paper appears slightly curved/wrinkled and is placed on a dark brown surface (likely a table or counter) The transaction is in **Indonesian Rupiah (IDR)**, indicating this purchase was made at an Auntie Anne's location in Indonesia. The customer bought one Cinnamon Sugar pretzel for 17,000 IDR and received 3,000 IDR in change after paying with 20,000 IDR cash. ``` #### 4.2.2 Reasoning Output Kimi-K2.7-Code forces thinking mode and preserve-thinking behavior. **Thinking Mode (default)** — reasoning content is automatically separated: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="moonshotai/Kimi-K2.7-Code", messages=[ {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."} ] ) print("====== Reasoning Content (Thinking Mode) ======") print(response.choices[0].message.reasoning_content) print("====== Response (Thinking Mode) ======") print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} ====== Reasoning Content (Thinking Mode) ====== The user is asking which number is bigger: 9.11 or 9.9. This seems straightforward, but there's a viral internet debate about this due to decimal confusion. Let me think carefully: - 9.11 means 9 + 11/100 = 9.11 - 9.9 means 9 + 9/10 = 9.90 So 9.9 = 9.90, and 9.90 > 9.11 because 0.90 > 0.11. The confusion often comes from people thinking of software versioning (where 9.11 comes after 9.9) or comparing the numbers after the decimal as whole numbers (11 vs 9, thinking 11 > 9). So mathematically, 9.9 is clearly bigger. 9.9 - 9.11 = 0.79. I should explain this clearly and address the common misconception. ====== Response (Thinking Mode) ====== Mathematically, **9.9 is bigger**. Here's why: **9.9 = 9.90** When comparing decimals, you need to look at the same place values: - 9.11 = 9 ones, 1 tenth, and 1 hundredth - 9.9 = 9 ones, 9 tenths, and 0 hundredths (9.90) Since **0.90 > 0.11**, it follows that **9.9 > 9.11**. The difference is: 9.9 - 9.11 = 0.79 **Why people get confused:** Many mistakenly treat the decimals like whole numbers (thinking "11 is bigger than 9") or confuse this with software version numbering (where version 9.11 comes after version 9.9). But in standard mathematics, 9.9 is definitively larger. ``` #### 4.2.3 Preserve Thinking Kimi-K2.7-Code keeps reasoning content across multi-turn interactions. This behavior is enabled by default and cannot be disabled. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) messages = [ { "role": "user", "content": "Tell me three random numbers." }, { "role": "assistant", "reasoning_content": "I'll start by listing five numbers: 473, 921, 235, 215, 222, and I'll tell you the first three.", "content": "473, 921, 235" }, { "role": "user", "content": "What are the other two numbers you have in mind?" } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.7-Code", messages=messages, stream=False, max_tokens=4096, ) print(response.choices[0].message.content) ``` Some OpenAI-compatible deployments use `reasoning` instead of `reasoning_content` in assistant messages. Use the field your serving stack exposes. #### 4.2.4 Tool Calling Kimi-K2.7-Code supports tool calling capabilities for agentic tasks: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.7-Code", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) # Process streaming response tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'tool_calls') and delta.tool_calls: for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = {'name': None, 'arguments': ''} if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments if delta.content: print(delta.content, end="", flush=True) for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") ``` **Output Example:** ```text Output theme={null} Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Handling Tool Call Results:** ```python Example theme={null} # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": "The weather in Beijing is 22°C and sunny." } ] final_response = client.chat.completions.create( model="moonshotai/Kimi-K2.7-Code", messages=messages ) print(final_response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} The weather in Beijing is currently **22°C and sunny**. ☀️ It's a nice, warm day there—great for being outdoors! ``` #### 4.2.5 Multimodal + Tool Calling (Agentic Vision) Combine vision understanding with tool calling for advanced agentic tasks: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "search_product", "description": "Search for a product by name or description", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "The product name or description to search for" } }, "required": ["query"] } } } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.7-Code", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Can you identify this product and search for similar items?" } ] } ], tools=tools ) msg = response.choices[0].message # Print reasoning process if msg.reasoning_content: print("=== Reasoning ===") print(msg.reasoning_content) # Print response content if msg.content: print("=== Content ===") print(msg.content) # Print tool calls if msg.tool_calls: print("=== Tool Calls ===") for tc in msg.tool_calls: print(f" Function: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` **Output Example:** ```text Output theme={null} === Reasoning === The user wants me to identify the product from the receipt and search for similar items. Looking at the receipt, it's from Auntie Anne's and the item purchased is "CINNAMON SUGAR" for 17,000 IDR. This is likely a Cinnamon Sugar Pretzel from Auntie Anne's, which is a popular pretzel chain. I should search for this product using the search_product function. The query should be something like "Auntie Anne's Cinnamon Sugar Pretzel" or just "Cinnamon Sugar Pretzel" to find similar items. === Content === Based on the receipt, the product is a **Cinnamon Sugar Pretzel** from **Auntie Anne's** (a popular pretzel bakery chain). The receipt shows it was purchased for 17,000 Indonesian Rupiah (IDR). Let me search for this product and similar items for you. === Tool Calls === Function: search_product Arguments: {"query":"Auntie Anne's Cinnamon Sugar Pretzel"} ``` #### 4.2.6 Deployment Command Example Deploy Kimi-K2.7-Code with the following command (H200/B300, reasoning and tool parsing enabled): ```shell Command theme={null} sglang serve \ --model-path moonshotai/Kimi-K2.7-Code \ --tp 8 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` For GB300, use `--tp 4`. Deploy the MXFP4 checkpoint on AMD MI350X/MI355X (reasoning and tool-call parsers are auto-detected from the checkpoint): ```shell Command theme={null} SGLANG_USE_AITER=1 \ HIP_FORCE_DEV_KERNARG=1 \ SGLANG_EXPERT_PARALLEL_SIZE=1 \ SGLANG_USE_DYNAMIC_MXFP4_LINEAR=0 \ TORCH_BLAS_PREFER_HIPBLASLT=1 \ TENSILE_STREAMK_DYNAMIC_GRID=6 \ AITER_QUICK_REDUCE_QUANTIZATION=INT4 \ AITER_USE_FLYDSL_MOE_SORTING=1 \ AITER_AR_1STAGE_MAX_KB=512 \ AITER_MXFP4_INTERMEDIATE=1 \ ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \ sglang serve \ --model-path amd/Kimi-K2.7-Code-MXFP4 \ --tp 4 \ --trust-remote-code \ --attention-backend aiter \ --mem-fraction-static 0.90 \ --kv-cache-dtype fp8_e4m3 \ --disable-radix-cache \ --enable-aiter-allreduce-fusion \ --host 0.0.0.0 \ --port 30000 ``` ## 5. Benchmark ### 5.1 Model Card Benchmarks The following results are from the official Kimi-K2.7-Code model card. They were evaluated with thinking mode enabled through Kimi Code CLI at `temperature=1.0`, `top_p=0.95`, and a 262,144-token context length unless otherwise stated.
Category Benchmark Kimi-K2.6 Kimi-K2.7-Code
Coding Kimi Code Bench v2 50.9 62.0
Coding Program Bench 48.3 53.6
Coding MLS Bench Lite 26.7 35.1
Agentic Kimi Claw 24/7 Bench 42.9 46.9
Agentic MCP Atlas 69.4 76.0
Agentic MCP Mark Verified 72.8 81.1
### 5.2 MXFP4 Benchmark **Test Environment:** * Hardware: 4× AMD MI355X * Model: `amd/Kimi-K2.7-Code-MXFP4` (MXFP4) * Tensor Parallelism: 4 * Docker Image: `rocm/sgl-dev:v0.5.17-rocm720-mi35x-20260812` (ROCm 7.2) #### 5.2.1 Accuracy Benchmark Evaluate GSM8K on the deployed MXFP4 checkpoint across client parallelism levels (all 1319 test questions via sgl-eval): ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 1319 \ --num-threads '' ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above.
Parallel GSM8K Accuracy
100 0.951
300 0.953
600 0.942
1200 0.950
#### 5.2.2 Latency Benchmark We use SGLang's built-in `bench_serving` tool with the `random` dataset (input 8192, output 1024, range ratio 0.8). For each concurrency `C`, `--num-prompts` is set to `C × 5`. ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model amd/Kimi-K2.7-Code-MXFP4 \ --dataset-name random \ --random-input-len 8192 \ --random-output-len 1024 \ --random-range-ratio 0.8 \ --max-concurrency \ --num-prompts \ --host 0.0.0.0 \ --port 30000 ```
Concurrency Output Throughput (tok/s) Total Throughput (tok/s) Mean TTFT (ms) Median TTFT (ms) Mean TPOT (ms) Mean E2E Latency (ms)
4 416.22 3747.97 260.35 194.14 9.03 8720.79
8 670.88 6086.74 364.15 199.00 11.19 10687.52
16 963.47 8766.11 490.72 203.12 15.49 14680.68
32 1329.55 11857.47 775.37 215.45 22.61 21831.61
64 1740.46 15759.18 1343.98 300.81 34.41 33068.86
# Kimi-K3 Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3 Deploy Moonshot AI's Kimi-K3 with SGLang — a 2.8T-parameter hybrid Mixture-of-Experts vision-language model (Kimi Delta Attention + MLA, 16/896 active experts) with NVIDIA, AMD, and NPU recipes. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:latest # NVIDIA (CUDA) docker pull lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260916 # AMD MI350X / MI355X (ROCm) ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. ```bash Command theme={null} docker pull quay.io/ascend/sglang:main-cann9.0.0-a3 # Ascend A3 Series docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-950-B070 # Ascend 950PR/DT Series ``` For host and platform setup, see the [NPU installation guide](../../../docs/hardware-platforms/ascend-npus/getting-started/installation) and the [quick start guide](../../../docs/hardware-platforms/ascend-npus/getting-started/quick_start). **Weights (NPU):** [Kimi-K3](https://modelscope.cn/models/moonshotai/Kimi-K3) (950PR/DT Series, MXFP4) · [Kimi-K3-W4A8](https://www.modelscope.cn/models/sgl-npu/Kimi-K3-W4A8) (A3 Series, W4A8, 1.49 TB) · [Kimi-K3-DSpark](https://www.modelscope.cn/models/RadixArk/Kimi-K3-DSpark) (DSPARK draft, 4.5 GB) Pick your hardware, then the deployment shape and operating point. Node count follows the hardware recipe (B200 2×8, GB200 4×4, H100 4×8, B300 1×8, H200 2×8 — 4×8 on Unified High-Throughput, GB300 2×4, MI350X/MI355X 1×8, Ascend A3 Series 4×8 — 32 cards / 64 dies, Ascend 950PR/DT Series 4×8), so it is not a separate choice. If you serve the NVFP4 checkpoint (`nvidia/Kimi-K3-NVFP4`, the **Quantization** row in the panel below), use the `lmsysorg/sglang:dev-dev-kimi-k3-nvfp4` image. **PD Mode** — `Unified` serves prefill and decode together. `Prefill` / `Decode` split them into dedicated pools (see [PD disaggregation](#3-4-pd-disaggregation)); `Prefill` ships two strategies, both chunked at 16k. On the 8-GPU platforms (B300 1×8, GB300 2×4), `Default` is TP8 and `Long-Context` is `--pp-size 8 --tp-size 1`. On the 16-GPU platforms (B200 2×8, GB200 4×4), both are `--pp-size 16 --tp-size 1` and differ only in `--mem-fraction-static` (0.85 vs 0.90) — deep PP is the throughput shape there, not just the long-context one (see [Deep PP](#deep-pp-for-prefill)). **Ascend NPU PD** — both NPU platforms split over the Ascend memory fabric, with `Default` as the only prefill strategy: two symmetric 4-node TP32 pools on the 950PR/DT Series, an asymmetric **2P4D** on the A3 Series (TP16 × PP2 × DP2 prefill, NOSPEC; TP64 × DP4 decode). Pool layouts and the fabric env: [On Ascend NPUs](#npu-pd). **Strategy** — the operating point within that shape: * **Low-Latency** — no DCP, so the MLA KV stays TP-replicated. For chat. B200 splits its two nodes into PP2 × TP8; every other platform is flat TP. * **Balanced** — the accuracy-preserving default: PP2 × DCPEP8 on B200 (the two pipeline stages and DCP8 split KV and KDA state), TP16/DCP16 on GB200, TP8/DCP8 on B300/GB300, TP8/DCP8 ROCm/AITER on MI35x. * **High-Throughput** — the large-scale lane: pick a **Cluster Size** and **Large-Scale Preset** in the Playground ([details](#large-scale-presets)). The cell itself is Balanced, except on H100 (plus `extra_buffer_lazy`) and H200 (widens to 4×8 TP32/EP32 at `--mem-fraction-static 0.90`). `Long-Context` appears only under the `Prefill` PD mode; for long-context unified serving on B200, start from High-Throughput and raise `--context-length`. **Spec Decode** — layers onto the strategy without changing it, on every platform except B200. DSPARK proposes 7 draft tokens per step (tune in the Playground) and requires `pp_size == 1` on the shipped recipes, so on B200 it also drops the pipeline and re-lays the same 16 GPUs flat: PP2 × TP8 → TP16, PP2 × DCPEP8 → DCPEP16. DFLASH uses the [Kimi-K3-DFlash](https://huggingface.co/modal-labs/Kimi-K3-DFlash) draft at block size 8; it doesn't support pipeline parallelism or DP attention yet, so it is unavailable on the pipelined B200 Unified recipes. It has only been validated on Blackwell, so the Hopper and AMD recipes don't offer it yet. The win is largest on short interactive traffic and fades as the prompt grows. `--mamba-full-memory-ratio` is the one sizing flag, computed live: set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator); everything else follows the panels, and the result is pinned into the command. (The Ascend NPU recipes don't use the calculator and set no ratio at all — the Ascend path sizes both pools itself: the A3 Series Unified cell pins `--max-mamba-cache-size` and its PD roles size the pool from `--max-running-requests` instead, while the A5 recipe serves with the radix cache off.) B300 1×8 Unified speed numbers are measured on `v0.5.18 @ 71de97b2` with `--random-range-ratio 1.0`, `--warmup-requests 64`, `--flush-cache`, at ISL 8192 / OSL 1024. DSPARK cells pin the acceptance length via the serve env `SGLANG_SIMULATE_ACC_LEN=4.5` — they report what block size 7 delivers at that acceptance, not a measured acceptance rate for this workload. Balanced DSPARK adds `--max-running-requests 256`; without it speculation resets the cap to 48. The KDA state pool still clamps admission below that (101 / 68 / 91 / 60 concurrent requests for MXFP4 NOSPEC / MXFP4 DSPARK / NVFP4 NOSPEC / NVFP4 DSPARK), which is why no point past concurrency 64 is published for Balanced. ### AMD AITER with DCP8 The MI350X/MI355X unified Balanced recipe uses TP8/DCP8 with AITER prefill and decode attention. DCP shards the target MLA KV cache; RadixArk DSPARK's draft KV remains replicated. The pinned `v0.5.19-rocm720-mi35x-20260916` image records SGLang revision `e7f7447333`, which includes [AITER DCP support (#34432)](https://github.com/sgl-project/sglang/pull/34432) and the [DCP KV-free fix (#38941)](https://github.com/sgl-project/sglang/pull/38941). No source overlay is required for DCP. Keep `SGLANG_ROCM_K3_KDA_FUSED_BACKEND` unset with this image. The separate fused-KDA opt-in requires the [deferred-gate fix (#39066)](https://github.com/sgl-project/sglang/pull/39066), which is not included in this image. This updated recipe remains **Final Verification In Progress**; the recorded speed numbers use their original configurations and do not validate the new image or DCP8 recipe. ### Mamba ratio calculator `--mamba-full-memory-ratio` is the ratio between the KDA state pool and the MLA KV pool. Every parameter below except `L` is read live from the Deploy panel and Playground selection; the balanced value is the per-request cost ratio: ```text theme={null} ratio = (S + D) x state_bytes / (L x (mla_kv_bytes / DCP + draft_kv_bytes)) ``` * `S` — KDA state slots per request: `extra_buffer=5`, `extra_buffer_lazy=4`, `no_buffer=3`, disabled radix cache `=1`. `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK` frees one slot on the extra-buffer strategies; with the overlap scheduler off (or `pp > 1`, which disables it) the track buffer costs one slot instead of two. * `D` — verify intermediate states under speculative decoding: `0` when disabled, otherwise DSPARK block size + 1 (`8` at the default 7). ReplaySSM (`--enable-linear-replayssm-spec`) folds them into a per-slot ring, returning `D` to `0`. * `state_bytes` — one state slot's bytes, from K3's fixed geometry, the attention-TP width, and the SSM dtype. * `mla_kv_bytes` — one token's MLA latent KV bytes (KV-dtype dependent); DCP shards it across its ranks. The DSPARK draft model's KV (\~1.4 KB per token) is replicated on every rank, so it enters flat — negligible without DCP, the same order as the sharded MLA share under DCP8. * `L` — average total request length in tokens: input + output. ## Advanced Features Playground The Playground is where you experiment with **SGLang features beyond the deployment matrix**. The Deploy panel above emits the recipes the SGLang team is converging on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction **Kimi-K3** is Moonshot AI's flagship hybrid MoE vision-language model: **2.8 trillion parameters**, **16 of 896 experts** active per token, roughly **2.5× the scaling efficiency of Kimi-K2**. The backbone interleaves **Kimi Delta Attention (KDA)** with MLA across 93 layers (plus Attention Residuals and Stable LatentMoE); serving supports image input and a **1M-token** window with prefix caching. Weights ship in **MXFP4**: the FlashInfer MXFP4 (trtllm-gen SiTU) runner serves them on Blackwell, Marlin (W4A16) elsewhere, MegaMoE for short-context batch throughput. K3 **always runs with thinking enabled**, with reasoning depth controlled by `reasoning_effort` (`low` / `high` / `max`; default `max`). Kimi-K3 is Moonshot AI's first open-source model in the trillion-plus class; **full model weights are scheduled to release by July 27, 2026**. The recipes on this page were validated on the public [`sgl-project/sglang` `kimi-k3` branch](https://github.com/sgl-project/sglang/tree/kimi-k3) — the HuggingFace repository (`moonshotai/Kimi-K3`) and a public `lmsysorg/sglang` image with K3 support will be available at launch. The B300 1×8 `Unified` Low-Latency and Balanced cells are **Verified** — a speed round on the final weights is published below. Every other cell is still marked **Final Verification In Progress**: the recipe runs, but its serving round on the final weights and current code is still open. Accuracy has not been re-measured on any cell — re-measure before you rely on one. **Recommended generation:** `temperature=1.0`, `top_p=0.95`, `presence_penalty=0`, `frequency_penalty=0` (fixed by the model; informational — do not hardcode in sample code). **Resources:** [HuggingFace](https://huggingface.co/moonshotai/Kimi-K3) · [Kimi-K3 Quickstart](https://platform.kimi.ai/docs/guide/kimi-k3-quickstart). ## 2. Configuration Tips **Memory: two pools, one flag.** K3 splits static memory into a worst-case-reserved **KDA state pool** (it sets the concurrency ceiling) and a paged **MLA KV pool**, divided by `--mamba-full-memory-ratio`. The command panel pins that flag to the [calculator](#mamba-ratio-calculator)'s output — set your average request length there; every other calculator input follows the panels. (On the Ascend NPU recipes there is no calculator and no ratio flag: the Ascend path sizes both pools itself — the A3 Series pins `--max-mamba-cache-size`, and the A5 recipe runs the radix cache off.) After boot, read back `max_total_num_tokens` (the KV side) and the admitted-request cap (the state side). Capacity levers, all in the Playground. Each trades precision or cache behavior for capacity — re-verify accuracy on your workload: | Lever | Effect | | - | - | | `--mamba-radix-cache-strategy extra_buffer_lazy` | 4 state slots per request instead of 5 | | `--mamba-ssm-dtype bfloat16` | \~halves state bytes; with spec on, KDA verification falls back from the fused kernel to Triton | | `--kv-cache-dtype fp8_e4m3` | halves KV bytes per token; under PD both roles must match at connect | | `--mem-fraction-static` 0.90–0.92 | cheapest first win when the boot log shows a large idle `avail mem` | | `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` | frees one more slot per request (experimental, under validation) | Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request — the calculator folds this in — and an unset `--max-running-requests` resets to 48 under spec (the command panel reminds you; set it explicitly to raise). **MoE runner.** Leave `--moe-runner-backend` unset on Blackwell: FlashInfer MXFP4 (W4A8, official trtllm-gen SiTU kernels) is selected; H100/H200 pin Marlin. The B200 Balanced and High-Throughput cells pin `flashinfer_mxfp4` explicitly because that is the shape they were brought up on. **Attention backend.** Leave all three attention knobs unset on Blackwell: K3 resolves prefill, decode, and — under DSPARK — verification as a set (`trtllm_mla` across the board; `cutedsl_mla` takes decode and verification under DCP). On the non-DCP recipes, setting any one of the three cancels the auto-resolution for the others. The B200 Balanced and High-Throughput cells pin `--decode-attention-backend cutedsl_mla`, which is what auto-resolution picks for those DCP recipes anyway — it is written out because it is the shape they were brought up on, not because it changes the resolution. H100/H200 pin `flashmla` for decode. **Context length.** `--context-length` bounds the longest accepted request plus some context-scaled buffers; it does not size the KV pool. For long context the lever that adds capacity is `fp8_e4m3` KV. **DSPARK.** Adds `--speculative-algorithm DSPARK` plus the draft checkpoint on top of the showing strategy. Leave `--speculative-draft-attention-backend` unset. The published B300 DSPARK numbers pin the acceptance length with `SGLANG_SIMULATE_ACC_LEN`, so no measured acceptance rate exists for a real workload yet — measure against the same recipe running NOSPEC before adopting. **Per-platform notes:** | Platform | Topology | Notes | | - | - | - | | B300 1×8 | TP8 (+DCP8) | accuracy-first defaults on Low-Latency and Balanced | | GB300 2×4 | TP8/DCP8 | MNNVL transport and cuMem auto-detected | | B200 2×8 | PP2 × TP8 on Low-Latency, PP2 × DCPEP8 on Balanced and High-Throughput. DSPARK re-lays the same 16 GPUs as TP16 / TP16+DCP16+EP16. PD prefill is TP1 × PP16 | Unified serves all three operating points; `Long-Context` is a `Prefill`-only strategy | | GB200 4×4 | TP16/DCP16 | MNNVL auto-detected | | H200 2×8 (4×8 on Unified High-Throughput) | TP16/EP16 + symm-mem, Marlin + FlashMLA; High-Throughput widens to TP32/EP32 over 4 nodes at mem-frac 0.90 with `extra_buffer_lazy` | same block on every node; export the cross-node NIC (`GLOO_SOCKET_IFNAME` / `NCCL_SOCKET_IFNAME`, `SGLANG_HOST_IP`); keep `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1` | | H100 4×8 | TP32/EP32, Marlin + FlashMLA | SM90a build of the K3 image; pin NCCL/Gloo to the same NIC on all nodes; least post-weight headroom (80 GB) | | MI350X/MI355X 1×8 | TP8/DCP8 ROCm/AITER (Unified Balanced) | AITER A8W4 FlyDSL MoE, AITER prefill/decode attention with sharded target MLA KV, graph bs up to 256, fp8 kvcache; DSPARK supported. Activation-quant and fused-KDA-decode knobs: [AMD ROCm/AITER environment](#amd-env) | | Ascend A3 Series 4×8 (32 cards / 64 dies) | TP64/DP4 + DeepEP | `Unified` plus an asymmetric 2P4D PD split — 2-node prefill (TP16/PP2/DP2, NOSPEC, no CUDA graph) + 4-node decode (TP64/DP4, DSPARK); pin `GLOO`/`HCCL_SOCKET_IFNAME` on every node | | Ascend 950PR/DT Series 4×8 | TP32/dp1 + DeepEP | `Unified` plus PD — two 4-node pools over the Ascend memory fabric (`Default`-only prefill); DSPARK baked in; shared experts / dense MLP shard over attention-TP (`--shared-experts-tp-size 4`); radix cache off; pin `GLOO`/`HCCL_SOCKET_IFNAME` on every node | **Blackwell DCP notes** — the DCP cells are Balanced and High-Throughput on every Blackwell platform, in both the `Unified` and `Decode` roles: * DCP is the only axis that shards the TP-replicated MLA KV; Low-Latency skips it. * Leave `--dcp-comm-backend` unset (fabric-resolved: `fi_a2a` on GB200/GB300, `a2a` on B200/B300). * No `--enable-symm-mem` under DCP (force-disabled for decode-graph correctness). * Explicit `tokenspeed_mla` force-rewrites `--kv-cache-dtype` to fp8; the default `cutedsl_mla` serves either dtype. * Calculator ratios run well above 1 here (`r > 1` is legal): `bfloat16` state buys admission, `fp8` KV buys context. * Don't use EP with an a2a backend: a2a buffers reclaim the KV that DCP buys. Compose only to measure. a2a backend is set when `--moe-a2a-backend` is set. Outside the two verified B300 1×8 `Unified` cells, no cell has a serving round in this exact shape — treat those as starting points to verify. **AMD ROCm/AITER environment (MI350X / MI355X).** The MI35x cell emits `SGLANG_USE_AITER=1 SGLANG_AITER_K3_OPT=1 AITER_FLYDSL_FORCE=1 AITER_SITUV2_A8W4=1` — the first three turn on the AITER ROCm path, its K3-specific fused kernels, and the FlyDSL MoE kernels; the rest of this table is what you can change on top. Everything here is gfx950/ROCm-only and inert elsewhere. These knobs first ship in the `20260903` daily ROCm image pinned above — on an older image they are simply unread — and the two SiTU rows also need an AITER revision at or past [ROCm/aiter#4534](https://github.com/ROCm/aiter/pull/4534) (FlyDSL 0.3.0). | Env var | Default | Effect | | - | - | - | | `AITER_SITUV2_A8W4=1` | unset | SiTU v2 MoE with A8W4 activation quantization, on AITER's GU-interleaved preshuffled weight layout. The performance default the cell ships. | | `AITER_SITUV2_A4W4=1` | unset | A4W4 instead, on the generic separated shuffle layout. Numerically correct but slower than A8W4 (530.8 vs 537.3 tok/s median output on 8×MI35x). Setting **both** gives A8W4 precedence — SGLang follows AITER and keeps the GU-interleaved layout. | | `SGLANG_ROCM_K3_KDA_FUSED_BACKEND=aiter` | unset | Opt in to the fused ROCm KDA decode boundary: the `f_b` projection is deferred into the gfx950 FlyDSL kernel so decode fuses `f_b` + convolution + recurrent state update + gated RMSNorm. Any other value (or unset) keeps the unfused KDA path. | | `SGLANG_K3_FLYDSL_SOURCE` | `auto` | Which FlyDSL implementation backs the fused decode: `auto` prefers SGLang's vendored kernels and falls back to the AITER module, `sglang` / `aiter` pin one. Leave it alone unless you are bisecting the two. | The fused KDA backend is fail-closed at two levels: it arms during model init only when the flag is exactly `aiter` **and** the gfx950 FlyDSL kernels are importable, and each decode step re-validates shapes, dtypes, strides, state indices, and output buffers before dispatch — anything unexpected falls back to the stock KDA implementation rather than erroring. Batch size 2 automatically picks a separately validated kernel schedule; every other batch keeps the original build options. Measured on a 69-layer graph the fused boundary is 9.20 → 8.38 µs/layer (−8.9%), with GSM8K 1319 at 0.950. ## 3. Advanced Usage ### 3.1 Reasoning K3 always thinks; the `kimi_k3` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) separates that thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Control the reasoning depth with `reasoning_effort` (`low` / `high` / `max`; default `max`). ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="moonshotai/Kimi-K3", messages=[{"role": "user", "content": "What is 15% of 240?"}], reasoning_effort="high", # "low" | "high" | "max" (default max) ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Pending update... ``` ### 3.2 Tool Calling Enable the `kimi_k3` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Because K3 is a thinking model, the follow-up turn may put text in `reasoning_content` as well as `content` — print both. (Not yet supported on the Ascend NPU recipes — A3 and A5.) ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }] resp = client.chat.completions.create( model="moonshotai/Kimi-K3", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Pending update... ``` ### 3.3 HiCache (Hierarchical KV Caching) K3's hybrid HiCache tiers the paged MLA KV **and** the KDA/mamba state across L1 (GPU) / L2 (host) / L3 (Mooncake) — enable it from the **HiCache** card in the [Playground above](#playground) for long multi-turn workloads. * On the DCP recipes (Blackwell Balanced / High-Throughput, `Unified` and `Decode` roles), **L1+L2 keeps DCP**, with Spec Decode off or on DSPARK. On B200 the DSPARK pipeline collapse scales it too (PP2 × DCPEP8 → DCPEP16). * **L3 drops the DCP flags** on those recipes (storage keys are not dcp\_rank-aware yet; the command hints call it out, and per-request KV capacity shrinks). Only DCP goes: the MLA KV reverts to TP-replicated while the cell's other parallelism stays, so B300/GB300/GB200 land on plain TP and B200 Unified keeps its `--pp-size 2` / `--ep-size`. * Low-Latency and the Hopper recipes take all tiers unchanged. ### 3.4 PD Disaggregation PD splits prefill and decode into separate server groups; because K3 is hybrid, the transfer moves **both** the paged MLA KV and the KDA recurrent state. * **Transfer**: the CUDA cells emit **NiXL** (RDMA); Mooncake stays selectable in the Playground. The Ascend NPU cells emit `--disaggregation-transfer-backend ascend` — the KV/KDA transfer rides the Ascend memory fabric instead (see [below](#npu-pd)). * **Ports**: prefill `30000`, decode `30100` (derived ZMQ/dist ranges must not collide on a shared host). The positional `8998` after `--prefill` must match `--disaggregation-bootstrap-port`, or only the decode worker registers. * **Decode state pool**: radix cache disabled — one slot per request; `--mamba-radix-cache-strategy` is inert. Keep `--disaggregation-decode-extra-slots` pinned: unpinned it defaults to twice the batch below 32 requests and **zero** above. #### On Ascend NPUs (A3 Series / 950PR/DT Series) Both NPU platforms disaggregate over the **Ascend memory fabric** (`--disaggregation-transfer-backend ascend`), each with its own pool layout: | Pool layout | A3 Series (2P4D) | 950PR/DT Series | | - | - | - | | Prefill pool | 2 nodes, TP16 × PP2 × DP2, NOSPEC, no CUDA graph | 4 nodes, TP32 (dp1), DSPARK | | Decode pool | 4 nodes, TP64 × DP4, DSPARK, decode graph at bs 16 | 4 nodes, TP32 (dp1), DSPARK, decode graph at bs 32 | | Total | 6 nodes | 8 nodes | * Every rank of **both** pools registers its transfer buffers against a store served from the **prefill head node** (`ASCEND_MF_STORE_URL`) — set the prefill head's IP on the decode nodes too, not just the prefill pool. The 950PR/DT recipe also sources `/usr/local/memfabric_hybrid/set_env.sh` on every node where present. * The A3 prefill role is **pipelined** (PP2) and runs NOSPEC — the Spec Decode overlay ships DSPARK on the flat recipes here; newer releases admit it behind a PD prefill ([sgl-project/sglang#40045](https://github.com/sgl-project/sglang/pull/40045)), but no measured A3 command uses it yet. * Each pool is its own `--dist-init-addr` group, so fill `NODE0_IP` per role command (the A3 split's prefill head and decode head are different hosts) — pointing both pools at one address merges the rendezvous, and the mismatched `--nnodes` stalls the launch. * The router can run on any host that reaches both pools. #### Deep PP for prefill Deep PP is `--tp-size 1` with one pipeline stage per GPU — `--pp-size 8` on B300/GB300, `--pp-size 16` on B200/GB200. Pipeline P2P overlaps the next microbatch's compute, unlike TP/EP collectives, and each stage owns whole layers (a clean slice of KV and state). `--tp-size 1` is also what buys context: above TP1 the MLA KV is replicated across the TP ranks, so TP2 × PP8 holds roughly half the tokens of TP1 × PP16 for the same memory. * Use one stage per GPU; a shallow split still pays the in-stage all-reduce and can lose to flat TP. * Pays only with several requests in flight. On the 8-GPU platforms that is why `Default` stays TP8; on the 16-GPU platforms deep PP wins at the Default operating point too, so both strategies use it — measured on GB200 at ISL 8192 / concurrency 32, PP16 × TP1 reached 4550 prefill tok/s/GPU vs 3596 (PP8 × TP2), 2407 (TEP16), and 1652 (TP16). Below concurrency \~8 the pipeline cannot fill and TEP16 leads instead (1947 vs 1227) — use `--tp-size 16 --ep-size 16` there. * DSPARK off on the pipelined recipes — on B200/GB200 that applies to `Default` as well. * Fan one prefill role out to several decode roles; budget for in-transfer KV on the decode side. ```bash Command theme={null} python3 -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:30000 8998 \ --decode http://:30100 \ --host 0.0.0.0 --port 8000 \ --disable-circuit-breaker \ --health-check-interval-secs 999999 ``` Clients then send requests to the router (`:8000`) instead of an individual role server. ### 3.5 VLM Serving Profiles The open-source K3 serving contract currently supports **image input only** — its processor rejects video and audio input. #### VLM feature transport Use **VLM Transport** in the command picker. `Auto` is a topology-aware starting point, not a claim that one configuration is fastest for every workload. | Picker selection | Processor-to-scheduler feature path | | - | - | | Auto · single-node Unified CUDA | CUDA IPC | | Auto · Unified GB200/GB300 | CUDA VMM when IMEX is available; CPU otherwise | | Auto · PD or other topologies | CPU | | CPU | CPU, with no GPU feature pool | CUDA IPC and CUDA VMM reserve up to `SGLANG_MM_FEATURE_CACHE_MB` (1 GiB by default) on the base GPU and fall back to CPU per tensor when full. This setting does not control EPD encoder output or PD KV/KDA transfer. K3 already defaults to 2 processor workers and 16 I/O workers; leave those flags unset unless tuning. #### VLM compatibility | Feature | K3 behavior | | - | - | | PD | Supported. Image processing and ViT run on prefill; the PD transfer then moves both paged MLA KV and KDA recurrent state as described in [PD disaggregation](#pd-disaggregation). | | EPD | Supported on the public `kimi-k3` branch. Use an `--encoder-only` vision role and a `--language-only` prefill role; add the normal decode role for full EPD. See the [EPD guide](../../../docs/advanced_features/epd_disaggregation). | | MM encoder DP | Built in. K3 shards complete images across TP ranks, so leave `--mm-enable-dp-encoder` unset in unified, PD-prefill, and encoder-only roles. | | MM feature transport | Processor-to-scheduler features only. EPD encoder output and PD KV/KDA transfer use their own backends. | | ViT BCG | Compatible with unified and encoder-only roles, but recommended only for repeated encoder shapes after measuring the HBM trade-off below. | #### Should ViT BCG be enabled? Keep ViT BCG **off** for general serving; enable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` only for ViT-only / EPD encoder workloads with recurring image shapes and spare HBM. * The win is confined to the encoder — no reliable end-to-end TTFT/TPOT gain in full-model serving. * Each captured graph retains HBM (graph + per-entry metadata); measure on your own shapes. * The default cache captures after two hits and falls back to eager above 6,144 tokens; do not enlarge it without measuring. #### Low-HBM VLM Use this profile when keeping HBM headroom matters more than peak concurrency. It removes the GPU feature pool, keeps ViT BCG disabled, halves the context window, caps concurrency, and lowers the static-memory target: ```bash Command theme={null} sglang serve \ --trust-remote-code \ --model-path moonshotai/Kimi-K3 \ --tp-size 8 \ --context-length 65536 \ --enable-symm-mem \ --mem-fraction-static 0.82 \ --mm-feature-transport cpu \ --reasoning-parser kimi_k3 \ --tool-call-parser kimi_k3 \ --host 0.0.0.0 \ --port 30000 ``` `--mem-fraction-static 0.82` is a conservative B300 starting point, not a portable minimum: raise it toward `0.85` if startup reports insufficient memory; if HBM must go back to other workloads, reduce context/concurrency first. The precision levers (`fp8_e4m3` KV, `bfloat16` SSM state) save far more but stay accuracy-gated. ### 3.6 Large-Scale Serving Presets (16–64 GPUs, Blackwell) **The KDA state pool is the concurrency ceiling** — DP, EP, and DCP do not shard it; only attention-TP width, SSM dtype, and cache strategy change the per-GPU bill. The MLA KV is cheap to shrink (fp8) or deduplicate (DCP). Two presets come out of this, at `N = 8k` GPUs: | Preset | What it trades | Pick it for | | - | - | - | | **Peak Throughput** — `dp = k`, attention-TP 8 | State shards 8-way. The per-step KDA all-reduce stays within one 8-GPU B200/B300 node, or spans two 4-GPU GB200/GB300 nodes over MNNVL. `--kv-cache-dtype fp8_e4m3` is load-bearing — bf16 KV does not fit 128 requests per replica. | Maximum sustained TPS — the default large-scale shape. | | **Peak Capacity (+DCP8)** — `dp = k` + `--dcp-size 8` | Deduplicates the attention-TP group's MLA KV: concurrency ceiling +72% at the same engine throughput, \~1.8× ITL. | Context ≥ \~16K, or per-replica concurrency past 128. | * **Radix cache** is independent of the preset: for prefix-free traffic (offline batch, evals) switch it off (Playground's **Prefix Cache** card) — one state slot per request instead of 4–5. * The fully data-parallel extreme (`--attn-dp-size` = GPU count, attention-TP 1) — the shape behind the 64-GPU sweep's \~3K tok/s per GPU — is not a preset: 288 GB GPUs only, radix forced off, no head-to-head against the preset shape. The Peak Throughput preset at 32 GPUs on B200/B300 (4 nodes × 8; every node runs the same command with its own `--node-rank`). On GB200/GB300 the same 32-GPU shape uses 8 nodes × 4, and the Playground emits `--nnodes 8`: ```bash Command theme={null} SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=20480 \ sglang serve \ --trust-remote-code \ --model-path moonshotai/Kimi-K3 \ --tp-size 32 --ep-size 32 \ --attn-dp-size 4 --enable-dp-lm-head \ --nnodes 4 --node-rank --dist-init-addr :20000 \ --moe-a2a-backend megamoe --moe-runner-backend deep_gemm \ --kv-cache-dtype fp8_e4m3 \ --mamba-ssm-dtype bfloat16 \ --mamba-radix-cache-strategy extra_buffer_lazy \ --mem-fraction-static 0.92 \ --reasoning-parser kimi_k3 --tool-call-parser kimi_k3 \ --host 0.0.0.0 --port 30000 ``` Scale by holding the per-replica shape fixed and moving only the replica count; pool sizing rides the calculator-driven `--mamba-full-memory-ratio`, which folds in DP, DCP, precision, and speculation: | GPUs | B200/B300 nodes | GB200/GB300 nodes | `--tp-size` / `--ep-size` | `--attn-dp-size` | | - | - | - | - | - | | 16 | 2×8 | 4×4 | 16 | 2 | | 32 | 4×8 | 8×4 | 32 | 4 | | 64 | 8×8 | 16×4 | 64 | 8 | For Peak Capacity, add `--dcp-size 8` and re-derive the pool split with the [Mamba ratio calculator](#mamba-ratio-calculator). Both presets are one click away in the [Playground above](#playground): pick a **Cluster Size** and a **Large-Scale Preset** and the full command composes onto whichever cell is showing. Decisions the preset already makes: * **MegaMoE + `deep_gemm`** — the fused DeepGEMM all-to-all/MoE path used by these large-scale DP/EP throughput presets, with K3's SiTU activation. * **SP-MoE and shared-expert overlap** engage automatically under EP a2a; the K3 all-reduce fusion does not. * **Spec Decode follows the Deploy knob.** Acceptance thins at large batch; spec × EP × DP-attention is validated only at 8-GPU EP8 × DP2 (full GSM8K) — experimental at these scales. No preset has a full serving round on final weights; the constants derive from measured single- and dual-node rounds plus a 64-GPU sweep. Validate throughput and accuracy on your workload before committing a fleet. # Kimi-Linear Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-Linear ## AMD GPU Support ## 1. Model Introduction Kimi Linear is a hybrid linear attention architecture that outperforms traditional full attention methods across various contexts, including short, long, and reinforcement learning (RL) scaling regimes. At its core is Kimi Delta Attention (KDA)—a refined version of Gated DeltaNet that introduces a more efficient gating mechanism to optimize the use of finite-state RNN memory. This generation delivers comprehensive upgrades across the board: Kimi Delta Attention (KDA): A linear attention mechanism that refines the gated delta rule with finegrained gating. Hybrid Architecture: A 3:1 KDA-to-global MLA ratio reduces memory usage while maintaining or surpassing the quality of full attention. Superior Performance: Outperforms full attention in a variety of tasks, including long-context and RL-style benchmarks on 1.4T token training runs with fair comparisons. High Throughput: Achieves up to 6× faster decoding and significantly reduces time per output token (TPOT). For more details, please refer to the \[official Kimi Linear GitHub Repository]: [https://github.com/MoonshotAI/Kimi-Linear](https://github.com/MoonshotAI/Kimi-Linear) ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Launch the docker ```shell Command theme={null} docker pull lmsysorg/sglang:v0.5.7-rocm700-mi30x ``` ```shell Command theme={null} docker run -d -it --ipc=host --network=host --privileged \ --cap-add=CAP_SYS_ADMIN \ --device=/dev/kfd --device=/dev/dri --device=/dev/mem \ --group-add video --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v /:/work \ -e SHELL=/bin/bash \ --name Kimi-linear \ lmsysorg/sglang:v0.5.7-rocm700-mi30x \ /bin/bash ``` #### 4.2.2 pre-installation steps inside the docker ```shell Command theme={null} pip install sentencepiece tiktoken ``` #### 4.2.3 Launch the server ```shell Command theme={null} export SGLANG_ROCM_FUSED_DECODE_MLA=0 SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \ --model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \ --tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \ --tp 4 \ --trust-remote-code ``` ## 5. Benchmark ### 5.1 Speed Benchmark Test Environment: Hardware: AMD MI300X GPU Model: Kimi-Linear-48B-A3B-Instruct Tensor Parallelism: 4 sglang version: 0.5.7 * **Model Deployment** ```bash Command theme={null} SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \ --model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \ --tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \ --tp 4 \ --trust-remote-code ``` ### 5.1.1 Low Concurrency (Latency-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-Linear-48B-A3B-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 23.86 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4001 Request throughput (req/s): 0.42 Input token throughput (tok/s): 255.70 Output token throughput (tok/s): 176.86 Peak output token throughput (tok/s): 190.00 Peak concurrent requests: 2 Total token throughput (tok/s): 432.56 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2383.93 Median E2E Latency (ms): 1911.63 ---------------Time to First Token---------------- Mean TTFT (ms): 141.33 Median TTFT (ms): 126.27 P99 TTFT (ms): 294.76 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.32 Median TPOT (ms): 5.33 P99 TPOT (ms): 5.36 ---------------Inter-Token Latency---------------- Mean ITL (ms): 5.33 Median ITL (ms): 5.32 P95 ITL (ms): 5.44 P99 ITL (ms): 5.58 Max ITL (ms): 11.46 ================================================== ``` ### 5.1.2 Medium Concurrency (Balanced) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-Linear-48B-A3B-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 31.38 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 39667 Request throughput (req/s): 2.55 Input token throughput (tok/s): 1264.13 Output token throughput (tok/s): 1300.37 Peak output token throughput (tok/s): 1801.00 Peak concurrent requests: 21 Total token throughput (tok/s): 2564.50 Concurrency: 14.13 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5543.18 Median E2E Latency (ms): 5755.31 ---------------Time to First Token---------------- Mean TTFT (ms): 175.25 Median TTFT (ms): 137.87 P99 TTFT (ms): 292.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.75 Median TPOT (ms): 10.87 P99 TPOT (ms): 16.74 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.54 Median ITL (ms): 7.95 P95 ITL (ms): 13.68 P99 ITL (ms): 116.80 Max ITL (ms): 299.89 ================================================== ``` ### 5.1.3 High Concurrency (Throughput-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-Linear-48B-A3B-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 79.71 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 228448 Request throughput (req/s): 6.27 Input token throughput (tok/s): 3134.20 Output token throughput (tok/s): 3169.72 Peak output token throughput (tok/s): 6109.00 Peak concurrent requests: 110 Total token throughput (tok/s): 6303.92 Concurrency: 94.80 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 15113.92 Median E2E Latency (ms): 13851.52 ---------------Time to First Token---------------- Mean TTFT (ms): 564.46 Median TTFT (ms): 226.04 P99 TTFT (ms): 2683.14 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 29.63 Median TPOT (ms): 31.28 P99 TPOT (ms): 38.84 ---------------Inter-Token Latency---------------- Mean ITL (ms): 28.85 Median ITL (ms): 16.29 P95 ITL (ms): 123.42 P99 ITL (ms): 157.80 Max ITL (ms): 2481.11 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * Server Command ```shell Command theme={null} SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \ --model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \ --tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \ --tp 4 \ --trust-remote-code ``` * Benchmark Command ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * **Result**: ```text Output theme={null} Accuracy: 0.705 Invalid: 0.000 Latency: 11.855 s Output throughput: 3224.982 token/s ``` # Nemotron3-Nano Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3-Nano ## 1. Model Introduction `NVIDIA Nemotron3-Nano` is a 30B-parameter hybrid LLM that mixes Mixture-of-Experts (MoE) feed-forward layers, Mamba2 sequence-modeling layers, and standard self-attention layers in a single stack rather than classic “attention + MLP” transformer blocks. The BF16 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16`) is designed as a high-fidelity reference model. For optimized inference performance on modern NVIDIA GPUs, the FP8 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8`) and the NVFP4 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4`) are supported. At a high level: * **Hybrid layer stack (Mamba2 + MoE + attention):** The network is composed of interleaved layers that are *either* Mamba2, *or* MoE feed-forward, *or* attention-only. * **Non-uniform layer ordering:** The order and mix of these specialized layers is not a simple, rigid pattern, enabling the model to trade off sequence modeling, routing capacity, and expressivity across depth. * **Deployment-friendly precision:** Use BF16 for accuracy-sensitive and evaluation workloads; use FP8 for latency- and throughput-critical serving on recent NVIDIA GPUs. ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install), or install nightly wheel through: ```bash Command theme={null} uv pip install --prerelease=allow sglang==0.5.6.post3.dev1278+gad1b4e472 --extra-index-url https://sgl-project.github.io/whl/nightly/ ``` ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance tuning. ### 3.1 Basic Configuration The Nemotron3-Nano series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)). **Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command. ### 3.2 Configuration Tips * **Attention backend**: **H200**: Use flash attention 3 backend by default. **B200**: Use flashinfer backend by default. * **TP support**: To set tp size, use `--tp <1|2|4|8>`. * **FP8 KV cache**: To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`. ## 4. Model Invocation ### 4.1 Basic Usage (OpenAI-Compatible API) SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Summarize what MoE models are in 5 bullets."}, ], temperature=0.7, max_tokens=256, ) print(resp.choices[0].message.content) ``` Streaming chat completion ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) stream = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8", messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "What are the first 5 prime numbers?"} ], temperature=0.7, max_tokens=1024, stream=True, ) for chunk in stream: delta = chunk.choices[0].delta if delta and delta.content: print(delta.content, end="", flush=True) ``` ### 4.2 Reasoning To enable reasoning, `--reasoning-parser nemotron_3` should be appended to the launching command. The model supports two modes - Reasoning ON (default) vs OFF. This can be toggled by setting enable\_thinking to False, as shown below. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) # Reasoning on (default) print("Reasoning on") resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Write a haiku about GPUs."} ], temperature=0.7, max_tokens=512, ) print(resp.choices[0].message.reasoning_content) # Reasoning off print("Reasoning off") resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Write a haiku about GPUs."} ], temperature=0.6, max_tokens=256, extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) print(resp.choices[0].message.reasoning_content) ``` ### 4.3 Tool calling To enable reasoning, `--tool-call-parser qwen3_coder` should be appended to the launching command. Call functions using the OpenAI Tools schema and inspect returned tool\_calls. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) # Tool calling via OpenAI tools schema TOOLS = [ { "type": "function", "function": { "name": "calculate_tip", "parameters": { "type": "object", "properties": { "bill_total": { "type": "integer", "description": "The total amount of the bill" }, "tip_percentage": { "type": "integer", "description": "The percentage of tip to be applied" } }, "required": ["bill_total", "tip_percentage"] } } } ] completion = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8", messages=[ {"role": "system", "content": ""}, {"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"} ], tools=TOOLS, temperature=0.6, top_p=0.95, max_tokens=512, stream=False ) print(completion.choices[0].message.reasoning_content) print(completion.choices[0].message.tool_calls) ``` *** ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU **FP8 variant** * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \ --trust-remote-code \ --max-running-requests 1024 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 4096 \ --max-concurrency 256 ``` * **Serving benchmark results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 256 Successful requests: 4096 Benchmark duration (s): 183.18 Total input tokens: 2081726 Total input text tokens: 2081726 Total input vision tokens: 0 Total generated tokens: 2116125 Total generated tokens (retokenized): 1076256 Request throughput (req/s): 22.36 Input token throughput (tok/s): 11364.25 Output token throughput (tok/s): 11552.04 Peak output token throughput (tok/s): 24692.00 Peak concurrent requests: 294 Total token throughput (tok/s): 22916.30 Concurrency: 251.19 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 11233.74 Median E2E Latency (ms): 11142.97 ---------------Time to First Token---------------- Mean TTFT (ms): 172.99 Median TTFT (ms): 116.57 P99 TTFT (ms): 1193.68 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 21.74 Median TPOT (ms): 21.14 P99 TPOT (ms): 41.12 ---------------Inter-Token Latency---------------- Mean ITL (ms): 21.45 Median ITL (ms): 9.06 P95 ITL (ms): 62.59 P99 ITL (ms): 110.83 Max ITL (ms): 5368.19 ================================================== ``` **BF16 variant** * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ --trust-remote-code \ --max-running-requests 1024 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 4096 \ --max-concurrency 256 ``` * **Serving benchmark results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 256 Successful requests: 4096 Benchmark duration (s): 360.22 Total input tokens: 2081726 Total input text tokens: 2081726 Total input vision tokens: 0 Total generated tokens: 2087288 Total generated tokens (retokenized): 1940652 Request throughput (req/s): 11.37 Input token throughput (tok/s): 5779.10 Output token throughput (tok/s): 5794.55 Peak output token throughput (tok/s): 9169.00 Peak concurrent requests: 276 Total token throughput (tok/s): 11573.65 Concurrency: 249.76 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 21965.10 Median E2E Latency (ms): 21706.35 ---------------Time to First Token---------------- Mean TTFT (ms): 211.54 Median TTFT (ms): 93.06 P99 TTFT (ms): 2637.66 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 43.27 Median TPOT (ms): 43.04 P99 TPOT (ms): 61.15 ---------------Inter-Token Latency---------------- Mean ITL (ms): 42.77 Median ITL (ms): 28.46 P95 ITL (ms): 71.85 P99 ITL (ms): 113.20 Max ITL (ms): 5237.28 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark **Environment** * Hardware: NVIDIA B200 GPU * Model: BF16 checkpoint **Launch Model** ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ --trust-remote-code \ --reasoning-parser nemotron_3 ``` **Run Benchmark with sgl-eval** ```bash Command theme={null} pip install sgl-eval sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 --num-threads 4 --max-tokens 16384 --thinking ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. **Historical lm-eval results (not measurements of the sgl-eval command above):** ```text Output theme={null} |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.5603|± |0.0137| | | |strict-match | 5|exact_match|↑ |0.8453|± |0.0100| ``` # Nemotron 3 Nano Omni Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3-Nano-Omni ## 1. Model Introduction `NVIDIA Nemotron 3 Nano Omni` is a 30B-parameter hybrid MoE multimodal model that activates only 3B parameters per forward pass, combining vision and audio encoders into a unified architecture. Part of the Nemotron 3 family, it is designed to power multimodal sub-agents that perceive and reason across vision, audio, and language in a single inference loop — eliminating the fragmented stacks of separate models for each modality. Architecture and key features: * **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture for efficient routing and sequence modeling. * **30B total / 3B active parameters:** Delivers strong multimodal accuracy at a fraction of the cost of dense models. * **1M token context window:** Sustains coherent agent state across extended multimodal workflows — screen history, document content, and audio context remain in view without re-ingestion. * **Unified vision and audio encoders:** One model replaces fragmented multimodal stacks; vision and audio perception happen in the same forward pass. * **3D Convolution (Conv3D):** Efficient temporal-spatial processing for video inputs. * **Efficient Video Sampling (EVS):** Enables longer video processing at the same compute budget via temporal-aware perception and adaptive frame sampling. * **FP8 and NVFP4 quantization:** FP8 supports deployment from workstation (RTX 6000, DGX Spark) to cloud (H100, H200, B200, A100, L40S); NVFP4 requires Blackwell hardware. * **9x higher throughput** than other open omni models at the same interactivity level. * **\~20% higher multimodal intelligence** compared to the best open alternative. * **Post-trained with multi-environment reinforcement learning** via NVIDIA NeMo RL and NeMo Gym across text, image, audio, and video environments, improving instruction following and convergence to correct multimodal answers. **Modalities:** Input: text, image, video, audio — Output: text **Supported GPUs:** NVIDIA B200, H100, H200, A100, L40S, DGX Spark, RTX 6000 Available model variants on HuggingFace: * [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) * [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8) * [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4) **Agentic workloads this model enables:** * **Computer Use Agent:** Perception loop for agents navigating GUIs — reads screens, understands UI state over time, validates outcomes. Collapses vision and reasoning into a single loop. * **Document Intelligence:** Interprets documents, charts, tables, screenshots, and mixed media inputs for enterprise analysis and compliance workflows. * **Audio & Video Understanding Agents:** Maintains continuous audio-video context for customer service, research, and monitoring workflows, tying what was said, shown, and documented into a single reasoning stream. ## 2. SGLang Installation Install SGLang via pip or from source: ```shell Command theme={null} # Install via pip pip install sglang # Or install from source uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' # Or use Docker docker pull lmsysorg/sglang:latest ``` For the full Docker setup and other installation methods, refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance tuning. ### 3.1 Basic Configuration **Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command. ### 3.2 Configuration Tips * **Attention backend:** **H100/H200:** Use flash attention 3 backend by default. **B200:** Use flashinfer backend by default. * **TP support:** To set tensor parallelism, use `--tp <1|2|4|8>`. A 4×H100 setup is recommended for the BF16 variant. * **FP8 KV cache:** To enable FP8 KV cache, append `--kv-cache-dtype fp8_e4m3`. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload. * **Reasoning parser:** Append `--reasoning-parser deepseek-r1` to enable structured reasoning traces (`reasoning_content` field in the response). * **Tool calling:** Append `--tool-call-parser qwen3_coder` to enable tool calling support. ## 4. Model Invocation The command below launches the server for a 4×H100 setup with reasoning and tool calling enabled. See [Section 4.8](#4-8-fp8-and-nvfp4-deployment) for FP8 and NVFP4 variants. ```shell Command theme={null} sglang serve \ --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \ --host 0.0.0.0 \ --port 30000 \ --tp 4 \ --trust-remote-code \ --tool-call-parser qwen3_coder \ --reasoning-parser deepseek-r1 ``` ### 4.1 Basic Usage (Text) SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client: ```python Example theme={null} from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "Give me 3 bullet points about SGLang."}, ], temperature=0.6, max_tokens=512, ) print(resp.choices[0].message.reasoning_content, resp.choices[0].message.content) ``` Output: ```text Output theme={null} Reasoning: SGLang is a serving framework I know from my training data. Let me recall the key features... Content: - **Radix Attention** — SGLang reuses KV cache across requests sharing a common prefix, dramatically reducing memory and compute for multi-turn and few-shot workloads. - **OpenAI-compatible API** — Drop-in replacement for the OpenAI Python client; no application code changes required to serve a locally-hosted model. - **High-throughput serving** — Continuous batching, chunked prefill, and optimized CUDA kernels deliver state-of-the-art throughput on NVIDIA GPUs across A100, H100, and B200. ``` Streaming chat completion: ```python Example theme={null} from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") stream = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "What are the first 5 prime numbers?"}, ], temperature=0.6, max_tokens=512, stream=True, ) for chunk in stream: delta = chunk.choices[0].delta if delta and delta.content: print(delta.content, end="", flush=True) ``` ### 4.2 Image Understanding Pass image inputs using the OpenAI vision format. Supports both URLs and base64-encoded images: ```python Example theme={null} from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") # From URL resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}, }, {"type": "text", "text": "Describe this image in detail."}, ], } ], temperature=0.6, max_tokens=512, ) print(resp.choices[0].message.reasoning_content) print(resp.choices[0].message.content) ``` For local images, encode as base64: ```python Example theme={null} import base64 from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("screenshot.png", "rb") as f: image_b64 = base64.b64encode(f.read()).decode("utf-8") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}, }, {"type": "text", "text": "What UI elements are visible on this screen? What action would you take next?"}, ], } ], temperature=0.6, max_tokens=512, ) print(resp.choices[0].message.content) ``` ### 4.3 Video Understanding Nemotron 3 Nano Omni uses Conv3D layers and Efficient Video Sampling (EVS) for temporal-spatial video reasoning, processing longer videos at the same compute budget: ```python Example theme={null} import base64 from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("video.mp4", "rb") as f: video_b64 = base64.b64encode(f.read()).decode("utf-8") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ { "role": "user", "content": [ { "type": "video_url", "video_url": {"url": f"data:video/mp4;base64,{video_b64}"}, }, {"type": "text", "text": "Summarize what happens in this video step by step."}, ], } ], temperature=0.6, max_tokens=1024, ) print(resp.choices[0].message.reasoning_content) print(resp.choices[0].message.content) ``` ### 4.4 Audio Understanding Pass audio inputs as base64-encoded WAV or MP3 data: ```python Example theme={null} import base64 from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("audio.wav", "rb") as f: audio_b64 = base64.b64encode(f.read()).decode("utf-8") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ { "role": "user", "content": [ { "type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}, }, {"type": "text", "text": "Transcribe and summarize what was said in this audio."}, ], } ], temperature=0.6, max_tokens=512, ) print(resp.choices[0].message.content) ``` ### 4.5 Mixed Multimodal Input Combine modalities in a single request. For example, an image alongside an audio question about it: ```python Example theme={null} import base64 from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("chart.png", "rb") as f: image_b64 = base64.b64encode(f.read()).decode("utf-8") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}, }, {"type": "text", "text": "Analyze this chart. What are the key trends and what conclusion does the data support?"}, ], } ], temperature=0.6, max_tokens=1024, ) print(resp.choices[0].message.reasoning_content) print(resp.choices[0].message.content) ``` ### 4.6 Reasoning The model supports two modes — Reasoning ON (default) vs OFF. Toggle per-request by setting `enable_thinking` to `False`: ```python Example theme={null} from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") # Reasoning ON (default) print("Reasoning on") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the derivative of x^3 sin(x)?"}, ], temperature=0.6, max_tokens=1024, ) print(f"Reasoning:\n{resp.choices[0].message.reasoning_content[:300]}...\nContent:\n{resp.choices[0].message.content}") print("\n") # Reasoning OFF print("Reasoning off") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is 15% of 200?"}, ], temperature=0.6, max_tokens=256, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print(f"Content:\n{resp.choices[0].message.content}") ``` Output: ```text Output theme={null} Reasoning on Reasoning: The user wants the derivative of x^3 sin(x). I'll apply the product rule: d/dx[u·v] = u'v + uv'. Here u = x^3, v = sin(x). So u' = 3x^2, v' = cos(x). The result is 3x^2·sin(x) + x^3·cos(x)... Content: Using the product rule: d/dx[x³ sin(x)] = 3x² sin(x) + x³ cos(x) Reasoning off Content: 15% of 200 is **30**. ``` ### 4.7 Tool Calling Call functions using the OpenAI Tools schema. The server must be launched with `--tool-call-parser qwen3_coder`: ```python Example theme={null} from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") TOOLS = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "City and state, e.g. San Francisco, CA", }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], }, }, "required": ["location"], }, }, } ] completion = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the weather like in Santa Clara, CA?"}, ], tools=TOOLS, temperature=0.6, top_p=0.95, max_tokens=512, stream=False, ) print(completion.choices[0].message.reasoning_content) print(completion.choices[0].message.tool_calls) ``` Output: ```text Output theme={null} The user is asking about weather in Santa Clara, CA. I have a get_weather function that takes a location and optional unit. I should call it with location="Santa Clara, CA". [ChatCompletionMessageFunctionToolCall(id='call_abc123', function=Function(arguments='{"location": "Santa Clara, CA", "unit": "fahrenheit"}', name='get_weather'), type='function', index=0)] ``` ### 4.8 FP8 and NVFP4 Deployment **FP8 variant** (recommended for throughput-critical serving on H100/H200/B200): ```shell Command theme={null} sglang serve \ --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \ --host 0.0.0.0 \ --port 30000 \ --tp 4 \ --trust-remote-code \ --tool-call-parser qwen3_coder \ --reasoning-parser deepseek-r1 ``` **NVFP4 variant** (maximum efficiency on Blackwell B200): ```shell Command theme={null} sglang serve \ --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \ --host 0.0.0.0 \ --port 30000 \ --tp 4 \ --trust-remote-code \ --tool-call-parser qwen3_coder \ --reasoning-parser deepseek-r1 ``` *** ## 5. Benchmark ### 5.1 Efficiency Benchmark Nemotron 3 Nano Omni achieves **9x higher throughput** than other open omni models at the same interactivity level, delivering lower cost and better scalability without sacrificing responsiveness. It also achieves **\~20% higher multimodal intelligence** compared to the best open alternative across image, video, and audio reasoning tasks. ### 5.2 Speed Benchmark **Test Environment:** * Hardware: B200 (8×) * Model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning * Tensor Parallelism: 4 * SGLang Version: main branch Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \ --trust-remote-code \ --tp 4 \ --max-running-requests 1024 \ --host 0.0.0.0 \ --attention-backend flashinfer \ --port 30000 ``` Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 4096 \ --max-concurrency 256 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 256 Successful requests: 4096 Benchmark duration (s): 206.52 Total input tokens: 2081726 Total input text tokens: 2081726 Total generated tokens: 2087288 Total generated tokens (retokenized): 1945477 Request throughput (req/s): 19.83 Input token throughput (tok/s): 10080.25 Output token throughput (tok/s): 10107.18 Peak output token throughput (tok/s): 20199.00 Peak concurrent requests: 291 Total token throughput (tok/s): 20187.44 Concurrency: 250.83 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 12646.47 Median E2E Latency (ms): 12371.84 P90 E2E Latency (ms): 22889.81 P99 E2E Latency (ms): 26528.70 ---------------Time to First Token---------------- Mean TTFT (ms): 220.66 Median TTFT (ms): 97.67 P99 TTFT (ms): 2068.63 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 24.98 Median TPOT (ms): 24.36 P99 TPOT (ms): 44.97 ---------------Inter-Token Latency---------------- Mean ITL (ms): 24.43 Median ITL (ms): 10.91 P95 ITL (ms): 62.68 P99 ITL (ms): 100.60 Max ITL (ms): 2171.93 ================================================== ``` ### 5.3 Accuracy Benchmark **Environment** * Hardware: B200 (8×) * Model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning * Tensor Parallelism: 4 * SGLang Version: main branch **Launch Model** ```shell Command theme={null} sglang serve \ --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \ --trust-remote-code \ --tp 4 \ --attention-backend flashinfer \ --reasoning-parser deepseek-r1 ``` #### 5.3.1 GSM8K Benchmark **Run Benchmark** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. **Test Results:** ```text Output theme={null} Accuracy: 0.830 Invalid: 0.000 Latency: 13.970 s Output throughput: 1611.623 token/s ``` #### 5.3.2 MMLU Benchmark **Run Benchmark** ```shell Command theme={null} sgl-eval run mmlu \ --base-url http://127.0.0.1:30000/v1 \ --num-threads 64 --max-tokens 2048 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. **Test Results:** ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.510 subject: anatomy, #q:135, acc: 0.711 subject: astronomy, #q:152, acc: 0.829 subject: business_ethics, #q:100, acc: 0.760 subject: clinical_knowledge, #q:265, acc: 0.781 subject: college_biology, #q:144, acc: 0.854 subject: college_chemistry, #q:100, acc: 0.560 subject: college_computer_science, #q:100, acc: 0.700 subject: college_mathematics, #q:100, acc: 0.590 subject: college_medicine, #q:173, acc: 0.775 subject: college_physics, #q:102, acc: 0.559 subject: computer_security, #q:100, acc: 0.750 subject: conceptual_physics, #q:235, acc: 0.821 subject: econometrics, #q:114, acc: 0.605 subject: electrical_engineering, #q:145, acc: 0.759 subject: elementary_mathematics, #q:378, acc: 0.638 subject: formal_logic, #q:126, acc: 0.524 subject: global_facts, #q:100, acc: 0.400 subject: high_school_biology, #q:310, acc: 0.906 subject: high_school_chemistry, #q:203, acc: 0.759 subject: high_school_computer_science, #q:100, acc: 0.860 subject: high_school_european_history, #q:165, acc: 0.812 subject: high_school_geography, #q:198, acc: 0.889 subject: high_school_government_and_politics, #q:193, acc: 0.933 subject: high_school_macroeconomics, #q:390, acc: 0.785 subject: high_school_mathematics, #q:270, acc: 0.496 subject: high_school_microeconomics, #q:238, acc: 0.887 subject: high_school_physics, #q:151, acc: 0.675 subject: high_school_psychology, #q:545, acc: 0.895 subject: high_school_statistics, #q:216, acc: 0.731 subject: high_school_us_history, #q:204, acc: 0.858 subject: high_school_world_history, #q:237, acc: 0.873 subject: human_aging, #q:223, acc: 0.740 subject: human_sexuality, #q:131, acc: 0.855 subject: international_law, #q:121, acc: 0.851 subject: jurisprudence, #q:108, acc: 0.815 subject: logical_fallacies, #q:163, acc: 0.847 subject: machine_learning, #q:112, acc: 0.598 subject: management, #q:103, acc: 0.864 subject: marketing, #q:234, acc: 0.910 subject: medical_genetics, #q:100, acc: 0.880 subject: miscellaneous, #q:783, acc: 0.881 subject: moral_disputes, #q:346, acc: 0.780 subject: moral_scenarios, #q:895, acc: 0.543 subject: nutrition, #q:306, acc: 0.814 subject: philosophy, #q:311, acc: 0.733 subject: prehistory, #q:324, acc: 0.852 subject: professional_accounting, #q:282, acc: 0.553 subject: professional_law, #q:1534, acc: 0.565 subject: professional_medicine, #q:272, acc: 0.779 subject: professional_psychology, #q:612, acc: 0.760 subject: public_relations, #q:110, acc: 0.709 subject: security_studies, #q:245, acc: 0.759 subject: sociology, #q:201, acc: 0.831 subject: us_foreign_policy, #q:100, acc: 0.910 subject: virology, #q:166, acc: 0.560 subject: world_religions, #q:171, acc: 0.807 Total latency: 67.512 Average accuracy: 0.737 ``` # NVIDIA Nemotron3-Super Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3-Super ## 1. Model Introduction `NVIDIA Nemotron3-Super` is a leading open model in the Nemotron 3 family, built for running many collaborating agents together. It is optimized for agentic systems that chain planning, reasoning, and tool use workloads that generate far more tokens than single turn chat and require strong reasoning at every step. Nemotron 3 Super is a 120B parameter hybrid MoE model that activates only 12B parameters per forward pass, delivering strong accuracy for coding, tool calling, and instruction following at a fraction of the cost. It also supports a 1M token context window so agents can keep conversation history and plan state in view across long workflows. Architecture and key features: * **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture, enabling efficient routing and sequence modeling in a single stack. * **Highest throughput efficiency in its size category:** Delivers up to 5x higher throughput compared to the previous Nemotron Super model (Llama Nemotron Super 1.5). * **Multi-Token Prediction (MTP):** By predicting several future tokens simultaneously in a single forward pass, MTP drastically accelerates the generation of long-form text. * **Thinking Budget support:** Supports Thinking Budget for optimal accuracy with minimum reasoning token generation. ## 2. SGLang Installation SGLang from the main branch is required for Nemotron3-Super. You can install from source and with a nightly docker. ```bash Command theme={null} # Install from source uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' # Or use Docker docker pull lmsysorg/sglang:latest ``` For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance tuning. ### 3.1 Basic Configuration **Interactive Command Generator**: select hardware, tensor parallelism, and common knobs to generate a launch command. ### 3.2 Configuration Tips * **Attention backend**: **H200**: Use flash attention 3 backend by default. **B200**: Use flashinfer backend by default. * **TP support**: To set tp size, use `--tp <2|4|8>`. * **FP8 KV cache**: To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`. ## 4. Model Invocation ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \ --host 0.0.0.0 \ --port 5000 \ --trust-remote-code \ --tp 4 \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_3 ``` ### 4.1 Basic Usage (OpenAI-Compatible API) SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:5000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16", messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "Give me 3 bullet points about SGLang."}, ], temperature=0.6, max_tokens=1024, ) print("Reasoning:", resp.choices[0].message.reasoning_content, "\nContent:", resp.choices[0].message.content) print("\n") ``` Output: ```text Output theme={null} Reasoning: Okay, the user is asking for 3 bullet points about SGLang. Let me recall what I know about SGLang. It's a framework for serving large language models, right? Developed by the team at UC Berkeley and others. First, I should verify the key features. SGLang is known for its high-performance serving capabilities, especially with features like Radix Attention and chunked prefill. Those are important points to mention...(more tokens) Content: - SGLang introduces **Radix Attention**, an innovative attention mechanism that significantly reduces KV cache memory usage and improves computational efficiency during LLM serving by reusing intermediate states across tokens. - It features **chunked prefill** for handling long prompts efficiently, breaking input sequences into manageable chunks to minimize latency and memory pressure while maintaining high throughput. - Designed for **high-performance LLM serving**, SGLang achieves superior throughput and lower latency compared to traditional systems (like vLLM or TensorRT-LLM) through optimized kernel fusion, dynamic batching, and seamless integration with Hugging Face Transformers. ``` Streaming chat completion: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:5000/v1", api_key="EMPTY", ) stream = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16", messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "What are the first 5 prime numbers?"} ], temperature=0.7, max_tokens=1024, stream=True, ) for chunk in stream: delta = chunk.choices[0].delta if delta and delta.content: print(delta.content, end="", flush=True) ``` Output: ```text Output theme={null} The first 5 prime numbers are: **2, 3, 5, 7, 11**. ### Explanation: - A **prime number** is a natural number greater than 1 that has no positive divisors other than 1 and itself. - **2** is the smallest and only even prime number. - **3** is prime (divisible only by 1 and 3). - **4** is not prime (divisible by 2). - **5** is prime. - **6** is not prime (divisible by 2 and 3). - **7** is prime. - **8, 9, 10** are not prime. - **11** is prime (the fifth in the sequence). Note: **1 is not considered a prime number** by definition, as it has only one positive divisor. This list is universally accepted in mathematics. Let me know if you'd like to explore more primes or related concepts! 😊 ``` ### 4.2 Reasoning The model supports two modes — Reasoning ON (default) vs OFF. This can be toggled by setting `enable_thinking` to `False`, as shown below. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:5000/v1", api_key="EMPTY", ) # Reasoning on (default) print("Reasoning on") resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Write a haiku about GPUs. Please make thinking process short."} ], temperature=1, max_tokens=1024, ) print(f"Reasoning: \n{resp.choices[0].message.reasoning_content[:200]}... \nContent: \n{resp.choices[0].message.content[:200]}...") print("\n") # Reasoning off print("Reasoning off") resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Give me 3 facts about SGLang."} ], temperature=0, max_tokens=256, extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) print(f"Content: \n{resp.choices[0].message.reasoning_content[:200]}...") ``` Output: ```text Output theme={null} Reasoning on Reasoning: We need to output a haiku about GPUs, with short thinking process. Probably we just need to produce the haiku. No extra commentary needed. Provide a haiku: 5-7-5 syllable lines about GPUs. Let's deci... Content: Silicon hearts beat Paint vivid worlds with bright light GPU dreams rise... Reasoning off Content: Certainly! Here are three accurate and informative facts about **SGLang**: 1. **SGLang is a high-performance serving system for large language models (LLMs)** Developed by researchers at UC Berk... ``` ### 4.3 Tool Calling Call functions using the OpenAI Tools schema and inspect returned `tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:5000/v1", api_key="EMPTY", ) # Tool calling via OpenAI tools schema TOOLS = [ { "type": "function", "function": { "name": "calculate_tip", "parameters": { "type": "object", "properties": { "bill_total": { "type": "integer", "description": "The total amount of the bill" }, "tip_percentage": { "type": "integer", "description": "The percentage of tip to be applied" } }, "required": ["bill_total", "tip_percentage"] } } } ] completion = client.chat.completions.create( model="nemotron", messages=[ {"role": "system", "content": ""}, {"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"} ], tools=TOOLS, temperature=0.6, top_p=0.95, max_tokens=512, stream=False ) print(completion.choices[0].message.reasoning_content) print(completion.choices[0].message.tool_calls) ``` Output: ```text Output theme={null} The user wants to calculate a 15% tip on a $50 bill. I have a function called calculate_tip that takes bill_total and tip_percentage as parameters. The bill_total is $50, and tip_percentage is 15. I need to call the function with these values. Let me do that. [ChatCompletionMessageFunctionToolCall(id='call_ced9a83a3baa448e9d587aaf', function=Function(arguments='{"bill_total": 50, "tip_percentage": 15}', name='calculate_tip'), type='function', index=0)] ``` ### 4.4 Controlling Reasoning Budget The `reasoning_budget` parameter allows you to limit the length of the model's reasoning trace. When the reasoning output reaches the specified token budget, the model will attempt to gracefully end the reasoning at the next newline character. If no newline is encountered within 500 tokens after reaching the budget threshold, the reasoning trace will be forcibly terminated at `reasoning_budget + 500` tokens. ```python Example theme={null} from typing import Any, Dict, List import openai from transformers import AutoTokenizer class ThinkingBudgetClient: def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str): self.base_url = base_url self.api_key = api_key self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path) self.client = openai.OpenAI(base_url=self.base_url, api_key=self.api_key) def chat_completion( self, model: str, messages: List[Dict[str, Any]], reasoning_budget: int = 512, max_tokens: int = 1024, **kwargs, ) -> Dict[str, Any]: assert ( max_tokens > reasoning_budget ), f"reasoning_budget must be smaller than max_tokens. Given {max_tokens=} and {reasoning_budget=}" # 1. first call chat completion to get reasoning content response = self.client.chat.completions.create( model=model, messages=messages, max_tokens=reasoning_budget, **kwargs ) reasoning_content = response.choices[0].message.reasoning_content or "" if "" not in reasoning_content: # reasoning content is too long, closed with a period (.) reasoning_content = f"{reasoning_content}.\n\n\n" reasoning_tokens_used = len( self.tokenizer.encode(reasoning_content, add_special_tokens=False) ) remaining_tokens = max_tokens - reasoning_tokens_used assert ( remaining_tokens > 0 ), f"remaining tokens must be positive. Given {remaining_tokens=}. Increase max_tokens or lower reasoning_budget." # 2. append reasoning content to messages and call completion messages.append({"role": "assistant", "content": reasoning_content}) prompt = self.tokenizer.apply_chat_template( messages, tokenize=False, continue_final_message=True, ) response = self.client.completions.create( model=model, prompt=prompt, max_tokens=remaining_tokens, **kwargs ) response_data = { "reasoning_content": reasoning_content.strip().strip("").strip(), "content": response.choices[0].text, "finish_reason": response.choices[0].finish_reason, } return response_data ``` Usage example with `reasoning_budget=128`: ```python Example theme={null} SERVED_MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16" # Client client = ThinkingBudgetClient( base_url="http://127.0.0.1:5000/v1", api_key="null", tokenizer_name_or_path=SERVED_MODEL_NAME ) resp = client.chat_completion( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Write a haiku about GPUs."} ], temperature=1, max_tokens=512, reasoning_budget=128 ) print("Reasoning:", resp["reasoning_content"], "\nContent:", resp["content"]) ``` Output: ```text Output theme={null} Reasoning: Okay, the user wants a haiku about GPUs. Let me recall what a haiku is: a traditional Japanese poem with three lines, 5-7-5 syllable structure. So I need to make sure the syllable count is exact. First, I should think about what makes GPUs interesting. They're used for graphics rendering, parallel processing, AI, gaming, etc. Maybe focus on their speed, power, or how they handle many tasks at once. Let me brainstorm some words and phrases related to GPUs: silicon, cores, transistors, parallel, rendering, pixels, frames per second, CUDA, tensor. Content: Silicon minds awaken, Thousands of cores hum in unison— Lightning paints the void. ``` *** ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: H200 (4x) * Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 * Tensor Parallelism: 4 * SGLang Version: main branch * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \ --trust-remote-code \ --tp 4 \ --max-running-requests 1024 \ --host 0.0.0.0 \ --port 5000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 5000 \ --model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 4096 \ --max-concurrency 256 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 256 Successful requests: 4096 Benchmark duration (s): 623.49 Total input tokens: 2081726 Total input text tokens: 2081726 Total generated tokens: 2087288 Total generated tokens (retokenized): 2044666 Request throughput (req/s): 6.57 Input token throughput (tok/s): 3338.85 Output token throughput (tok/s): 3347.77 Peak output token throughput (tok/s): 6349.00 Peak concurrent requests: 270 Total token throughput (tok/s): 6686.62 Concurrency: 250.35 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 38108.46 Median E2E Latency (ms): 37186.80 P90 E2E Latency (ms): 69325.24 P99 E2E Latency (ms): 77776.90 ---------------Time to First Token---------------- Mean TTFT (ms): 436.49 Median TTFT (ms): 114.90 P99 TTFT (ms): 6938.11 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 75.02 Median TPOT (ms): 76.02 P99 TPOT (ms): 92.27 ---------------Inter-Token Latency---------------- Mean ITL (ms): 74.07 Median ITL (ms): 38.45 P95 ITL (ms): 230.42 P99 ITL (ms): 242.70 Max ITL (ms): 7181.72 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark **Environment** * Hardware: H200 (4x) * Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 * Tensor Parallelism: 4 * SGLang Version: main branch **Launch Model** ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \ --trust-remote-code \ --tp 4 \ --reasoning-parser nemotron_3 ``` **Run Benchmark** ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. **Test Results:** ```text Output theme={null} Accuracy: 0.950 Invalid: 0.000 Latency: 21.442 s Output throughput: 996.815 token/s ``` #### 5.2.2 MMLU Benchmark **Run Benchmark** ```bash Command theme={null} sgl-eval run mmlu \ --base-url http://127.0.0.1:30000/v1 \ --num-threads 64 --max-tokens 2048 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. **Test Results:** ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.730 subject: anatomy, #q:135, acc: 0.830 subject: astronomy, #q:152, acc: 0.934 subject: business_ethics, #q:100, acc: 0.830 subject: clinical_knowledge, #q:265, acc: 0.879 subject: college_biology, #q:144, acc: 0.931 subject: college_chemistry, #q:100, acc: 0.620 subject: college_computer_science, #q:100, acc: 0.840 subject: college_mathematics, #q:100, acc: 0.820 subject: college_medicine, #q:173, acc: 0.821 subject: college_physics, #q:102, acc: 0.794 subject: computer_security, #q:100, acc: 0.880 subject: conceptual_physics, #q:235, acc: 0.919 subject: econometrics, #q:114, acc: 0.746 subject: electrical_engineering, #q:145, acc: 0.828 subject: elementary_mathematics, #q:378, acc: 0.926 subject: formal_logic, #q:126, acc: 0.857 subject: global_facts, #q:100, acc: 0.570 subject: high_school_biology, #q:310, acc: 0.952 subject: high_school_chemistry, #q:203, acc: 0.828 subject: high_school_computer_science, #q:100, acc: 0.940 subject: high_school_european_history, #q:165, acc: 0.861 subject: high_school_geography, #q:198, acc: 0.939 subject: high_school_government_and_politics, #q:193, acc: 0.990 subject: high_school_macroeconomics, #q:390, acc: 0.928 subject: high_school_mathematics, #q:270, acc: 0.700 subject: high_school_microeconomics, #q:238, acc: 0.966 subject: high_school_physics, #q:151, acc: 0.834 subject: high_school_psychology, #q:545, acc: 0.960 subject: high_school_statistics, #q:216, acc: 0.852 subject: high_school_us_history, #q:204, acc: 0.926 subject: high_school_world_history, #q:237, acc: 0.937 subject: human_aging, #q:223, acc: 0.879 subject: human_sexuality, #q:131, acc: 0.939 subject: international_law, #q:121, acc: 0.934 subject: jurisprudence, #q:108, acc: 0.898 subject: logical_fallacies, #q:163, acc: 0.914 subject: machine_learning, #q:112, acc: 0.821 subject: management, #q:103, acc: 0.903 subject: marketing, #q:234, acc: 0.944 subject: medical_genetics, #q:100, acc: 0.980 subject: miscellaneous, #q:783, acc: 0.945 subject: moral_disputes, #q:346, acc: 0.861 subject: moral_scenarios, #q:895, acc: 0.542 subject: nutrition, #q:306, acc: 0.902 subject: philosophy, #q:311, acc: 0.884 subject: prehistory, #q:324, acc: 0.920 subject: professional_accounting, #q:282, acc: 0.805 subject: professional_law, #q:1534, acc: 0.681 subject: professional_medicine, #q:272, acc: 0.923 subject: professional_psychology, #q:612, acc: 0.889 subject: public_relations, #q:110, acc: 0.800 subject: security_studies, #q:245, acc: 0.837 subject: sociology, #q:201, acc: 0.960 subject: us_foreign_policy, #q:100, acc: 0.920 subject: virology, #q:166, acc: 0.590 subject: world_religions, #q:171, acc: 0.906 Total latency: 150.267 Average accuracy: 0.841 ``` # NVIDIA Nemotron3-Ultra Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3-Ultra Deploy NVIDIA Nemotron3-Ultra with SGLang - 550B hybrid MoE model (55B active) with 1M context window, BF16/NVFP4 support, built for long-running autonomous agents. ## 1. Model Introduction `NVIDIA Nemotron3-Ultra` is an open frontier reasoning model in the Nemotron 3 family, built for long-running autonomous agents. It is optimized for complex orchestration across coding, deep research, enterprise workflows, and EDA use cases where agents must sustain reasoning across many steps and large context windows. Nemotron 3 Ultra is a 550B parameter hybrid MoE model that activates only 55B parameters per forward pass, delivering frontier reasoning accuracy with high-throughput inference. It supports a 1M token context window so agents can keep conversation history, tool outputs, and plan state in view across persistent workflows. Architecture and key features: * **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture, enabling efficient routing and sequence modeling in a single stack. * **Long-horizon agentic reasoning:** Tuned for agents that plan, call tools, inspect results, recover from failures, and continue working across long task horizons — coding, deep research, enterprise automation, and EDA. * **1M token context window:** Sustains coherent agent state across extended workflows without re-ingestion. * **BF16 and NVFP4 quantization:** Deployable from multi-node H100 down to a single Blackwell node with NVFP4. * **Multi-environment RL post-training:** Post-trained with reinforcement learning across multiple environments for robust reasoning and reliable agentic behavior. * **Open weights, open data, open recipes:** Customizable for domain-specific agents and deployable across your own infrastructure. **Modalities:** Input: text — Output: text **Supported GPUs:** * **BF16:** 16×H100, 16×H200, 8×B200/B300 * **NVFP4:** 4/8×B200/B300, 4×GB200/GB300 Available model variants on HuggingFace: * [`nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) * [`nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4) ## 2. SGLang Installation Nemotron3-Ultra support is included in the latest stable release. ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance tuning. ### 3.1 Basic Configuration **Interactive Command Generator**: select model precision, hardware, tensor parallelism, and common knobs to generate a launch command. The generator only emits a runnable command for combinations that NVIDIA / SGLang have validated. Selecting an unverified tuple (e.g. NVFP4 on H100/H200, BF16 with TP=4 on H100, …) is **blocked** — the command pane shows an explicit error and the verified support matrix instead of a launch line, so unvalidated commands can't be copied by accident. ### 3.2 Configuration Tips * **Attention backend**: **H100/H200**: Use flash attention 3 backend by default. **B200/GB200/B300/GB300**: Set `--attention-backend trtllm_mha`. The flashinfer default breaks the overlap scheduler on Blackwell, so `trtllm_mha` is required there. * **Mamba scheduler strategy**: Always launch with `--mamba-radix-cache-strategy extra_buffer`. This hybrid Transformer-Mamba model requires the `extra_buffer` strategy for correct scheduling of its Mamba state. * **Mamba backend**: The Mamba layers use the Triton SSM kernels by default. For better performance, set `--mamba-backend flashinfer` to use the FlashInfer Mamba kernels instead. * **Mamba SSM precision**: The SSM state dtype defaults to the model config value. Set `--mamba-ssm-dtype float16` to store the Mamba states in FP16, which reduces mamba cache memory without significant accuracy loss. * **Mamba SSM stochastic rounding**: When storing the Mamba states in FP16, add `--enable-mamba-cache-stochastic-rounding` to round SSM cache writes stochastically and reduce accumulation bias. It requires `--mamba-ssm-dtype float16` and CUDA; with the default `--mamba-backend triton` it additionally requires SM100. Use `--mamba-cache-philox-rounds` to control the number of Philox rounds (`0` uses the backend default). * **TP support**: To set tp size, use `--tp <4|8|16>`. Recommended pairings: * BF16: `--tp 16` on H100/H200, `--tp 8` on B200/B300 * NVFP4: `--tp 4` or `--tp 8` on B200/B300, `--tp 4` on GB200/GB300 * **Multi-node BF16 on H100**: The 16×H100 BF16 setup spans two nodes. Use `--dist-init-addr :5000 --nnodes 2 --node-rank <0|1>` on each node and keep `--tp 16`. * **DP attention**: By default the attention layers are tensor-parallel (sharded across all TP ranks). Enabling DP attention (the toggle above, or `--attn-dp-size `) instead runs attention as `N` data-parallel groups: each DP rank serves its own slice of the requests with its own KV cache. `--attn-dp-size` must divide `--tp`. * **Expert parallel (EP)**: This MoE only supports `ep_size == 1` (off) or `ep_size == tp_size`. To enable expert parallelism, set `--ep ` with the same value as `--tp`. * **Multi-token prediction (MTP)**: Enable MTP speculative decoding (the toggle above) for low latency. * **FP8 KV cache**: To enable fp8 kv cache, set `--kv-cache-dtype fp8_e4m3`. This is enabled by default on the NVFP4 checkpoint. * **Reasoning parser**: Set `--reasoning-parser nemotron_3` to enable structured reasoning traces (`reasoning_content` field in the response). * **Tool calling**: Set `--tool-call-parser qwen3_coder` to enable tool calling support. ## 4. Model Invocation ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 \ --trust-remote-code \ --tp 8 \ --mamba-radix-cache-strategy extra_buffer \ --attention-backend trtllm_mha \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_3 ``` ### 4.1 Basic Usage (OpenAI-Compatible API) SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16", messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "Give me 3 bullet points about SGLang."}, ], temperature=0.6, max_tokens=1024, ) print("Reasoning:", resp.choices[0].message.reasoning_content, "\nContent:", resp.choices[0].message.content) print("\n") ``` Output: ```text Output theme={null} Reasoning: The user wants 3 bullet points about SGLang. Let me recall what I know about SGLang — it's a high-performance serving framework for large language models with a focus on structured generation and efficient KV cache reuse...(more tokens) Content: - **Radix Attention** — SGLang reuses KV cache across requests sharing a common prefix, dramatically reducing memory and compute for multi-turn agent loops and few-shot workloads. - **OpenAI-compatible API and structured generation** — Drop-in replacement for the OpenAI client, with first-class support for constrained decoding (JSON schema, regex) and OpenAI-style tool calling. - **High-throughput serving on NVIDIA GPUs** — Continuous batching, chunked prefill, FP8/NVFP4 quantization, and optimized CUDA kernels deliver state-of-the-art throughput across H100, H200, B200, and GB200. ``` Streaming chat completion: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) stream = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16", messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "What are the first 5 prime numbers?"} ], temperature=0.7, max_tokens=1024, stream=True, ) for chunk in stream: delta = chunk.choices[0].delta if delta and delta.content: print(delta.content, end="", flush=True) ``` Output: ```text Output theme={null} The first 5 prime numbers are: **2, 3, 5, 7, 11**. ### Explanation: - A **prime number** is a natural number greater than 1 whose only positive divisors are 1 and itself. - **2** is the smallest prime and the only even prime. - **3, 5, 7, 11** are each divisible only by 1 and themselves. - **1** is not prime by definition (it has only one positive divisor). - **4, 6, 8, 9, 10** are composite. ``` ### 4.2 Reasoning The model supports two modes — Reasoning ON (default) vs OFF. This can be toggled by setting `enable_thinking` to `False`, as shown below. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) # Reasoning on (default) print("Reasoning on") resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Plan a 3-step approach to debug a flaky integration test. Keep the thinking process short."} ], temperature=1, max_tokens=1024, ) print(f"Reasoning: \n{resp.choices[0].message.reasoning_content[:200]}... \nContent: \n{resp.choices[0].message.content[:200]}...") print("\n") # Reasoning off print("Reasoning off") resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Give me 3 facts about SGLang."} ], temperature=0, max_tokens=256, extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) print(f"Content: \n{resp.choices[0].message.content[:200]}...") ``` Output: ```text Output theme={null} Reasoning on Reasoning: The user wants a short reasoning chain plus a 3-step debug plan for a flaky integration test. I'll think briefly about common causes (timing/race, shared state, external service variance) and pick a t... Content: 1. **Reproduce deterministically** — run the test in a loop (e.g. 50–100x) with logging at the suspected race points to confirm the failure rate and surface ordering. 2. **Isolate state** — re-run with... Reasoning off Content: Here are 3 facts about SGLang: 1. **High-performance LLM serving system** developed at UC Berkeley with contributions from a broad open-source community, focused on throughput and latency at scale. ... ``` ### 4.3 Tool Calling Call functions using the OpenAI Tools schema and inspect returned `tool_calls`. The server must be launched with `--tool-call-parser qwen3_coder`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) # Tool calling via OpenAI tools schema TOOLS = [ { "type": "function", "function": { "name": "search_codebase", "description": "Search the project codebase for a symbol or pattern.", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "The symbol, function name, or regex to search for" }, "path": { "type": "string", "description": "Optional sub-path to restrict the search to" } }, "required": ["query"] } } } ] completion = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16", messages=[ {"role": "system", "content": "You are a coding agent. Use tools to inspect the repo before answering."}, {"role": "user", "content": "Where is the `RadixCache` class defined?"} ], tools=TOOLS, temperature=0.6, top_p=0.95, max_tokens=512, stream=False ) print(completion.choices[0].message.reasoning_content) print(completion.choices[0].message.tool_calls) ``` Output: ```text Output theme={null} The user is asking where the RadixCache class is defined. I should search the codebase for the symbol "RadixCache" to find the file and line. I'll call search_codebase with that query. [ChatCompletionMessageFunctionToolCall(id='call_8a7f2c4e1b9d4a3e8c2f1d6b', function=Function(arguments='{"query": "class RadixCache"}', name='search_codebase'), type='function', index=0)] ``` ### 4.4 Controlling Reasoning Budget The `reasoning_budget` parameter allows you to limit the length of the model's reasoning trace. When the reasoning output reaches the specified token budget, the model will attempt to gracefully end the reasoning at the next newline character. If no newline is encountered within 500 tokens after reaching the budget threshold, the reasoning trace will be forcibly terminated at `reasoning_budget + 500` tokens. ```python Example theme={null} from typing import Any, Dict, List import openai from transformers import AutoTokenizer class ThinkingBudgetClient: def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str): self.base_url = base_url self.api_key = api_key self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path) self.client = openai.OpenAI(base_url=self.base_url, api_key=self.api_key) def chat_completion( self, model: str, messages: List[Dict[str, Any]], reasoning_budget: int = 512, max_tokens: int = 1024, **kwargs, ) -> Dict[str, Any]: assert ( max_tokens > reasoning_budget ), f"reasoning_budget must be smaller than max_tokens. Given {max_tokens=} and {reasoning_budget=}" # 1. first call chat completion to get reasoning content response = self.client.chat.completions.create( model=model, messages=messages, max_tokens=reasoning_budget, **kwargs ) reasoning_content = response.choices[0].message.reasoning_content or "" if "" not in reasoning_content: # reasoning content is too long, closed with a period (.) reasoning_content = f"{reasoning_content}.\n\n\n" reasoning_tokens_used = len( self.tokenizer.encode(reasoning_content, add_special_tokens=False) ) remaining_tokens = max_tokens - reasoning_tokens_used assert ( remaining_tokens > 0 ), f"remaining tokens must be positive. Given {remaining_tokens=}. Increase max_tokens or lower reasoning_budget." # 2. append reasoning content to messages and call completion messages.append({"role": "assistant", "content": reasoning_content}) prompt = self.tokenizer.apply_chat_template( messages, tokenize=False, continue_final_message=True, ) response = self.client.completions.create( model=model, prompt=prompt, max_tokens=remaining_tokens, **kwargs ) response_data = { "reasoning_content": reasoning_content.strip().strip("").strip(), "content": response.choices[0].text, "finish_reason": response.choices[0].finish_reason, } return response_data ``` Usage example with `reasoning_budget=256`: ```python Example theme={null} SERVED_MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16" # Client client = ThinkingBudgetClient( base_url="http://127.0.0.1:30000/v1", api_key="null", tokenizer_name_or_path=SERVED_MODEL_NAME ) resp = client.chat_completion( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Outline a research plan to evaluate the throughput of two MoE serving strategies."} ], temperature=1, max_tokens=1024, reasoning_budget=256 ) print("Reasoning:", resp["reasoning_content"], "\nContent:", resp["content"]) ``` Output: ```text Output theme={null} Reasoning: The user wants a research plan to compare throughput of two MoE serving strategies. I should outline goals, baselines, datasets, metrics (tokens/s, TTFT, ITL, MFU), variables to sweep (TP, batch size, sequence length, concurrency), and statistical handling. Keep it concise since reasoning_budget is 256... Content: **Research plan** 1. **Define goal & metrics** — peak token throughput (input+output), TTFT, P99 ITL, MFU; measured at fixed accuracy. 2. **Choose baselines** — Strategy A (TP-only) vs Strategy B (TP + expert-parallel). Hold model checkpoint, precision, and KV-cache dtype constant. 3. **Sweep** — `{batch ∈ 1,4,16,64, concurrency ∈ 16,64,256, seq_len ∈ 1k,8k,32k}` per strategy. 4. **Workload** — `sglang.bench_serving --dataset-name random` with matched input/output budgets. 5. **Analysis** — per-config throughput table + roofline overlay; bootstrap CIs over 3 reruns to bound noise. ``` *** ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: GB200 (4x) * Model: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 * Tensor Parallelism: 4 * SGLang Version: main branch * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \ --trust-remote-code \ --tp 4 \ --mamba-radix-cache-strategy extra_buffer \ --attention-backend trtllm_mha \ --max-running-requests 1024 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 4096 \ --max-concurrency 256 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 256 Successful requests: 4096 Benchmark duration (s): 1184.58 Total input tokens: 2081726 Total input text tokens: 2081726 Total generated tokens: 2087288 Total generated tokens (retokenized): 1990224 Request throughput (req/s): 3.46 Input token throughput (tok/s): 1757.35 Output token throughput (tok/s): 1762.05 Peak output token throughput (tok/s): 3150.00 Peak concurrent requests: 266 Total token throughput (tok/s): 3519.40 Concurrency: 249.55 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 72169.95 Median E2E Latency (ms): 71994.47 P90 E2E Latency (ms): 99898.56 P99 E2E Latency (ms): 107119.61 ---------------Time to First Token---------------- Mean TTFT (ms): 40057.33 Median TTFT (ms): 41375.93 P99 TTFT (ms): 46377.89 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 63.15 Median TPOT (ms): 63.65 P99 TPOT (ms): 78.16 ---------------Inter-Token Latency---------------- Mean ITL (ms): 63.14 Median ITL (ms): 35.92 P95 ITL (ms): 178.10 P99 ITL (ms): 182.10 Max ITL (ms): 2466.36 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark **Environment** * Hardware: GB200 (4x) * Model: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 * Tensor Parallelism: 4 * SGLang Version: main branch **Launch Model** ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \ --trust-remote-code \ --tp 4 \ --mamba-radix-cache-strategy extra_buffer \ --attention-backend trtllm_mha \ --reasoning-parser nemotron_3 ``` **Run Benchmark** ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. **Test Results:** ```text Output theme={null} Accuracy: 0.970 Invalid: 0.000 Latency: 29.129 s Output throughput: 745.333 token/s ``` #### 5.2.2 MMLU Benchmark **Run Benchmark** ```bash Command theme={null} sgl-eval run mmlu \ --base-url http://127.0.0.1:30000/v1 \ --num-threads 64 --max-tokens 2048 ``` **Test Results:** ```text Output theme={null} TBD ``` *** # Nemotron3.5-Lightning Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3.5-Lightning Deploy NVIDIA Nemotron 3.5 Lightning with SGLang — NVFP4 serving with MTP, DFlash, and DSpark speculative decoding, reasoning, and tool calling. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv SGLANG_BUILD_RUST_EXTS=none uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev-nemotron3-5-lightning ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware and recipe to generate the launch command. Every platform publishes four operating points: **Balanced** (no speculation) plus three speculative decoders — **MTP**, **DFlash**, and **DSpark**. Use the Playground below to explore knobs beyond them. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction **NVIDIA Nemotron 3.5 Lightning** is a 30B-A3B hybrid reasoning LLM. See the Hugging Face model cards below for architecture and evaluation details.
Checkpoint Precision Use
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 NVFP4 Serving — the checkpoint this page deploys
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 BF16 Full-precision reference
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash W4A16 DFlash speculative draft model
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark W4A16 DSpark speculative draft model
MTP needs no separate download — the draft head is embedded in the target checkpoint. ## 2. Usage The server speaks the OpenAI API. With `--reasoning-parser nemotron_3` enabled, the thinking trace lands in `message.reasoning_content` and the answer in `message.content`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://127.0.0.1:8000/v1", api_key="null", ) response = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Briefly explain: what is SGLang?"}, ], temperature=1.0, top_p=0.95, max_tokens=1024, ) choice = response.choices[0] print("Reasoning:", choice.message.reasoning_content) print("Content:", choice.message.content) ``` ### 2.1 Tool Calling With `--tool-call-parser qwen3_coder` enabled, structured tool calls are returned in `message.tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://127.0.0.1:8000/v1", api_key="null", ) TOOLS = [ { "type": "function", "function": { "name": "calculate_tip", "parameters": { "type": "object", "properties": { "bill_total": {"type": "integer", "description": "The total amount of the bill"}, "tip_percentage": {"type": "integer", "description": "The percentage of tip to be applied"}, }, "required": ["bill_total", "tip_percentage"], }, }, } ] response = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4", messages=[{"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}], tools=TOOLS, max_tokens=1024, ) choice = response.choices[0] print("Content:", choice.message.content) print("Tool calls:", choice.message.tool_calls) ``` # GPT-OSS Source: https://docs.sglang.io/cookbook/autoregressive/OpenAI/GPT-OSS ## 1.Model Introduction [GPT-OSS](https://huggingface.co/openai/gpt-oss-20b) is an advanced large language model developed by OpenAI designed for power reasoning, agentic tasks, and versatile developer use cases. It has versions with two model sizes. * **gpt-oss-120b** — for production, general purpose, high reasoning use cases that fit into a single 80GB GPU (like NVIDIA H100 80GB or AMD MI300X 192GB) (117B parameters with 5.1B active parameters) * **gpt-oss-20b** — for lower latency, and local or specialized use cases (21B parameters with 3.6B active parameters) GPT-OSS introduces several groundbreaking innovations: * **Configurable reasoning effort**: Easily adjust the reasoning effort (low, medium, high) based on your specific use case and latency needs. * **Full chain-of-thought**: Gain complete access to the model’s reasoning process, facilitating easier debugging and increased trust in outputs. It’s not intended to be shown to end users. * **Fine-tunable**: Fully customize models to your specific use case through parameter fine-tuning. * **Agentic capabilities**: Use the models’ native capabilities for function calling, web browsing, Python code execution, and Structured Outputs. * **MXFP4 quantization**: The models were post-trained with MXFP4 quantization of the MoE weights, making gpt-oss-120b run on a single 80GB GPU (like NVIDIA H100 80GB or AMD MI300X 192GB) and the gpt-oss-20b model run within 16GB of memory. All evals were performed with the same MXFP4 quantization. ## 2.SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3.Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The GPT-OSS series comes in two sizes. Recommended starting configurations vary depending on hardware. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities. ### 3.2 Configuration Tips * **Native web search:** Set `EXA_API_KEY` in the SGLang server environment to enable built-in web search (Exa). No `--tool-server` is required, and requests are tagged with `x-exa-integration: sglang`. * **Web search defaults:** `numResults=10`, search `type="auto"`, and `contents.highlights=true`. Override with `SGLANG_EXA_NUM_RESULTS`, `SGLANG_EXA_SEARCH_TYPE`, and `SGLANG_EXA_INCLUDE_HIGHLIGHTS`. * **Python tool:** Add `--tool-server demo` to enable the Python interpreter. Runs in a Docker sandbox by default; set `PYTHON_EXECUTION_BACKEND=UV` to run on the host (model-generated code executes locally — use with care). * **MCP tool servers:** For production, point SGLang at external MCP SSE servers with `--tool-server ip-1:port-1,ip-2:port-2`. * **Responses API:** GPT-OSS supports OpenAI's Responses API (`client.responses.create`) in addition to the standard Chat Completions API (see section 4.2.4). * **Use Python 3.12** when running the demo Python tool. * **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4.Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser GPT-OSS supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model openai/gpt-oss-120b \ --reasoning-parser gpt-oss \ --tp 8 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="openai/gpt-oss-120b", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user asks: "Solve this problem step by step: What is 15% of 240?" So we need to provide step-by-step solution. Compute 15% of 240: 0.15 * 240 = 36. Provide steps: convert percent to decimal, multiply, maybe use fraction. Provide answer. =============== Content ================= **Step‑by‑step solution** 1. **Understand what “percent” means** “15 %” means 15 out of every 100 parts, i.e. the fraction \(\displaystyle \frac{15}{100}\). 2. **Convert the percent to a decimal (or fraction)** \[ \frac{15}{100}=0.15 \] 3. **Set up the multiplication** To find 15 % of 240 we multiply 240 by the decimal 0.15: \[ 240 \times 0.15 \] 4. **Do the multiplication** One convenient way is to break it into two easier parts: \[ 240 \times 0.15 = 240 \times \left(\frac{15}{100}\right) = \frac{240 \times 15}{100} \] - First compute \(240 \times 15\): \[ 240 \times 15 = 240 \times (10 + 5) = 2400 + 1200 = 3600 \] - Then divide by 100: \[ \frac{3600}{100} = 36 \] 5. **Write the result** \[ 15\% \text{ of } 240 = 36 \] --- **Answer:** \(36\) ``` #### 4.2.2 Tool Calling GPT-OSS supports tool calling capabilities. Enable the tool call parser: **Python Example (without Thinking Process):** Start sglang server: ```shell Command theme={null} python -m sglang.launch_server \ --model openai/gpt-oss-120b \ --tool-call-parser gpt-oss \ --tp 8 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="openai/gpt-oss-120b", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"🔧 Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Python Example (with Thinking Process):** Start sglang server: ```shell Command theme={null} python -m sglang.launch_server \ --model openai/gpt-oss-120b \ --reasoning-parser gpt-oss \ --tool-call-parser gpt-oss \ --tp 8 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="openai/gpt-oss-120b", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"🔧 Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= User asks: "What's the weather in Beijing?" We need to get current weather. Use function get_weather with location "Beijing". No unit specified; default? Probably use default (maybe Celsius). We can specify unit as "celsius". We'll call function. =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="openai/gpt-oss-120b", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The current weather in Beijing is 22 °C and sunny. Let me know if you’d like a forecast for the next few days or any other details!" ``` #### 4.2.3 EAGLE3 Speculative Decoding SGLang supports speculative decoding for GPT-OSS models using the EAGLE3 algorithm. This can significantly improve decoding speed, especially for small batch sizes. ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path openai/gpt-oss-120b \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path lmsys/EAGLE3-gpt-oss-120b-bf16 \ --tp 2 ``` The spec-v2 overlap scheduler is enabled by default. It improves performance by overlapping draft and verification stages. Pass `--disable-overlap-schedule` to disable. #### 4.2.4 Responses API and Built-in Tools GPT-OSS supports the OpenAI Responses API with built-in tool use (web search and Python interpreter). Set `EXA_API_KEY` to enable native web search; add `--tool-server demo` only when you also want the Python tool: ```shell Command theme={null} export EXA_API_KEY=YOUR_EXA_KEY # Optional: server-side Exa tuning (defaults shown) export SGLANG_EXA_NUM_RESULTS=10 export SGLANG_EXA_SEARCH_TYPE=auto export SGLANG_EXA_INCLUDE_HIGHLIGHTS=true # Optional: run Python tool on host instead of Docker (model code executes locally) export PYTHON_EXECUTION_BACKEND=UV python3 -m sglang.launch_server \ --model-path openai/gpt-oss-120b \ --tp 2 ``` For production, use external MCP SSE servers instead of `demo`: ```shell Command theme={null} mcp run -t sse browser_server.py:mcp mcp run -t sse python_server.py:mcp python -m sglang.launch_server \ --model-path openai/gpt-oss-120b \ --tool-server ip-1:port-1,ip-2:port-2 \ --tp 2 ``` **Example using Responses API:** ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="sk-123456") search_tools = [{"type": "web_search"}] python_tools = [{"type": "code_interpreter"}] # Configurable reasoning effort: "high", "medium", or "low" response = client.responses.create( model="openai/gpt-oss-120b", instructions="You are a helpful assistant.", reasoning_effort="high", input="In one sentence, explain the transformer architecture.", ) print(response.output_text) # Web search (requires EXA_API_KEY on the SGLang server) response = client.responses.create( model="openai/gpt-oss-120b", instructions="You are a helpful assistant, you can search the web when needed.", input="Search the web for the latest news about Nvidia stock price", tools=search_tools, ) print(response.output_text) # Python tool (requires launching SGLang with --tool-server demo) response = client.responses.create( model="openai/gpt-oss-120b", instructions="You are a helpful assistant, you could use python tool to execute code.", input="Use python tool to calculate the sum of 29138749187 and 29138749187", tools=python_tools, ) print(response.output_text) # Output: The sum is 58,277,498,374. ``` ## 5.Benchmark ### 5.1 Speed Benchmark * Hardware: NVIDIA B200 GPU (8x) * Tensor Parallelism: 8 * Model: openai/gpt-oss-120b * sglang version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. #### 5.1.1 Latency-Sensitive Benchmark * Server Command: ```shell Command theme={null} python -m sglang.launch_server \ --model openai/gpt-oss-120b \ --tp 8 ``` * Test Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --num-prompt 100 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 100 Benchmark duration (s): 52.35 Total input tokens: 33178 Total input text tokens: 33178 Total input vision tokens: 0 Total generated tokens: 21251 Total generated tokens (retokenized): 20868 Request throughput (req/s): 1.91 Input token throughput (tok/s): 633.76 Output token throughput (tok/s): 405.93 Peak output token throughput (tok/s): 433.00 Peak concurrent requests: 8 Total token throughput (tok/s): 1039.69 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 523.30 Median E2E Latency (ms): 389.91 ---------------Time to First Token---------------- Mean TTFT (ms): 33.71 Median TTFT (ms): 31.79 P99 TTFT (ms): 108.98 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 2.31 Median TPOT (ms): 2.31 P99 TPOT (ms): 2.39 ---------------Inter-Token Latency---------------- Mean ITL (ms): 2.31 Median ITL (ms): 2.31 P95 ITL (ms): 2.35 P99 ITL (ms): 2.38 Max ITL (ms): 3.54 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Server Command: ```shell Command theme={null} python -m sglang.launch_server \ --model openai/gpt-oss-120b \ --tp 8 ``` * Test Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --num-prompt 1000 \ --max-concurrency 100 ``` **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 24.76 Total input tokens: 297156 Total input text tokens: 297156 Total input vision tokens: 0 Total generated tokens: 192432 Total generated tokens (retokenized): 187145 Request throughput (req/s): 40.39 Input token throughput (tok/s): 12003.57 Output token throughput (tok/s): 7773.26 Peak output token throughput (tok/s): 13780.00 Peak concurrent requests: 156 Total token throughput (tok/s): 19776.83 Concurrency: 89.23 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2208.97 Median E2E Latency (ms): 1591.11 ---------------Time to First Token---------------- Mean TTFT (ms): 102.94 Median TTFT (ms): 31.53 P99 TTFT (ms): 674.32 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.31 Median TPOT (ms): 11.00 P99 TPOT (ms): 91.28 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.00 Median ITL (ms): 5.75 P95 ITL (ms): 25.35 P99 ITL (ms): 43.18 Max ITL (ms): 621.42 ================================================== ``` ### 5.2 Accuracy Benchmark ### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:8000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * **Results**: * GPT-OSS-120b ```text Output theme={null} Accuracy: 0.880 Invalid: 0.005 Latency: 5.262 s Output throughput: 12143.675 token/s ``` * GPT-OSS-20b ```text Output theme={null} Accuracy: 0.535 Invalid: 0.165 Latency: 4.157 s Output throughput: 19589.165 token/s ``` # MiniCPM-V 4.6 Source: https://docs.sglang.io/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6 ## 1. Model Introduction MiniCPM-V 4.6 is the next-generation multimodal model from [OpenBMB](https://huggingface.co/openbmb), the team behind the MiniCPM-V series. The model combines a **Qwen3.5-style hybrid LLM backbone** (Gated Delta Net + full attention) with a **NaViT-packed vision encoder** that handles arbitrary aspect ratios and high-resolution slicing natively, plus end-to-end video support. OpenBMB ships two variants on HuggingFace: * [`openbmb/MiniCPM-V-4.6`](https://huggingface.co/openbmb/MiniCPM-V-4.6) — base instruct model. Use this for general multimodal serving; thinking mode is still available per-request via `chat_template_kwargs.enable_thinking=true`. * [`openbmb/MiniCPM-V-4.6-Thinking`](https://huggingface.co/openbmb/MiniCPM-V-4.6-Thinking) — thinking-tuned variant with stronger chain-of-thought behavior. Pair with the same `--reasoning-parser qwen3` flag. **Key Features:** * **Hybrid LLM backbone**: Qwen3.5-style mix of Gated Delta Net (linear-attention) layers and full-attention layers, providing long-context efficiency without giving up modeling power. * **Native variable-resolution vision**: NaViT-packed vision encoder with mid-ViT merger and per-image window attention. Images of any aspect ratio are processed without forced letterboxing. * **High-resolution slicing**: Source image plus a configurable grid of slice tiles (up to 9 tiles in the open test variant) lets the model reason over fine detail in 1280×720+ images. * **Video**: Frame-by-frame multi-modal data items routed through the same vision encoder; any number of frames per request. * **Reasoning Parser**: switchable thinking mode (Qwen3.5 lineage), exposed via `chat_template_kwargs.enable_thinking` per request and SGLang's `--reasoning-parser qwen3` on the server side. * **Tool Calling**: Qwen3.5-style `…` XML format, surfaced as OpenAI-compatible `message.tool_calls` via SGLang's `--tool-call-parser qwen3_coder`. Composes with thinking mode and with image / video inputs. **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0). ## 2. SGLang Installation Pull the nightly Docker image (rolling tag, tracks `main`): ```bash theme={null} # CUDA 13 (Hopper / Blackwell, default) docker pull lmsysorg/sglang:dev # CUDA 12 (Ampere or older drivers). Final build: the CUDA 12 lane is retired, # so this tag no longer tracks main. docker pull lmsysorg/sglang:dev-cu12 ``` For the general SGLang installation guide (PyPI, source, Docker) see the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to generate the appropriate deployment command. The `Variant` toggle switches between `openbmb/MiniCPM-V-4.6` (base) and `openbmb/MiniCPM-V-4.6-Thinking`. The `Reasoning Parser` and `Tool Call Parser` toggles add `--reasoning-parser qwen3` and `--tool-call-parser qwen3_coder` respectively; see §4.4 for usage details. ### 3.2 Configuration Tips * **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs. * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). * The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload. * Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities. * To speed up weight loading for this large model, add `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'` to the launch command. * **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`. * **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200/B300. * For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors. * Multi-image and high-resolution images: the image processor produces one source patch plus per-slice tile patches; each is its own `MultimodalDataItem`. No special server-side flag needed. * Video: decoded frame-by-frame through the same image-style slicer. No extra flag needed; pass `video_url` in the OpenAI chat completion request. * **Chunked Prefill**: For high-concurrency vision benchmarking with many large/sliced images, pass `--chunked-prefill-size -1` to disable prefill chunking. The default chunked-prefill path can mis-split a request across an image boundary in `mm_utils.embed_mm_inputs` and crash the server; disabling chunking sidesteps this at the cost of higher TTFT under concurrency. For interactive serving leave the default on. ## 4. Model Invocation Deploy the model on an H200: ```bash Command theme={null} sglang serve --model-path openbmb/MiniCPM-V-4.6 \ --trust-remote-code \ --dtype bfloat16 \ --mem-fraction-static 0.15 \ --mamba-radix-cache-strategy extra_buffer \ --page-size 64 \ --host 0.0.0.0 --port 30000 ``` ### 4.1 Basic Usage (Image) ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="openbmb/MiniCPM-V-4.6", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://www.ilankelman.org/stopsigns/australia.jpg", }, }, {"type": "text", "text": "Describe this image in one sentence."}, ], } ], max_tokens=200, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} A black SUV drives past a Chinese-style gate with a red stop sign and traditional architecture, while storefronts and street signs line the sidewalk. ``` ### 4.2 High-Resolution / Sliced Images The image processor automatically picks a slice grid (up to 9 tiles) for high-resolution inputs. A 1280×720 source produces grid `[2, 3]` * 7 patches with `tgt_sizes=[(24, 44), 6×(28, 36)]`, byte-for-byte matching the HF reference implementation. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="openbmb/MiniCPM-V-4.6", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg", }, }, {"type": "text", "text": "Describe this image in one sentence."}, ], } ], max_tokens=200, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} The Statue of Liberty stands tall against a cloudy sky, holding a torch aloft and a document in her left hand, symbolizing freedom and enlightenment. ``` ### 4.3 Video Input ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="openbmb/MiniCPM-V-4.6", messages=[ { "role": "user", "content": [ { "type": "video_url", "video_url": {"url": ""}, }, {"type": "text", "text": "Describe what happens in this video in one sentence."}, ], } ], max_tokens=200, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print(response.choices[0].message.content) ``` **Output Example** (run against an 8-frame synthetic test mp4 of shifting colored squares): ```text Output theme={null} The video shows a grid of colored squares moving in a random pattern. ``` ### 4.4 Advanced Usage #### 4.4.1 Reasoning Parser Pass `--reasoning-parser qwen3` to the server (toggle "Reasoning Parser" on in §3.1, default) so SGLang splits each response on the `` / `` boundaries: the pre-`` block goes to `reasoning_content`, the post-`` text to `content`. Per-request, the chat template's `enable_thinking` flag toggles whether the model actually emits reasoning. * **Thinking mode** (default, `enable_thinking=true`): assistant prompt ends with `\n`; the model writes reasoning, closes with ``, then the answer. `reasoning_content` and `content` are both populated. * **Instruct mode** (`enable_thinking=false`): the chat template injects an empty `` placeholder so the model emits no thinking tokens; `reasoning_content` ends up empty. ```python Example (thinking mode) theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="openbmb/MiniCPM-V-4.6", messages=[{"role": "user", "content": "Reply with the single word 'hi'. No explanation."}], max_tokens=200, ) msg = response.choices[0].message print("reasoning_content:", msg.reasoning_content) print("content :", msg.content) ``` ```text Output theme={null} reasoning_content: Got it, let's see. The user wants a reply with "hi" and no explanation. So I need to just say "hi" as the response. ... content : hi ``` ```python Example (instruct mode) theme={null} response = client.chat.completions.create( model="openbmb/MiniCPM-V-4.6", messages=[{"role": "user", "content": "Reply with the single word 'hi'. No explanation."}], max_tokens=200, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) msg = response.choices[0].message print("reasoning_content:", msg.reasoning_content) print("content :", msg.content) ``` ```text Output theme={null} reasoning_content: content : hi ``` #### 4.4.2 Tool Calling Pass `--tool-call-parser qwen3_coder` to the server (toggle "Tool Call Parser" on in §3.1) so SGLang extracts `` blocks from the model output into the OpenAI-style `message.tool_calls` field (with `finish_reason="tool_calls"`). The model speaks the Qwen3.5 XML tool-call format (`v`); the `qwen3_coder` parser is the right one. Tool calls compose with both reasoning modes and with image / video inputs. Do **not** use `--tool-call-parser qwen` for MiniCPM-V 4.6 — that parser expects the older Qwen2.5 JSON format `{"name":..., "arguments":...}`, but both public 4.6 variants emit the Qwen3.5-style XML format with nested `` and `` tags. With `qwen` the outer `` markers match but the inner JSON parse fails, so `tool_calls` returns empty and the raw markup is left in `content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city.", "parameters": { "type": "object", "properties": { "location": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, }, ] response = client.chat.completions.create( model="openbmb/MiniCPM-V-4.6", messages=[{"role": "user", "content": "What is the weather in San Francisco? Use the tool."}], tools=tools, max_tokens=200, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) choice = response.choices[0] print("finish_reason:", choice.finish_reason) for tc in choice.message.tool_calls or []: print(f" {tc.function.name}({tc.function.arguments})") ``` ```text Output theme={null} finish_reason: tool_calls get_weather({"location": "San Francisco", "unit": "celsius"}) ``` To get the final natural-language answer, feed the tool's result back as a `tool` role message and call the API again with the same `tools` list — the model emits `finish_reason="stop"` with the answer in `content`. ## 5. Benchmark **Common Test Environment (all benchmarks below):** * Hardware: 1× NVIDIA H200 (141 GB), single GPU (no TP / DP) * Docker Image: `lmsysorg/sglang:dev` (transformers 5.6.0, sgl-kernel 0.4.2.post1) * Precision: BF16 **Common Server Launch Command:** ```bash Command theme={null} CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server \ --model-path openbmb/MiniCPM-V-4.6 \ --trust-remote-code \ --dtype bfloat16 \ --mem-fraction-static 0.5 \ --mamba-radix-cache-strategy extra_buffer \ --chunked-prefill-size -1 \ --host 0.0.0.0 --port 30000 ``` (`--chunked-prefill-size -1` is required for the vision throughput run; see §3.2.) ### 5.1 Accuracy Benchmark #### 5.1.1 MMMU Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/mmmu/bench_sglang.py --port 30000 --concurrency 48 --max-new-tokens 2048 ``` * Test Result ``` {'Accounting': {'acc': 0.767, 'num': 30}, 'Agriculture': {'acc': 0.533, 'num': 30}, 'Architecture_and_Engineering': {'acc': 0.4, 'num': 30}, 'Art': {'acc': 0.6, 'num': 30}, 'Art_Theory': {'acc': 0.667, 'num': 30}, 'Basic_Medical_Science': {'acc': 0.533, 'num': 30}, 'Biology': {'acc': 0.333, 'num': 30}, 'Chemistry': {'acc': 0.333, 'num': 30}, 'Clinical_Medicine': {'acc': 0.467, 'num': 30}, 'Computer_Science': {'acc': 0.333, 'num': 30}, 'Design': {'acc': 0.533, 'num': 30}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.333, 'num': 30}, 'Economics': {'acc': 0.633, 'num': 30}, 'Electronics': {'acc': 0.5, 'num': 30}, 'Energy_and_Power': {'acc': 0.633, 'num': 30}, 'Finance': {'acc': 0.533, 'num': 30}, 'Geography': {'acc': 0.367, 'num': 30}, 'History': {'acc': 0.533, 'num': 30}, 'Literature': {'acc': 0.7, 'num': 30}, 'Manage': {'acc': 0.367, 'num': 30}, 'Marketing': {'acc': 0.733, 'num': 30}, 'Materials': {'acc': 0.367, 'num': 30}, 'Math': {'acc': 0.567, 'num': 30}, 'Mechanical_Engineering': {'acc': 0.333, 'num': 30}, 'Music': {'acc': 0.267, 'num': 30}, 'Overall': {'acc': 0.527, 'num': 900}, 'Overall-Art and Design': {'acc': 0.517, 'num': 120}, 'Overall-Business': {'acc': 0.607, 'num': 150}, 'Overall-Health and Medicine': {'acc': 0.553, 'num': 150}, 'Overall-Humanities and Social Science': {'acc': 0.617, 'num': 120}, 'Overall-Science': {'acc': 0.473, 'num': 150}, 'Overall-Tech and Engineering': {'acc': 0.443, 'num': 210}, 'Pharmacy': {'acc': 0.667, 'num': 30}, 'Physics': {'acc': 0.767, 'num': 30}, 'Psychology': {'acc': 0.567, 'num': 30}, 'Public_Health': {'acc': 0.767, 'num': 30}, 'Sociology': {'acc': 0.667, 'num': 30}} eval out saved to ./val_sglang.json Overall accuracy: 0.527 ``` ### 5.2 Speed Benchmark We use SGLang's built-in `bench_serving` tool with random text prompts (1000 input / 1000 output tokens) to characterize text-only serving performance. #### 5.2.1 Latency Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model openbmb/MiniCPM-V-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 7.47 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 3554 Request throughput (req/s): 1.34 Input token throughput (tok/s): 816.44 Output token throughput (tok/s): 564.73 Peak output token throughput (tok/s): 690.00 Peak concurrent requests: 4 Total token throughput (tok/s): 1381.17 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 746.20 Median E2E Latency (ms): 590.05 P90 E2E Latency (ms): 1446.13 P99 E2E Latency (ms): 1709.38 ---------------Time to First Token---------------- Mean TTFT (ms): 138.12 Median TTFT (ms): 103.70 P99 TTFT (ms): 330.79 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1.44 Median TPOT (ms): 1.44 P99 TPOT (ms): 1.45 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.44 Median ITL (ms): 1.45 P95 ITL (ms): 1.49 P99 ITL (ms): 1.57 Max ITL (ms): 5.79 ================================================== ``` #### 5.2.2 Throughput Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model openbmb/MiniCPM-V-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 1000 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 47.07 Total input tokens: 502493 Total input text tokens: 502493 Total generated tokens: 500251 Total generated tokens (retokenized): 469844 Request throughput (req/s): 21.24 Input token throughput (tok/s): 10675.32 Output token throughput (tok/s): 10627.69 Peak output token throughput (tok/s): 25911.00 Peak concurrent requests: 130 Total token throughput (tok/s): 21303.01 Concurrency: 97.24 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4576.94 Median E2E Latency (ms): 4331.97 P90 E2E Latency (ms): 8634.07 P99 E2E Latency (ms): 9636.44 ---------------Time to First Token---------------- Mean TTFT (ms): 206.50 Median TTFT (ms): 184.72 P99 TTFT (ms): 624.23 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.73 Median TPOT (ms): 9.16 P99 TPOT (ms): 13.63 ---------------Inter-Token Latency---------------- Mean ITL (ms): 8.75 Median ITL (ms): 0.05 P95 ITL (ms): 29.95 P99 ITL (ms): 108.91 Max ITL (ms): 448.40 ================================================== ``` ### 5.3 Vision Speed Benchmark We use SGLang's built-in `bench_serving` tool with random images. Each request has 128 input text tokens, one 720p image, and 1024 output tokens. #### 5.3.1 Latency Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model openbmb/MiniCPM-V-4.6 \ --dataset-name image \ --image-count 1 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 10.26 Total input tokens: 767 Total input text tokens: 750 Total input vision tokens: 17 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.97 Input token throughput (tok/s): 74.77 Output token throughput (tok/s): 411.39 Peak output token throughput (tok/s): 654.00 Peak concurrent requests: 2 Total token throughput (tok/s): 486.16 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1024.04 Median E2E Latency (ms): 897.99 P90 E2E Latency (ms): 1584.25 P99 E2E Latency (ms): 1781.78 ---------------Time to First Token---------------- Mean TTFT (ms): 416.94 Median TTFT (ms): 403.18 P99 TTFT (ms): 477.49 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1.44 Median TPOT (ms): 1.44 P99 TPOT (ms): 1.45 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.44 Median ITL (ms): 1.44 P95 ITL (ms): 1.48 P99 ITL (ms): 1.56 Max ITL (ms): 2.89 ================================================== ``` #### 5.3.2 Throughput Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model openbmb/MiniCPM-V-4.6 \ --dataset-name image \ --image-count 1 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 360.01 Total input tokens: 79925 Total input text tokens: 78283 Total input vision tokens: 1642 Total generated tokens: 510855 Total generated tokens (retokenized): 430289 Request throughput (req/s): 2.78 Input token throughput (tok/s): 222.01 Output token throughput (tok/s): 1419.01 Peak output token throughput (tok/s): 19620.00 Peak concurrent requests: 105 Total token throughput (tok/s): 1641.02 Concurrency: 99.69 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 35888.57 Median E2E Latency (ms): 35321.48 P90 E2E Latency (ms): 41017.37 P99 E2E Latency (ms): 60343.22 ---------------Time to First Token---------------- Mean TTFT (ms): 35096.32 Median TTFT (ms): 34301.37 P99 TTFT (ms): 59966.25 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1.63 Median TPOT (ms): 1.45 P99 TPOT (ms): 10.15 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.58 Median ITL (ms): 0.12 P95 ITL (ms): 0.23 P99 ITL (ms): 0.77 Max ITL (ms): 2086.12 ================================================== ``` # MiniCPM5-2B Source: https://docs.sglang.io/cookbook/autoregressive/OpenBMB/MiniCPM5-2B Deploy OpenBMB MiniCPM5-2B with SGLang — a 2.5B dense Llama-architecture on-device model with 131K context, thinking mode, XML tool calling and DSpark speculative decoding on H200, RTX PRO 6000, RTX 5090 and DGX Spark. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow "sglang>=0.5.12" ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your card to generate the launch command. MiniCPM5-2B is a 2.5B dense model and runs single-GPU at TP=1 on every supported card, so the page ships one operating point per card plus an optional **DSpark** speculative-decoding row. Speed numbers exist for the RTX 5090 and DGX Spark cells — a single card at random 1024/1024, recorded as Mean. The H200 and RTX PRO 6000 cells are pending measurement, and no accuracy numbers have been taken on any platform yet. The DSpark overlay carries no speed numbers on any card. ## Playground The Playground is where you experiment with **SGLang features beyond the recipes above**. The Deploy panel emits this model's documented launch recipes; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction **MiniCPM5-2B** is the second model in OpenBMB's MiniCPM5 series, following MiniCPM5-1B. It is a dense 2B-class Transformer built for on-device assistants, local deployment, coding agents and tool-use workflows — scenarios where a compact model with a small deployment footprint is preferred. OpenBMB reports 2B-class open-source SOTA within its comparison set (average 53.9), with its clearest advantages in code reasoning, math reasoning, long-context understanding, tool use and agentic tasks. The checkpoint uses the standard `LlamaForCausalLM` architecture — no custom kernels and no model-code fork — so SGLang loads it through the stock Llama path. It is 2,516,756,480 parameters (1,981,982,720 non-embedding) over 42 layers with GQA (16 query heads, 2 KV heads), and a native context length of 131,072 tokens. Post-training runs SFT → RL → On-Policy Distillation, which merges 16 RL expert models into the single released checkpoint. Weights are released under the [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) license.
Checkpoint Precision Role on this page
openbmb/MiniCPM5-2B BF16 The served model in every cell above (final release, post-trained with RL + OPD).
openbmb/MiniCPM5-2B-DSpark BF16 draft Draft model loaded by the DSPARK row of the Deploy panel; not served on its own.
OpenBMB also publishes SFT-only, mid-training and base checkpoints, plus GGUF, MLX and GPTQ-Int4 exports for llama.cpp / Ollama / LM Studio / Apple Silicon. Those target other runtimes and are not part of the SGLang matrix above. **Recommended generation:** `temperature=1.0`, `top_p=0.95` (informational — SGLang reads the checkpoint's `generation_config.json`; do not hardcode these in client code). **Resources:** [HuggingFace](https://huggingface.co/openbmb/MiniCPM5-2B) · [ModelScope](https://www.modelscope.cn/models/OpenBMB/MiniCPM5-2B) · [GitHub](https://github.com/OpenBMB/MiniCPM) · [Tech report](https://arxiv.org/pdf/2506.07900). ## 2. Advanced Usage ### 2.1 Thinking Mode Thinking is controlled by the chat template's `enable_thinking` flag, passed per request through `chat_template_kwargs`. The `qwen3` reasoning parser — already in every generated command, and toggleable from the **Parsers** card in the [Playground above](#playground) — splits the `` segment into `message.reasoning_content` and leaves the final answer in `message.content`. Drop the flag and the thinking text stays inline in `content`, closing tag and all. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") # Thinking on (chat-template default). resp = client.chat.completions.create( model="openbmb/MiniCPM5-2B", messages=[{"role": "user", "content": "What is 15% of 240?"}], extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) # Thinking off — a direct answer, no deliberation segment. resp = client.chat.completions.create( model="openbmb/MiniCPM5-2B", messages=[{"role": "user", "content": "What is 15% of 240?"}], extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print("Without thinking:", resp.choices[0].message.content) ``` ```text Output theme={null} Reasoning: We are asked: "What is 15% of 240?" This is a simple percentage calculation. To find 15% of 240, we multiply 240 by 15% (which is 0.15). So: 240 × 0.15 = 36. Alternatively, we can think of it as (15/100) × 240 = (15 × 240)/100 = 3600/100 = 36. So the answer is 36. We need to respond in a helpful way. The user might be testing or seeking quick answer. Let's provide a clear response with explanation if needed, but since it's straightforward, we can just give the answer directly or briefly explain. Answer: 15% of 240 is **36**. To calculate: \( 240 \times 0.15 = 36 \) Reasoning: None Answer: To find 15% of 240, multiply 240 by 15% (which is 0.15): \[ 240 \times 0.15 = 36 \] So, 15% of 240 is **36**. ``` ### 2.2 Tool Calling MiniCPM5-2B emits XML-style tool calls (`...`), and SGLang's built-in `minicpm5` detector converts them to OpenAI-compatible `tool_calls`, parallel calls included. The parser is already in every command the Deploy panel generates; the **Tool Call Parser** chip in the Playground's **Parsers** card is an opt-out, not an opt-in. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city.", "parameters": { "type": "object", "properties": { "city": {"type": "string", "description": "City name"}, "date": {"type": "string", "description": "YYYY-MM-DD"}, }, "required": ["city"], }, }, }] resp = client.chat.completions.create( model="openbmb/MiniCPM5-2B", messages=[{"role": "user", "content": "What is the weather in Beijing today?"}], tools=tools, ) msg = resp.choices[0].message # With the qwen3 reasoning parser on, the thinking segment lands in # `reasoning_content` and `content` may be empty on a tool-call turn. print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) for call in (msg.tool_calls or []): print("Tool:", call.function.name, call.function.arguments) ``` ```text Output theme={null} Reasoning: The user is asking about the weather in Beijing today. I have access to a get_weather function that can help with this. Let me call it to get the current weather for Beijing. Content: Tool: get_weather {"city": "Beijing"} ``` # PPLX-Decider-v1-27B Source: https://docs.sglang.io/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B Deploy PPLX-Decider-v1-27B with SGLang. Perplexity's decision model, fine-tuned from Qwen3.8-27B, returns calibrated probabilities for choice, yes or no, and score questions on /v1/systemone, on one NVIDIA H200 or GB300 in BF16. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. PPLX-Decider support landed after v0.5.21, so until a release includes it, install a [nightly build](/docs/get-started/install#nightly-builds). ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` The image is multi-arch, so the same tag runs on x86 H200 hosts and Arm GB300 hosts. For how to launch it, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your GPU to generate the launch command. The model runs on one GPU in BF16, and the decision support needs no flag: SGLang reads the checkpoint's `decision_config.json`, loads its decision readout, and answers on `/v1/systemone` with the prompt, answer codes, and calibrated temperature the model was trained with. The GB300 recipe adds `--disable-radix-cache` for speed, as explained in [Configuration Tips](#2-configuration-tips). The H200 recipe is the original one, validated against the checkpoint's own reference implementation on one H200 when support landed ([#42183](https://github.com/sgl-project/sglang/pull/42183)), and it has no speed numbers yet. The GB300 recipe adds `--disable-radix-cache` (see [Configuration Tips](#2-configuration-tips)) and was measured end to end on one GB300 at SGLang commit `70f0b7351e`: four request shapes from 1 to 256 concurrent requests, Belebele and WinoGrande accuracy, and per-question parity with the reference implementation. [§3](#3-benchmarks) has the method and the full tables. ## Playground The Playground layers SGLang features on top of the recipe above. Any override changes the badge to **Not Verified** until that exact configuration is tested end to end. The [Configuration Tips](#2-configuration-tips) say which overrides were measured on GB300 and what they did. Tensor parallelism times data-parallel replicas must fit the GPUs on one node. ## 1. Model Introduction **PPLX-Decider-v1-27B** is a decision model from Perplexity, fine-tuned from [Qwen3.8-27B](/cookbook/autoregressive/Qwen/Qwen3.8-27B) under the Apache-2.0 license. It replaces the language model head with a readout over 255 answer codes and answers choice, yes or no (`noul`), and score questions about a text state and optional images with a probability for every option, without generating text. Each question is one prefill pass, so the cost of a decision is the cost of reading its prompt. Its successor, [PPLX-Decider-v1.1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1.1-27B), adds noncausal full attention and more training data. The backbone is the Qwen3.8-27B hybrid: 64 layers in which 48 Gated DeltaNet (linear attention) layers alternate with 16 gated full-attention layers, a 5120-wide hidden state, and the Qwen3.8 vision encoder, for 26.1B parameters (52 GB in BF16). The checkpoint ships a bare `Qwen3_5Model` backbone, the 255-row readout in `readout.safetensors`, and `decision_config.json` with the answer-code token ids and a calibrated temperature of 2.2076. SGLang serves it as `Qwen3_5ForConditionalGeneration` and writes the readout into the language model head rows of the code tokens, so the probabilities come out of the regular scoring path.
Property Value
Checkpoint perplexity-ai/pplx-decider-v1-27b (BF16)
Base model Qwen3.8-27B, one epoch of supervised fine-tuning on decision data
Question types choice (up to 255 options), noul (yes or no), score (up to 10 levels)
Inputs Text or JSON state, optional images
Prompt length Trained on question prompts of up to 8,192 tokens, and the backbone accepts 262,144
License Apache-2.0
Perplexity reports 85.71% overall accuracy across 11 public benchmarks for this checkpoint, against 74.76% for Qwen3.8-27B, measured through the Perplexity API. The per-benchmark table is on the model card. [§3](#3-benchmarks) measures accuracy and speed on SGLang. **Resources:** [HuggingFace](https://huggingface.co/perplexity-ai/pplx-decider-v1-27b), [reference inference code](https://huggingface.co/perplexity-ai/pplx-decider-v1-27b/blob/main/inference.py), [Decision models in SGLang](/docs/supported-models/decision_models). ## 2. Configuration Tips * **Route.** Send requests to `/v1/systemone`, the System One API the checkpoint was trained for. `/v1/decisions` refuses this checkpoint because its labels and prompt are not the ones the readout learned, and chat or generation requests are not meaningful without a language model head. * **Clients.** Clients of the System One API, including the TypeSafe SDKs, work by pointing their base URL at the server. See [System One compatible API](/docs/supported-models/decision_models#system-one-compatible-api) for the request and response reference. * **Images.** Pass each image in `images` as base64 bytes, a base64 data URL, or an `http(s)` URL. They precede the text in every question, as in the checkpoint's reference code. The checkpoint's processor resizes every image to between 65,536 and 262,144 pixels, so one image costs 64 to 256 prompt tokens: a 1024×768 screenshot is 234 tokens, and a 4K one is no more expensive than 1080p (252). Crop to the region that matters rather than sending a larger image. * **What a decision costs.** Each question is its own prompt: an \~84-token wrapper (system message, chat template, answer instructions), the state, the question, and its options, read in one prefill pass. Several questions in one request are scored together, but each re-reads the whole state, so the cost of a request is roughly *questions × state tokens*. * **Prefix cache.** On this hybrid model the prefix cache resumes from where an earlier prompt ended, because that is where the Gated DeltaNet state is saved, not from any shared prefix. An identical request measured 215 ms cold and 56 ms cached on GB300, but a different question about the same state, and the other questions in the same request, reused none of the state. The GB300 recipe therefore turns the cache off with `--disable-radix-cache`: unique-state traffic loses no reuse, each request holds one state slot instead of five, and the per-request state bookkeeping goes away, which took a short question from 59 to 51 ms and an image request from 126 to 87 ms with the attention kernel held fixed. Turn it back on only if your traffic repeats identical requests, the same state and the same question, often enough to matter. * **Prompt length.** The model was fine-tuned on question prompts of up to 8,192 tokens, and the reference code refuses longer ones. SGLang serves the backbone's full 262,144-token context, so keep states within the trained length. A longer prompt is answered, but its accuracy is untested. * **Throughput is compute-bound.** A decision is pure prefill, and one GB300 reads about 27,000 prompt tokens per second in BF16 once batches are full: about 71 requests per second for a 382-token question, and 3.3 for an 8K-token state. Past that point, more concurrency only adds queueing. The tables in [§3](#3-benchmarks) map concurrency to latency, so pick the highest concurrency that fits your latency budget. * **Memory.** The weights take 52 GB, and the default `--mem-fraction-static` hands the rest of the GPU to the KV and Gated DeltaNet state pools. The startup log line `max_running_requests is capped to ... by the mamba state cache` is expected: on GB300, each request reserves five state slots with the prefix cache on (a cap of 118) and one with it off (593). Even the lower cap only binds for questions shorter than about 140 tokens, because a prefill batch fills its 16,384-token chunk first. `--mamba-ssm-dtype bfloat16` halves the state pool but measured no faster, so keep the checkpoint's FP32 state. * **GB300 kernels.** With the prefix cache off, SGLang picks the TRT-LLM attention kernel with 64-token pages on GB300 (Triton with 1-token pages while the cache is on), and that kernel is where the long-state gain comes from: an 8K-token state drops from 377 to 325 ms and saturates at 3.30 instead of 2.87 requests per second, the same as forcing `--attention-backend trtllm_mha --page-size 64` with the cache on. FlashInfer's Gated DeltaNet prefill (`--linear-attn-prefill-backend flashinfer`) adds another 4-8% on text, taking the 8K-token state to 300 ms and 3.56 requests per second, but it made image requests slower (101 vs 87 ms alone, 9% less throughput), so the recipe leaves it to the Playground for text-only traffic. A larger prefill chunk (`--chunked-prefill-size 32768`) did not help: one 16,384-token chunk already keeps the GPU busy. * **More GPUs.** The model fits one GPU, so add throughput with data-parallel replicas: `--dp-size 2` on two GB300s served 139 short questions per second (1.96× one GPU) and 6.5 8K-token states per second, and because the questions of one request are spread over the replicas, it also cut four questions on a 1K-token state from 179 to 104 ms. The extra dispatch hop added about 15 ms to a lone short question. Tensor parallelism only pays off for latency on long states: `--tp 2` took an 8K-token state from 325 to 208 ms, but delivered less throughput than two replicas and made a short question slower (68 ms), because the all-reduce dominates small batches. SGLang Model Gateway does not route `/v1/systemone` yet, so to spread load over several servers instead of `--dp-size`, use a plain HTTP load balancer. * **FP8.** `--quantization fp8` quantizes the BF16 weights at load time and measured about 20% more throughput on GB300 (84 vs 70 requests per second for short questions), with the same Belebele and WinoGrande scores. But its probabilities moved by 0.010 on average and up to 0.20 against the reference implementation, four times the drift of any BF16 configuration, and 0.9% of answers changed their top option. The calibrated probabilities are this model's output, so the recipes stay in BF16, and the Playground offers FP8 for workloads that only use the top choice. * **Reproducibility.** Two launches of the same recipe returned bit-identical probabilities for the same requests at the same concurrency. Against the checkpoint's reference implementation, the largest probability difference per question averages 0.003 (see [§3](#3-benchmarks)), because batch composition and kernels differ, and near-ties at 0.500 can flip. ## 3. Benchmarks All numbers in this section come from one NVIDIA GB300 (288 GB) at SGLang commit `70f0b7351e`, with checkpoint revision `5117a6c`, served with the GB300 recipe above unless a row says otherwise. **Method.** A closed-loop client keeps a fixed number of `/v1/systemone` requests in flight. Every request carries a unique random state, so requests share nothing but the system message, and each concurrency level starts after 8 warmup requests and a cache flush. Latency is the median end-to-end time of a request, and prompt tokens come from each response's `usage.input_tokens`. Four request shapes cover the common cases: one four-option choice question about a 256-token state, four questions (two choice, one yes or no, one score) about a 1,024-token state, one question about an 8,192-token state, and one question about a 1024×768 JPEG with a 64-token state. ### 3.1 Speed
Request Concurrent Median latency Requests/s Prompt tokens/s
Short question, 382 tokens 1 51 ms 19.4 7,413
4 103 ms 32.8 12,525
16 267 ms 59.0 22,531
64 882 ms 70.8 27,039
256 3,521 ms 71.1 27,164
Four questions on a 1K-token state, 4,540 tokens 1 179 ms 5.6 (22 decisions/s) 25,291
8 1,321 ms 6.0 (24 decisions/s) 27,332
32 5,206 ms 6.0 (24 decisions/s) 27,208
8K-token state, 8,224 tokens 1 325 ms 3.08 25,297
4 1,220 ms 3.26 26,785
16 4,796 ms 3.30 27,175
One 1024×768 image, 429 tokens 1 87 ms 11.5 4,945
16 414 ms 37.5 16,078
64 1,170 ms 52.6 22,572
One GB300 tops out at about 27,000 prompt tokens per second for every text shape, so requests per second follow from prompt length. Latency rises with concurrency once that ceiling is reached, so choose the concurrency from the latency you can accept. A decision has no output tokens, so in the benchmark card above TTFT is the whole request latency, and TPOT and interactivity do not apply. ### 3.2 Accuracy and parity Both sets go through `/v1/systemone` as one `choice` question per item. Belebele sends the passage as the state, the question as the instructions, and the four answers as options `1` to `4`. WinoGrande sends the sentence as the state, asks which option fills the blank, and offers the two candidates. The reference row runs the same items through the checkpoint's own `DecisionModel` one at a time.
Implementation Belebele (eng\_Latn, 900) WinoGrande (xl dev, 1,267)
SGLang, GB300 recipe 96.67% 84.21%
SGLang, no extra flag 96.67% 84.37%
Checkpoint's reference DecisionModel (Transformers, BF16, GB300) 96.67% 84.37%
Perplexity, model card (Perplexity API, its own prompt conversion) 94.00% 83.30%
Against the reference implementation, the recipe picks the same option on 2,161 of 2,167 items, and each of the six differences is a near-tie that one side scores 0.500 against 0.500 and the other 0.486 against 0.514. Per item, the largest probability difference averages 0.003 (99th percentile 0.024, maximum 0.054). Perplexity's figures come from a different prompt conversion, so they are context rather than a target. ### 3.3 What else was measured
Change from the GB300 recipe Measured effect Verdict
Prefix cache on (no extra flag) Short question 58 ms alone, 8K-token state 377 ms alone and 2.87 requests/s saturated (recipe: 51 ms, 325 ms, 3.30) Only for traffic that repeats identical requests
--linear-attn-prefill-backend flashinfer Text 4-8% faster (8K-token state 300 ms, 3.56 requests/s), images slower (101 ms alone, 9% less throughput) Text-only traffic
--dp-size 2 (2 GPUs) 139 short questions/s and 6.5 8K-token states/s, four questions on a 1K-token state in 104 ms, a lone short question 15 ms slower Scale-out
--tp 2 (2 GPUs) 8K-token state 208 ms alone, 124 short questions/s saturated, a lone short question in 68 ms Long states with a latency target
--quantization fp8 About 20% more throughput, same accuracy, probabilities shifted by 0.010 on average and up to 0.20 Only when you use the top choice alone
--chunked-prefill-size 32768 --max-prefill-tokens 32768 No gain (within 4%) Keep the default
--mamba-ssm-dtype bfloat16 No gain, though it halves the state pool Keep the default
```python Example theme={null} import asyncio import aiohttp from datasets import load_dataset URL = "http://localhost:30000/v1/systemone" async def main(): rows = load_dataset("facebook/belebele", "eng_Latn", split="test") limit = asyncio.Semaphore(32) async def ask(session, row): body = { "model": "perplexity-ai/pplx-decider-v1-27b", "state": row["flores_passage"], "questions": { "answer": { "type": "choice", "instructions": row["question"], "criteria": {str(k): row[f"mc_answer{k}"] for k in range(1, 5)}, } }, } async with limit, session.post(URL, json=body) as response: response.raise_for_status() answer = (await response.json())["answers"]["answer"] return answer["choice"] == row["correct_answer_num"] async with aiohttp.ClientSession() as session: correct = await asyncio.gather(*(ask(session, row) for row in rows)) print(f"Belebele eng_Latn: {sum(correct)}/{len(correct)} = {100 * sum(correct) / len(correct):.2f}%") asyncio.run(main()) ``` ```text Output theme={null} Belebele eng_Latn: 870/900 = 96.67% ``` Run it as `python bench.py `. With 270 words, a request is 378 tokens, close to the short shape above. ```python Example theme={null} import asyncio import random import statistics import sys import time import aiohttp BASE = "http://localhost:30000" STATE_WORDS, CONCURRENCY, REQUESTS = int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3]) WORDS = ("customer payment refund invoice order delivery account login error update server outage " "contract price plan report urgent today please help the a and with for from to of in on").split() QUESTION = { "type": "choice", "instructions": "Which team should handle this request?", "criteria": {"billing": "Charges and refunds", "technical_support": "Integration errors", "sales": "Questions about buying a product", "legal": "Contracts and liability"}, } def body(rng): # A unique random state per request, so no request reuses another's cache. state = " ".join(rng.choices(WORDS, k=STATE_WORDS)) return {"model": "perplexity-ai/pplx-decider-v1-27b", "state": state, "questions": {"q": QUESTION}} async def main(): rng = random.Random(0) bodies = [body(rng) for _ in range(REQUESTS)] latencies, tokens = [], 0 async with aiohttp.ClientSession() as session: for item in [body(rng) for _ in range(8)]: # warmup async with session.post(f"{BASE}/v1/systemone", json=item) as response: response.raise_for_status() async with session.post(f"{BASE}/flush_cache") as response: await response.read() queue = iter(bodies) async def worker(): nonlocal tokens for item in queue: start = time.perf_counter() async with session.post(f"{BASE}/v1/systemone", json=item) as response: response.raise_for_status() tokens += (await response.json())["usage"]["input_tokens"] latencies.append(time.perf_counter() - start) start = time.perf_counter() await asyncio.gather(*(worker() for _ in range(CONCURRENCY))) elapsed = time.perf_counter() - start print(f"{REQUESTS / elapsed:.1f} req/s, {tokens / elapsed:.0f} prompt tok/s, " f"P50 {1000 * statistics.median(latencies):.1f} ms, {tokens / REQUESTS:.0f} tokens/request") asyncio.run(main()) ``` ```text Output theme={null} $ python bench.py 270 1 64 19.6 req/s, 7393 prompt tok/s, P50 51.0 ms, 378 tokens/request $ python bench.py 270 64 512 68.4 req/s, 25865 prompt tok/s, P50 899.0 ms, 378 tokens/request ``` ## 4. Advanced Usage ### 4.1 Text Decisions ```python Example theme={null} import requests response = requests.post( "http://localhost:30000/v1/systemone", json={ "model": "perplexity-ai/pplx-decider-v1-27b", "state": "My Stripe integration keeps failing. I'm losing sales. Please help ASAP.", "questions": { "routing": { "type": "choice", "instructions": "Which team should handle this request?", "criteria": { "billing": "Charges and refunds", "technical_support": "Integration errors", "sales": "Questions about buying a product", }, }, "urgency": {"type": "noul", "instructions": "Does this message express urgency?"}, }, }, timeout=60, ) response.raise_for_status() for name, answer in response.json()["answers"].items(): print(name, answer) ``` ```text Output theme={null} routing {'type': 'choice', 'choice': 'technical_support', 'confidence': 0.987854396908662, 'probabilities': {'billing': 0.004840538662952211, 'technical_support': 0.9919029312724412, 'sales': 0.0032565300646064375}} urgency {'type': 'noul', 'noul': 0.9929980993021188} ``` ### 4.2 Image Decisions The example output below came from a mostly red test image. ```python Example theme={null} import base64 from pathlib import Path import requests image = base64.b64encode(Path("screenshot.png").read_bytes()).decode("ascii") response = requests.post( "http://localhost:30000/v1/systemone", json={ "model": "perplexity-ai/pplx-decider-v1-27b", "state": "Look at the supplied image.", "images": [image], "questions": { "color": { "type": "choice", "instructions": "What is the dominant color?", "criteria": {"red": "Red", "green": "Green", "blue": "Blue", "other": "Another color"}, } }, }, timeout=60, ) response.raise_for_status() print(response.json()["answers"]["color"]) ``` ```text Output theme={null} {'type': 'choice', 'choice': 'red', 'confidence': 0.9498445161931509, 'probabilities': {'red': 0.9623833871448632, 'green': 0.009006826727044857, 'blue': 0.00926546931819573, 'other': 0.01934431680989631}} ``` # PPLX-Decider-v1.1-27B Source: https://docs.sglang.io/cookbook/autoregressive/Perplexity/PPLX-Decider-v1.1-27B Deploy PPLX-Decider-v1.1-27B with SGLang. Perplexity's updated decision model reads the whole prompt in its full-attention layers and returns calibrated probabilities for choice, yes or no, and score questions on /v1/systemone, on one NVIDIA GB300 or H200 in BF16. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. PPLX-Decider-v1.1 support lands in [#42645](https://github.com/sgl-project/sglang/pull/42645), after v0.5.21, so until a release includes it, install a [nightly build](/docs/get-started/install#nightly-builds) built after that merge. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` The image is multi-arch, so the same tag runs on x86 H200 hosts and Arm GB300 hosts. For how to launch it, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your GPU to generate the launch command. The model runs on one GPU in BF16 and needs no flag: SGLang reads the checkpoint's `decision_config.json`, runs the full-attention layers over the whole prompt as the model was trained, and turns off the radix cache and chunked prefill, which would show those layers only part of the prompt. The GB300 recipe was measured end to end on one GB300 at [#42645](https://github.com/sgl-project/sglang/pull/42645): four request shapes from 1 to 256 concurrent requests, Belebele and WinoGrande accuracy, and per-question parity with the checkpoint's own reference implementation. The H200 command is the same and has not been run yet. [§3](#3-benchmarks) has the method and the full tables. ## Playground The Playground layers SGLang features on top of the recipe above. Any override changes the badge to **Not Verified** until that exact configuration is tested end to end. The [Configuration Tips](#2-configuration-tips) say which overrides were measured on GB300. Tensor parallelism times data-parallel replicas must fit the GPUs on one node. ## 1. Model Introduction **PPLX-Decider-v1.1-27B** is the successor to [PPLX-Decider-v1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B), Perplexity's decision model fine-tuned from [Qwen3.8-27B](/cookbook/autoregressive/Qwen/Qwen3.8-27B) under the Apache-2.0 license. Like v1, it replaces the language model head with a readout over 255 answer codes and answers choice, yes or no (`noul`), and score questions about a text state and optional images with a probability for every option, without generating text. Two things changed. The 16 full-attention layers no longer use a causal mask, so every token attends to the whole prompt, while the 48 Gated DeltaNet layers stay causal. And training grew from 73,000 to 626,033 rows, most of them from [tasksource](https://github.com/sileod/tasksource). The prompt format, answer codes, and readout layout are the same as v1's, but the calibrated temperature is 1.0087 instead of 2.2076, so probabilities from the two versions are not interchangeable.
Property Value
Checkpoint perplexity-ai/pplx-decider-v1.1-27b (BF16)
Base model Qwen3.8-27B, one epoch of supervised fine-tuning on 626,033 decision rows (530,103 from tasksource)
Attention Noncausal in the 16 full-attention layers, causal in the 48 Gated DeltaNet layers
Question types choice (up to 255 options), noul (yes or no), score (up to 10 levels)
Inputs Text or JSON state, optional images
Prompt length Trained on question prompts of up to 8,192 tokens, and the backbone accepts 262,144
License Apache-2.0
Perplexity reports a Decision Index of 61.56 for v1.1, up from 56.4 for v1:
Decision Index category Jev PPLX-Decider-v1-27B PPLX-Decider-v1.1-27B
Knowledge 51.4 40.9 48.18
Language 62.0 63.5 69.45
Retrieval 55.4 54.9 61.26
Tools 75.1 79.3 78.88
Arts 37.7 39.4 44.66
Overall 57.9 56.4 61.56
**Resources:** [HuggingFace](https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b), [reference model code](https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b/blob/main/source/src/autojev/model.py), [Decision models in SGLang](/docs/supported-models/decision_models). ## 2. Configuration Tips * **Route.** Send requests to `/v1/systemone`, the System One API the checkpoint was trained for. `/v1/decisions` refuses this checkpoint because its labels and prompt are not the ones the readout learned, and chat or generation requests are not meaningful without a language model head. * **Clients.** Clients of the System One API, including the TypeSafe SDKs, work by pointing their base URL at the server. See [System One compatible API](/docs/supported-models/decision_models#system-one-compatible-api) for the request and response reference. * **Images.** Pass each image in `images` as base64 bytes, a base64 data URL, or an `http(s)` URL. They precede the text in every question. The processor resizes every image to between 65,536 and 262,144 pixels, so one image costs 64 to 256 prompt tokens. * **Noncausal attention is automatic.** SGLang reads `attention_mode: noncausal_full_attention` from `decision_config.json` and runs the full-attention layers as encoder layers, which the TRT-LLM, FlashInfer, and Triton attention backends support. It also turns off the radix cache and chunked prefill, logging `Radix cache and chunked prefill are disabled for a decision checkpoint with noncausal full attention`: a cached prefix or a prefill chunk would see only part of a prompt whose every position depends on the rest. These two settings override any flag you pass. A prompt longer than the 16,384-token prefill budget is still read in one pass: an 18,585-token prompt matched the reference implementation to four decimal places on GB300. * **No prefix reuse.** Because every layer output depends on the whole prompt, nothing carries over between requests, not even an identical one. The cost of a request is roughly *questions × prompt tokens*, as on v1. * **Prompt length.** Like v1, the model was trained on question prompts of up to 8,192 tokens, and the reference code refuses longer ones. SGLang serves longer prompts in one pass, but their accuracy is untested. * **Attention backend.** On GB300, Auto resolves to the TRT-LLM kernel with 64-token pages. `--attention-backend flashinfer` and `--attention-backend triton` matched the reference implementation as closely, with the same top option on 99.8% of the accuracy items. The H200 path has not been run yet. * **Speed.** One GB300 reads about 27,000 prompt tokens per second once batches are full, the same as v1. Measured back to back on the same GPU against v1's recipe, v1.1 was within 2% on short questions and four-question requests, and an 8K-token state saturated 6% lower (3.13 against 3.32 requests per second), because noncausal attention computes the full attention matrix in its 16 full-attention layers instead of half of it. * **More GPUs.** The backbone and prompt sizes are v1's, so the [v1 page's scaling measurements](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B#2-configuration-tips) are the reference: data-parallel replicas (`--dp-size N`) for throughput, tensor parallelism only for latency on long prompts. They were not repeated for v1.1. * **Upgrading from v1.** Re-check any thresholds you tuned on v1 probabilities: v1.1 has a different temperature and changed its top answer on 8% of the Belebele and WinoGrande items. ## 3. Benchmarks All numbers in this section come from one NVIDIA GB300 (288 GB) at [#42645](https://github.com/sgl-project/sglang/pull/42645), with checkpoint revision `3b45dea`, served with the GB300 recipe above unless a row says otherwise. The method and request shapes are those of the [v1 page](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B#3-benchmarks), whose speed script works here unchanged. ### 3.1 Speed
Request Concurrent Median latency Requests/s Prompt tokens/s
Short question, 382 tokens 1 53 ms 18.9 7,203
4 106 ms 37.6 14,362
16 275 ms 57.4 21,909
64 931 ms 67.5 25,770
256 3,489 ms 71.1 27,139
Four questions on a 1K-token state, 4,540 tokens 1 183 ms 5.5 (22 decisions/s) 24,782
8 1,315 ms 6.0 (24 decisions/s) 27,398
32 4,882 ms 6.1 (24 decisions/s) 27,649
8K-token state, 8,224 tokens 1 334 ms 2.99 24,620
4 1,280 ms 3.12 25,682
16 5,101 ms 3.13 25,726
One 1024×768 image, 429 tokens 1 118 ms 8.3 3,541
16 466 ms 33.6 14,398
64 1,197 ms 49.9 21,416
### 3.2 Accuracy and parity Both sets go through `/v1/systemone` as one `choice` question per item, with the same conversion as the v1 page. The reference rows run the same items through the checkpoint's own `DecisionModel` one at a time.
Implementation Belebele (eng\_Latn, 900) WinoGrande (xl dev, 1,267)
SGLang, GB300 recipe (TRT-LLM attention) 97.67% 92.42%
SGLang, --attention-backend flashinfer 97.67% 92.58%
SGLang, --attention-backend triton 97.67% 92.42%
Checkpoint's reference DecisionModel (Transformers, BF16, GB300) 97.67% 92.50%
v1.1 served with causal attention, as SGLang did before #42645 97.00% 88.87%
PPLX-Decider-v1-27B on SGLang, for comparison 96.67% 84.21%
Against the reference implementation, the recipe picks the same option on 2,164 of 2,167 items, and each of the three differences is a near-tie within 0.031 of an even split. Per item, the largest probability difference averages 0.002 (99th percentile 0.030, maximum 0.089). Prompts of 11,099 and 18,585 tokens, an image question, and a score question also matched the reference, the largest difference being 0.015 on the score question. Served with causal attention instead, v1.1 changed its top answer on 101 of the 2,167 items and lost 3.6 points on WinoGrande. ```python Example theme={null} import asyncio import aiohttp from datasets import load_dataset URL = "http://localhost:30000/v1/systemone" async def main(): rows = load_dataset("facebook/belebele", "eng_Latn", split="test") limit = asyncio.Semaphore(32) async def ask(session, row): body = { "model": "perplexity-ai/pplx-decider-v1.1-27b", "state": row["flores_passage"], "questions": { "answer": { "type": "choice", "instructions": row["question"], "criteria": {str(k): row[f"mc_answer{k}"] for k in range(1, 5)}, } }, } async with limit, session.post(URL, json=body) as response: response.raise_for_status() answer = (await response.json())["answers"]["answer"] return answer["choice"] == row["correct_answer_num"] async with aiohttp.ClientSession() as session: correct = await asyncio.gather(*(ask(session, row) for row in rows)) print(f"Belebele eng_Latn: {sum(correct)}/{len(correct)} = {100 * sum(correct) / len(correct):.2f}%") asyncio.run(main()) ``` ```text Output theme={null} Belebele eng_Latn: 879/900 = 97.67% ``` ## 4. Advanced Usage ### 4.1 Text Decisions ```python Example theme={null} import requests response = requests.post( "http://localhost:30000/v1/systemone", json={ "model": "perplexity-ai/pplx-decider-v1.1-27b", "state": "My Stripe integration keeps failing. I'm losing sales. Please help ASAP.", "questions": { "routing": { "type": "choice", "instructions": "Which team should handle this request?", "criteria": { "billing": "Charges and refunds", "technical_support": "Integration errors", "sales": "Questions about buying a product", }, }, "urgency": {"type": "noul", "instructions": "Does this message express urgency?"}, }, }, timeout=60, ) response.raise_for_status() for name, answer in response.json()["answers"].items(): print(name, answer) ``` ```text Output theme={null} routing {'type': 'choice', 'choice': 'technical_support', 'confidence': 0.9977658531751561, 'probabilities': {'billing': 0.0014030901008239556, 'technical_support': 0.9985105687834377, 'sales': 8.634111573847184e-05}} urgency {'type': 'noul', 'noul': 0.9995671977710632} ``` ### 4.2 Image Decisions The example output below came from a 512×512 solid red test image. ```python Example theme={null} import base64 from pathlib import Path import requests image = base64.b64encode(Path("screenshot.png").read_bytes()).decode("ascii") response = requests.post( "http://localhost:30000/v1/systemone", json={ "model": "perplexity-ai/pplx-decider-v1.1-27b", "state": "Look at the supplied image.", "images": [image], "questions": { "color": { "type": "choice", "instructions": "What is the dominant color?", "criteria": {"red": "Red", "green": "Green", "blue": "Blue", "other": "Another color"}, } }, }, timeout=60, ) response.raise_for_status() print(response.json()["answers"]["color"]) ``` ```text Output theme={null} {'type': 'choice', 'choice': 'red', 'confidence': 0.9989117512916561, 'probabilities': {'red': 0.9991838134687421, 'green': 6.743375714875418e-05, 'blue': 8.12085765657198e-05, 'other': 0.0006675441975434462}} ``` # Laguna-M.1 Source: https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-M.1 Deploy poolside's Laguna-M.1 — a 225B-parameter Mixture-of-Experts model (23B active) for agentic coding — with SGLang on NVIDIA H200, B200, B300, GB200, and GB300, across BF16, FP8, and NVFP4. ## Deployment Laguna-M.1 support is already on SGLang `main` — **softplus per-element attention-output gating** ([PR #28400](https://github.com/sgl-project/sglang/pull/28400)) and a **global-attention fix** ([PR #28604](https://github.com/sgl-project/sglang/pull/28604), since M.1 is full-attention `sliding_window: 0`) — but not yet in a tagged release. The two paths below match the **Python / Docker** toggle in the command panel: install from `main` (Python tab), or use the **Docker** image, which bundles the same build (CUDA 13, covers H200 + all Blackwell). The model ships custom config code on the Hub, so `--trust-remote-code` is required (it is included in the launch commands). ```bash Command theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate # Laguna-M.1 support is on SGLang main (PRs #28400 + #28604, plus #28649 for FP8), not yet in a # tagged release — install from main. The serving runtime is in the base dependencies, no extra needed: git clone https://github.com/sgl-project/sglang.git cd sglang uv pip install --prerelease=allow -e python ``` Then run the **Python** output of the command panel below in that environment. The **Docker** tab is simpler — `lmsysorg/sglang:latest` bundles the CUDA-13 runtime and the M.1 code. ```bash Command theme={null} # CUDA 13 — covers H200 + all Blackwell: docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + quantization to generate the launch command. Laguna-M.1 ships a single **Balanced** recipe per cell — poolside's recommended operating point, a good speed/throughput trade-off for typical multi-user serving. The 8-GPU HGX platforms (H200 / B200 / B300) use `--tp 8`; the 4-GPU Grace-Blackwell single nodes (GB200 / GB300) use `--tp 4`. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs (parsers, DP-Attention, DeepEP / EP) on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction [Laguna-M.1](https://huggingface.co/poolside/Laguna-M.1) is an open-weight, **225B-parameter** Mixture-of-Experts model (**23B activated per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software-engineering work. It is released under Apache 2.0. **Key Features:** * **Large sparse MoE**: 70-layer transformer — the first 3 layers are dense SwiGLU, the remaining 67 are sparse MoE with **256 experts, top-16 routing** (+1 shared expert) and auxiliary-loss-free load balancing. * **Global attention with output gating**: global attention across all layers, 64 Q-heads / 8 KV-heads (head dim 128), with **softplus attention output gating** (requires [PR #28400](https://github.com/sgl-project/sglang/pull/28400)). * **Long context**: 262,144 tokens, RoPE with YaRN. * **Agentic coding**: competitive on SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro, and Terminal-Bench 2.0. * **Native reasoning**: interleaved thinking between tool calls, toggled per request via `chat_template_kwargs={"enable_thinking": ...}`. **Available Quantizations:**
Quantization Hugging Face path
BF16 [`poolside/Laguna-M.1`](https://huggingface.co/poolside/Laguna-M.1)
FP8 [`poolside/Laguna-M.1-FP8`](https://huggingface.co/poolside/Laguna-M.1-FP8)
NVFP4 [`poolside/Laguna-M.1-NVFP4`](https://huggingface.co/poolside/Laguna-M.1-NVFP4)
**License:** Apache 2.0 **Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-M.1) · [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) · [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf) · [API platform](https://platform.poolside.ai). ## 2. Configuration Tips * **Trust remote code** (`--trust-remote-code`): Laguna-M.1 ships custom modeling/config code on the Hugging Face Hub, so this flag is required for the server to load the model. * **Long-context memory**: M.1 is global-attention (no sliding-window), so the 262,144-token KV cache is large. If you hit OOM at full context, lower `--mem-fraction-static` or cap `--context-length`. * **FP8**: On **Blackwell** the recipe adds `--fp8-gemm-backend triton` — the compressed-tensors block-FP8 weight scales aren't UE8M0-packed, so the default DeepGEMM path emits garbage on Blackwell (sm\_100); the Triton backend is correct (\~19% slower). Temporary workaround pending [PR #28662](https://github.com/sgl-project/sglang/pull/28662) (which fixes the scales and restores the DeepGEMM fast path). On **Hopper (H200)** FP8 uses DeepGEMM with no extra flag — pre-warm its multi-session JIT with `python3 -m sglang.compile_deep_gemm --model poolside/Laguna-M.1-FP8` to avoid paying it on each restart. * **Parsers** (`poolside_v1`): for agentic / tool-using deployments enable the **Reasoning Parser** and **Tool Call Parser** in the Playground above — they emit `--reasoning-parser poolside_v1` (thinking → `reasoning_content`) and `--tool-call-parser poolside_v1` (structured `tool_calls`). * **Thinking default**: thinking is **off by default**; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. * **Served model id**: the server registers the model under whatever you pass to `--model-path`, so a client's `model` field must match it — `poolside/Laguna-M.1` (BF16) or `poolside/Laguna-M.1-FP8` / `-NVFP4` for the quantized cells. The §3 examples use the BF16 id; swap in the id you launched. * **Recommended sampling**: poolside benchmarks M.1 at `temperature=1.0`, `top_k=20` with thinking enabled. These are per-request sampling params (not launch flags) — e.g. `temperature=1.0, extra_body={"top_k": 20}` on the OpenAI client. ## 3. Advanced Usage ### 3.1 Reasoning Launch with `--reasoning-parser poolside_v1` (or toggle **Reasoning Parser** in the **Parsers** card of the Playground above). Reasoning is **opt-in**: the Laguna chat template gates it on `enable_thinking=True` (passed via `chat_template_kwargs`) — the generic `thinking` key is ignored. The `` trace then lands in `message.reasoning_content`, separate from the final answer in `message.content` — no client-side tag stripping needed. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="poolside/Laguna-M.1", messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}], max_tokens=2048, extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) message = response.choices[0].message print("=============== Reasoning ===============") print(message.reasoning_content) print("=============== Answer ==================") print(message.content) ``` ```text Output theme={null} =============== Reasoning =============== Okay, so I need to find out what 15% of 240 is. Hmm, percentages can sometimes be tricky, but let me think. I remember that "percent" means per hundred, right? So 15% is the same as 15 per 100 or 15/100. Maybe I can convert that percentage into a decimal first? ... 15 divided by 100 is 0.15. ... Now, to find 15% of 240, I just need to multiply 240 by 0.15. ... 240 times 0.1 is 24 (10% of 240), and 240 times 0.05 is 12 (half of that), so 24 + 12 = 36. [… verifies the same result several more ways: 15/100 × 240, 240 × 15 ÷ 100, 1% × 15, and the fraction 3/20 × 240 — all give 36 …] So ... all methods are pointing to 36. I'm pretty confident that 15% of 240 is 36. =============== Answer ================== To find 15% of 240, convert the percentage to a decimal (0.15) and multiply by 240: **240 × 0.15 = 36**. **Step-by-Step Explanation:** 1. **Convert 15% to a decimal:** 15% = 15/100 = 0.15. 2. **Multiply by 240:** - Break it down: - 10% of 240 = 24 (since 240 × 0.1 = 24). - 5% of 240 = 12 (half of 24). - Add them: 24 + 12 = **36**. **Answer:** 15% of 240 is **36**. ``` Laguna-M.1's reasoning traces are long — the model explores and re-verifies an answer multiple ways. Give it a generous `max_tokens` for harder problems (reasoning regularly exceeds 3k tokens). The trace above is abbreviated; the model emits it in full. ### 3.2 Tool Calling Launch with `--tool-call-parser poolside_v1` (or toggle **Tool Call Parser** in the **Parsers** card of the Playground above). The parser converts Laguna's `` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (`enable_thinking=False`, the default). ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="poolside/Laguna-M.1", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) message = response.choices[0].message if message.tool_calls: for call in message.tool_calls: print(f"Tool: {call.function.name}") print(f"Args: {call.function.arguments}") ``` ```text Output theme={null} Tool: get_weather Args: {"location": "Beijing"} ``` ### 3.3 Prefill-Decode (PD) Disaggregation [PD disaggregation](../../../docs/advanced_features/pd_disaggregation) runs prefill and decode on **separate** SGLang servers linked by an RDMA KV-transfer fabric (mooncake or NIXL), fronted by the PD router. Laguna-M.1 is **global-attention with a standard KV cache** (no sliding window, no sparse "index" side-buffer), so its KV pages transfer with **no model-specific flags** — just the `--disaggregation-*` knobs. Both roles auto-select the same attention backend (FlashAttention-3) and page size because they share the model and flags, so the KV layout lines up for transfer. **Supported / validated topology:** * **Equal tensor parallelism** — prefill and decode run the same `--tp`. * **Single pipeline stage** — PP = 1 (the default). * **mooncake or NIXL** transfer backend over RDMA / InfiniBand. * Validated on **2 × 8×H200** (TP8 prefill + TP8 decode, BF16), one node each, over an 8× 400 Gb/s NDR InfiniBand fabric. Launch the prefill server, then the decode server — the same recipe with `--disaggregation-mode decode` and no bootstrap port. Point `--disaggregation-ib-device` at your RDMA NIC(s). ```bash Prefill server (node A) theme={null} sglang serve \ --model-path poolside/Laguna-M.1 \ --trust-remote-code \ --reasoning-parser poolside_v1 \ --tool-call-parser poolside_v1 \ --tp 8 \ --disaggregation-mode prefill \ --disaggregation-transfer-backend mooncake \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \ --host 0.0.0.0 --port 30000 \ --disaggregation-bootstrap-port 8998 ``` ```bash Decode server (node B) theme={null} sglang serve \ --model-path poolside/Laguna-M.1 \ --trust-remote-code \ --reasoning-parser poolside_v1 \ --tool-call-parser poolside_v1 \ --tp 8 \ --disaggregation-mode decode \ --disaggregation-transfer-backend mooncake \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \ --host 0.0.0.0 --port 30001 ``` Then start the PD router, pointing it at the prefill bootstrap (URL plus its `--disaggregation-bootstrap-port`) and the decode endpoint: ```bash PD router theme={null} python3 -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:30000 8998 \ --decode http://:30001 \ --policy round_robin \ --host 0.0.0.0 --port 8000 ``` Clients hit the router exactly like a single server — it splits each request across the two stages transparently: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="poolside/Laguna-M.1", messages=[{"role": "user", "content": "What is 2 + 2?"}], max_tokens=64, ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} 2 + 2 = 4 ``` **Transfer backend — mooncake (recommended).** mooncake honors `--disaggregation-ib-device` and establishes its RDMA connection at registration, so the **first request is already fast** (no cold start). It works with a single NIC or all eight; using **all 8 NICs lowers TTFT** (more aggregate bandwidth for the KV payload — the gap widens at longer context). On 8×H200 (random isl=512 / osl=256, 16 concurrent) it served ≈ **717 tok/s** output (≈ 2.2k tok/s total), mean **TTFT 244 ms**, mean **TPOT 17.7 ms**; with a single `mlx5_0` NIC, ≈ 697 tok/s and TTFT 287 ms (TPOT unchanged — decode is compute-bound). **Transfer backend — NIXL (works, with two caveats).** The NIXL path **ignores `--disaggregation-ib-device`** — that flag is mooncake-only. NIXL uses its UCX backend, whose NIC is selected by the **`UCX_NET_DEVICES`** environment variable. **Set it** (e.g. `export UCX_NET_DEVICES=mlx5_0:1`) on both servers; without it UCX cannot establish a working cross-node path and every KV transfer hangs until it hits the 300 s timeout (`Request … timed out … in KVPoll.WaitingForInput`) and returns a 500. With `UCX_NET_DEVICES` pinned, NIXL matches mooncake on quality and steady-state speed (≈ 720 tok/s, TTFT 230 ms, TPOT 17.7 ms). One difference: the **first request after launch pays a \~38 s one-time UCX connection cold-start** (a single port or all eight behave the same). Warm the path with one throwaway request after startup, or raise `SGLANG_DISAGGREGATION_WAITING_TIMEOUT` (default 300 s) so the first real request isn't dropped while UCX connects. **Validation.** PD disaggregation preserves output quality — disaggregated output matches non-disaggregated serving, and GSM8K (no-thinking, 200-question subset via the router) scored **0.945** (mooncake, 8 NICs) / **0.940** (NIXL) / **0.950** (mooncake, 1 NIC), all with 100% stop-rate and 0% errors — in line with single-node BF16 (≈ 0.93 on the full split). Logs confirm the split: the prefill node logs `Prefill batch` (CUDA graph off), the decode node logs `Decode batch` (CUDA graph on). # Laguna-S-2.1 Source: https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-S-2.1 Deploy poolside's Laguna-S-2.1 — a 118B hybrid-SWA Mixture-of-Experts model (8B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4. ## Deployment Laguna-S-2.1 uses the same `laguna` model architecture as [Laguna-XS-2.1](./Laguna-XS-2.1), which is fully supported in SGLang `main`. The model ships custom config code on the Hub, so `--trust-remote-code` is required (included in the launch commands). ```bash Command theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate git clone https://github.com/sgl-project/sglang.git cd sglang uv pip install --prerelease=allow -e python ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + quantization + strategy to generate the launch command. The two serving strategies cover the common operating points: * **Low-latency** — DFlash speculative decoding with a matched draft model. Pick for chat and interactive agents. * **High-throughput** — plain serving. Best for batch workloads, where speculation's draft + rejection overhead costs more than it saves. On the 8-GPU HGX platforms (H200 / B300) all quantizations run `--tp 8`. The 4-GPU GB300 node runs `--tp 4` throughout. NVFP4 is Blackwell-only (B300 / GB300 only). ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs (TP degree, parsers) on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction [Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) is an open-weight **118B-parameter** hybrid sliding-window-attention MoE model (**\~8B active per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering. It sits between [Laguna XS 2.1](./Laguna-XS-2.1) (33B/3B active) and Laguna M.1 (222B/23B active) in the Laguna family. **Key Features:** * **Sparse MoE**: 48 layers, 256 routed experts, top-10 routing, plus 1 shared expert. * **Hybrid attention**: 36 sliding-window layers (window 512) interleaved with 12 full-attention layers (1:3 global-to-SWA ratio); 8 KV heads, head dim 128; per-head sigmoid output gating with per-layer-type rotary scales. * **Long context**: 1,048,576 tokens. * **DFlash drafts**: matched draft models ship per quantization for low-latency serving. * **Hybrid reasoning**: `…` toggled per request via `chat_template_kwargs={"enable_thinking": …}`. **Available quantizations:**
Precision Target model Draft model
BF16 [`poolside/Laguna-S-2.1`](https://huggingface.co/poolside/Laguna-S-2.1) [`poolside/Laguna-S-2.1-DFlash`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash)
FP8 [`poolside/Laguna-S-2.1-FP8`](https://huggingface.co/poolside/Laguna-S-2.1-FP8) [`poolside/Laguna-S-2.1-DFlash-FP8`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-FP8)
NVFP4 [`poolside/Laguna-S-2.1-NVFP4`](https://huggingface.co/poolside/Laguna-S-2.1-NVFP4) [`poolside/Laguna-S-2.1-DFlash-NVFP4`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-NVFP4)
INT4 [`poolside/Laguna-S-2.1-INT4`](https://huggingface.co/poolside/Laguna-S-2.1-INT4) [`poolside/Laguna-S-2.1-DFlash-INT4`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-INT4)
The drafts are small BF16 models, each *calibrated against its quantized target* — always pair a target with its matched draft (mixing precisions degrades accept-length). **License:** [OpenMDW-1.1](https://openmdw.ai/) **Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-S-2.1) · [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf) · [API platform](https://platform.poolside.ai) ## 2. Configuration Tips **Attention backend** Leave `--attention-backend` unset for High-throughput cells — auto-select is correct (`fa3` on Hopper, `trtllm_mha` on Blackwell). With DFlash active, auto-select instead falls back to `flashinfer`, which breaks this hybrid-SWA model at `tp ≥ 4` on Blackwell (reproduced on Laguna-XS-2.1, greedy GSM8K 76% → 28%), so the Low-latency commands pin the target backend explicitly. Leave `--speculative-draft-attention-backend` unset. Other attention backend choices have not been fully validated on Laguna; keep the default. **BF16 memory on H200** BF16 on H200 leaves less headroom for CUDA-graph capture and NCCL allocations than FP8/INT4. The High-throughput BF16 command carries `--mem-fraction-static 0.80`. FP8, INT4, and all B300/GB300 cells use the default heuristic. **FP8 shared expert** `SGLANG_SHARED_EXPERT_TP1=1` is required for FP8 cells on **all hardware** — confirmed on both H200 (TP=8) and GB300 (TP=4). The FP8 checkpoint block-quantizes the shared expert (128×128 scales), which cannot TP-shard cleanly at either TP degree on S-2.1. This env var replicates the shared expert instead of sharding it. INT4 keeps the shared expert in BF16 (no flag needed); BF16 is unquantized. Note: this differs from Laguna-XS-2.1 where TP=4 does not require the flag — the constraint is architecture-specific. **FP8 and NVFP4 DFlash drafts** Fixed upstream on 2026-07-21: all DFlash draft configs now use a flat top-level `rope_theta` (the `rope_parameters` block was removed). If a server crashes at draft-model load with `KeyError: 'rope_theta'`, you are serving a draft checkpoint cached before 2026-07-21 — re-download it (e.g. `hf download poolside/Laguna-S-2.1-DFlash-FP8`) to pick up the corrected config. **DFlash memory** Low-latency cells carry `--mem-fraction-static 0.7` (sufficient even for BF16 on H200). Dense cells use the default heuristic (except BF16 on H200 — see above). **BF16 reasoning length** BF16 reasons approximately 2× longer than FP8/INT4 on AIME25 (median 34.8 k vs 16.9 k tokens), consistently truncating at `max_tokens=64000`. FP8/INT4 truncate at ≈ 2%. For a valid BF16 AIME25 score, serve with `max_tokens ≥ 131072` (the model supports a 1 M context window). **Chat template** On transformers ≥ 5.10 the standalone `chat_template.jinja` auto-loads — no flag needed (the server logs `Auto-detected template features: reasoning_parser=poolside_v1, ...`). On older transformers (≤ \~5.8) pass `--chat-template /chat_template.jinja` explicitly. **Thinking** Off by default; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. The template gates on `enable_thinking` — the generic `thinking` key is ignored. **Served model id** The server registers the model under whatever you pass to `--model-path`; a client's `model` field must match it (`poolside/Laguna-S-2.1`, or the `-FP8` / `-NVFP4` / `-INT4` id). ## 3. Advanced Usage ### 3.1 DFlash Speculative Decoding DFlash is a block-wise speculative decoder: the draft proposes a block of tokens and the target verifies the whole block in one forward pass — output quality is the target's by construction. The speedup lever is **accept-length**, the number of draft tokens surviving verification per target step. Best for interactive / few-stream serving. Under batch-saturated load prefer High-throughput: once the GPU is compute-bound, draft + rejected-token overhead costs aggregate throughput. The generated commands always pair the draft calibrated for the selected target precision. ### 3.2 Reasoning Launch with `--reasoning-parser poolside_v1` (baked into every generated command). Reasoning is opt-in via `enable_thinking=True`; the `` trace lands in `message.reasoning_content`, separate from the final answer in `message.content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="poolside/Laguna-S-2.1", messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}], max_tokens=4096, extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) message = response.choices[0].message print("=============== Reasoning ===============") print(message.reasoning_content) print("=============== Answer ==================") print(message.content) ``` Give generous `max_tokens` when thinking is enabled — hard problems regularly reason for thousands of tokens. Keep thinking off for short-form tasks. ### 3.3 Tool Calling Launch with `--tool-call-parser poolside_v1` (baked into every generated command). The parser converts Laguna's `` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (the default). ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="poolside/Laguna-S-2.1", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) message = response.choices[0].message if message.tool_calls: for call in message.tool_calls: print(f"Tool: {call.function.name}") print(f"Args: {call.function.arguments}") ``` # Laguna-XS-2.1 Source: https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-XS-2.1 Deploy poolside's Laguna-XS-2.1 — a 33B hybrid-SWA Mixture-of-Experts model (3B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4. ## Deployment Laguna-XS-2.1 support is fully merged to SGLang `main` ([PR #29446](https://github.com/sgl-project/sglang/pull/29446): DFlash speculative decoding + shared-expert fix; [PR #29761](https://github.com/sgl-project/sglang/pull/29761): INT4 loader fix). Any build at or past their merge covers every cell below. The model ships custom config code on the Hub, so `--trust-remote-code` is required (included in the launch commands). ```bash Command theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate git clone https://github.com/sgl-project/sglang.git cd sglang uv pip install --prerelease=allow -e python ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + quantization + strategy to generate the launch command. The two serving strategies cover the common operating points: * **Low-latency** — DFlash speculative decoding with a matched draft model. Pick for chat and interactive agents. * **High-throughput** — plain serving. Best for batch workloads, where speculation's draft + rejection overhead costs more than it saves. On the 8-GPU HGX platforms (H200 / B300), BF16 and NVFP4 run plain `--tp 8`; FP8 and INT4 run `--tp 8 --ep-size 8` because their quantization scales cannot shard the MoE 8-way (see [Configuration Tips](#2-configuration-tips)). The 4-GPU GB300 node runs plain `--tp 4` throughout. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs (TP degree, parsers) on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction [Laguna-XS-2.1](https://huggingface.co/poolside/Laguna-XS-2.1) is an open-weight **33B-parameter** hybrid sliding-window-attention MoE model (**\~3B active per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering — the extra-small sibling of [Laguna-M.1](./Laguna-M.1). **Key Features:** * **Sparse MoE**: 40 layers, 256 routed experts, top-8 routing. * **Hybrid attention**: 30 sliding-window layers (window 512) interleaved with 10 full-attention layers; 48 Q / 8 KV heads. * **Long context**: 262,144 tokens (RoPE + YaRN on the full-attention layers). * **DFlash drafts**: matched draft models (5-layer, \~0.9 GB) ship per quantization for low-latency serving. * **Hybrid reasoning**: `…` toggled per request via `chat_template_kwargs={"enable_thinking": …}`. **Available quantizations:**
Precision Target model Draft model
BF16 [`poolside/Laguna-XS-2.1`](https://huggingface.co/poolside/Laguna-XS-2.1) [`poolside/Laguna-XS-2.1-DFlash`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash)
FP8 [`poolside/Laguna-XS-2.1-FP8`](https://huggingface.co/poolside/Laguna-XS-2.1-FP8) [`poolside/Laguna-XS-2.1-DFlash-FP8`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-FP8)
NVFP4 [`poolside/Laguna-XS-2.1-NVFP4`](https://huggingface.co/poolside/Laguna-XS-2.1-NVFP4) [`poolside/Laguna-XS-2.1-DFlash-NVFP4`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-NVFP4)
INT4 [`poolside/Laguna-XS-2.1-INT4`](https://huggingface.co/poolside/Laguna-XS-2.1-INT4) [`poolside/Laguna-XS-2.1-DFlash-INT4`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-INT4)
The drafts themselves are small bf16 models, each *calibrated against its quantized target* — always pair a target with its matched draft (mixing precisions degrades accept-length). **License:** Apache 2.0 **Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-XS-2.1) · [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) · [API platform](https://platform.poolside.ai). ## 2. Configuration Tips **Attention backend** Leave `--attention-backend` unset for High-throughput cells — auto-select is correct (`fa3` on Hopper, `trtllm_mha` on Blackwell). With DFlash active, auto-select instead falls back to `flashinfer`, which breaks this hybrid-SWA model at `tp ≥ 4` on Blackwell (greedy GSM8K 76% → 28%), so the Low-latency commands pin the target backend explicitly. Leave `--speculative-draft-attention-backend` unset. Never use `triton` attention with Laguna (GSM8K 13%). **Quantized checkpoints cap plain TP at 4** `moe_intermediate_size=512` with FP8 block `[128,128]` / INT4 `group_size=128` scales cannot shard 8-way (512/8 = 64 \< 128 granularity): FP8 fails at weight creation, INT4 crashes in the Marlin kernel, on any hardware. The generated 8-GPU FP8/INT4 commands therefore use `--tp 8 --ep-size 8` — expert parallelism keeps whole experts per rank, using all 8 GPUs on one instance. FP8 additionally needs `SGLANG_SHARED_EXPERT_TP1=1` (its shared expert is also block-quantized; INT4's stays bf16). Alternatives: plain `--tp 4`, or `--tp 4 --dp-size 2`. Accuracy is parallelism-independent within eval noise (verified tp1 ≡ tp4 on GB300 and tp4 ≡ tp8+ep8 on H200). **DFlash memory** Low-latency cells carry `--mem-fraction-static 0.7`: the default fraction OOMs in the draft vocab all-gather at `tp 4` on GB300. Dense cells use the default heuristic. **INT4 is mixed-precision** The INT4 checkpoint quantizes MoE layers in mixed 4-bit / 8-bit config groups. Builds older than [PR #29761](https://github.com/sgl-project/sglang/pull/29761) crash at load with `KeyError: 'Linear'`. **Chat template** On transformers ≥ 5.10 the standalone `chat_template.jinja` auto-loads — no flag needed (the server logs `Auto-detected template features: reasoning_parser=poolside_v1, ...`). On older transformers (≤ \~5.8) the `{% include %}` stub in `tokenizer_config.json` cannot resolve and the server silently falls back to a generic template — pass `--chat-template /chat_template.jinja` explicitly there. **Thinking** Off by default; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. The template gates on `enable_thinking` — the generic `thinking` key is ignored. **Served model id** The server registers the model under whatever you pass to `--model-path`; a client's `model` field must match it (`poolside/Laguna-XS-2.1`, or the `-FP8` / `-NVFP4` / `-INT4` id). ## 3. Advanced Usage ### 3.1 DFlash Speculative Decoding DFlash is a block-wise speculative decoder: the 5-layer draft proposes a block of tokens and the target verifies the whole block in one forward pass, so only target-approved tokens are emitted — output quality is the target's by construction (GSM8K matches dense within noise on every quantization). The speedup lever is **accept-length**, the number of draft tokens surviving verification per target step: * Measured \~6 tokens/step at `tp 1`, \~4 at `tp 4` (greedy GSM8K, matched-precision pairs; \~3 under mixed reasoning-heavy traffic; FP8 reached 6.75 at `tp 8 + ep 8` on H200) — versus 1 token/step dense. * Best for interactive / few-stream serving. Under batch-saturated load prefer High-throughput: once the GPU is compute-bound, draft + rejected-token overhead costs aggregate throughput. * The generated commands always pair the draft calibrated for the selected target precision. ### 3.2 Reasoning Launch with `--reasoning-parser poolside_v1` (baked into every generated command). Reasoning is opt-in via `enable_thinking=True`; the `` trace lands in `message.reasoning_content`, separate from the final answer in `message.content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="poolside/Laguna-XS-2.1", messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}], max_tokens=2048, extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) message = response.choices[0].message print("=============== Reasoning ===============") print(message.reasoning_content) print("=============== Answer ==================") print(message.content) ``` XS-2.1 is an extra-small model — give it generous `max_tokens` when thinking is enabled (hard problems regularly reason for thousands of tokens), and keep thinking off for short-form tasks. ### 3.3 Tool Calling Launch with `--tool-call-parser poolside_v1` (baked into every generated command). The parser converts Laguna's `` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (the default). ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="poolside/Laguna-XS-2.1", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) message = response.choices[0].message if message.tool_calls: for call in message.tool_calls: print(f"Tool: {call.function.name}") print(f"Args: {call.function.arguments}") ``` # Laguna-XS.2 Source: https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-XS.2 ## 1. Model Introduction [Laguna-XS.2](https://huggingface.co/poolside/Laguna-XS.2) is an open-source hybrid sliding-window-attention MoE model from [Poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering work. **Key Features:** * **MoE**: 33.4B total parameters, 3.0B active per token, 256 routed experts (top-8) plus 1 shared. * **Long context**: 131,072 tokens. * **Agentic coding**: Tuned for tool-using software engineering agents and long-horizon execution. * **Hybrid reasoning**: `...` segments toggled per request via `chat_template_kwargs={"enable_thinking": ...}`. **Available Quantizations:**
Variant Hugging Face path
BF16 [`poolside/Laguna-XS.2`](https://huggingface.co/poolside/Laguna-XS.2)
FP8 [`poolside/Laguna-XS.2-FP8`](https://huggingface.co/poolside/Laguna-XS.2-FP8)
NVFP4 [`poolside/Laguna-XS.2-NVFP4`](https://huggingface.co/poolside/Laguna-XS.2-NVFP4)
**License:** Apache 2.0 For details, see the [Hugging Face model card](https://huggingface.co/poolside/Laguna-XS.2) and the [Laguna deeper-dive blog post](https://poolside.ai/blog/laguna-a-deeper-dive). ## 2. SGLang Installation Laguna-XS.2 support is on `main` but not yet in a tagged release; install from the SGLang nightly wheel index, or pull a pre-built Docker image: ```bash Command theme={null} # Install SGLang via pip (CUDA 13) — requires Python 3.10 (nightly wheels are cp310 only) python3 -m pip install --upgrade pip python3 -m pip install --extra-index-url https://docs.sglang.ai/whl/cu130 \ "sglang[all]==0.5.12.dev20260509+g096ad02b0" # CUDA 12: swap to the cu129 index python3 -m pip install --extra-index-url https://docs.sglang.ai/whl/cu129 \ "sglang[all]==0.5.12.dev20260509+g096ad02b0" # Or use Docker (multi-arch amd64/arm64; CUDA 13, H200 / B200) docker pull lmsysorg/sglang:latest ``` For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to generate a launch command for your hardware. ### 3.2 Configuration Tips * **Trust remote code** (`--trust-remote-code`): Laguna-XS.2 ships custom modeling/config code on the Hugging Face Hub, so this flag is required for the server to load the model. * **Quantization**: NVFP4 requires Blackwell (B200 / B300); BF16 and FP8 run on either H200 or B200. FP8's first launch triggers a multi-session DeepGEMM JIT pre-compile (\~10-20 min); pre-warm with `python3 -m sglang.compile_deep_gemm --model poolside/Laguna-XS.2-FP8` to avoid that cost on every restart. * **Reasoning parser** (`--reasoning-parser poolside_v1`): Splits `...` segments into `reasoning_content` so `content` holds only the final answer. Disable only if you want the raw `` tags in `content`. * **Tool call parser** (`--tool-call-parser poolside_v1`): Required for OpenAI-compatible tool-call streaming. Disable only for chat-only deployments. * **DP attention**: For higher-throughput deployments, enable the DP-Attention toggle — it emits `--attn-dp-size ` with `--attn-dp-size` matching `--tp` (tune independently if needed). * **Thinking default**: Thinking is **off by default** at the model level. Opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. ## 4. Model Invocation The samples below assume the server is reachable at `http://localhost:30000/v1`. ### 4.1 Basic Chat ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="poolside/Laguna-XS.2", messages=[ {"role": "user", "content": "What is the difference between TCP and UDP?"} ], max_tokens=1024, ) print(resp.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) are two core protocols of the Internet Protocol (IP) suite, both used for network communication but with key differences: ## Connection Handling - **TCP**: Connection-oriented protocol that establishes a connection before data transfer (like a phone call) - **UDP**: Connectionless protocol that sends data without establishing a connection (like sending a letter) ## Reliability - **TCP**: Guaranteed delivery with error checking, retransmission of lost packets, and flow control - **UDP**: No guarantee of delivery; packets may be lost, duplicated, or arrive out of order ## Speed & Overhead - **TCP**: Slower due to connection setup, acknowledgment overhead, and error correction mechanisms - **UDP**: Faster with minimal overhead since it doesn't wait for acknowledgments or retransmit lost data ## Use Cases - **TCP**: Web browsing (HTTP/HTTPS), email (SMTP), file transfers (FTP), database connections - **UDP**: Video streaming, online gaming, VoIP calls, DNS queries, live broadcasts In essence, TCP prioritizes reliability over speed, while UDP prioritizes speed over reliability. ``` ### 4.2 Reasoning (Thinking Mode) Laguna-XS.2 emits reasoning between `...` tags. The `--reasoning-parser poolside_v1` flag separates the thinking text into `reasoning_content` so `content` holds only the final answer. Thinking is opt-in per request: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="poolside/Laguna-XS.2", messages=[ {"role": "user", "content": "If a train travels at 60 km/h for 2.5 hours, how far does it go?"} ], max_tokens=4096, extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) print("====== Reasoning Content ======") print(resp.choices[0].message.reasoning_content) print("====== Answer ======") print(resp.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} ====== Reasoning Content ====== The user is asking a straightforward math problem about distance, speed, and time. I need to calculate the distance using the formula: Distance = Speed × Time Given: - Speed = 60 km/h - Time = 2.5 hours So the calculation would be: Distance = 60 × 2.5 = 150 km This is a simple multiplication problem. I should provide a clear, direct answer and maybe explain the calculation briefly. ====== Answer ====== To find the distance, use the formula: Distance = Speed × Time Distance = 60 km/h × 2.5 h = 150 km The train travels **150 kilometers**. ``` To disable thinking, omit `extra_body` (off by default) or pass `chat_template_kwargs={"enable_thinking": False}` explicitly. ### 4.3 Tool Calling ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, }, "required": ["location"], }, }, } ] resp = client.chat.completions.create( model="poolside/Laguna-XS.2", messages=[{"role": "user", "content": "What's the weather in Tokyo?"}], tools=tools, ) msg = resp.choices[0].message print("====== Reasoning Content ======") print(msg.reasoning_content) print("====== Content ======") print(msg.content) print("====== Tool Calls ======") for tc in msg.tool_calls or []: print(f" Function: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` **Output Example:** ```text Output theme={null} ====== Reasoning Content ====== None ====== Content ====== I'll check the current weather in Tokyo for you. ====== Tool Calls ====== Function: get_weather Arguments: {"location": "Tokyo"} ``` `reasoning_content` is `None` because thinking is off by default; `content` carries the brief assistant message that precedes the tool call. Add `extra_body={"chat_template_kwargs": {"enable_thinking": True}}` if you want interleaved reasoning before the tool call. ## 5. Benchmark ### 5.1 Accuracy Benchmark **Test Environment:** * Hardware: NVIDIA H200 (4×H200) * Model: `poolside/Laguna-XS.2` (BF16) * Tensor Parallelism: 4 * SGLang Version: `0.5.12.dev20260509+g096ad02b0` (nightly wheel containing the #24204 merge commit; same code path as the original PR runs) * Reasoning Parser: `poolside_v1` * Tool Call Parser: `poolside_v1` * Sampling: `temperature=0.6`, `max_tokens=16384`, `chat_template_kwargs={"enable_thinking": true}`, `n_repeats=1` * Grader: NeMo-Skills `math_verify` (math) and `eval_mcq` (multichoice) **Results (from [PR #24204](https://github.com/sgl-project/sglang/pull/24204)):** | Eval | Accuracy | | - | -: | | GPQA Diamond | 0.5556 | | AIME 25 | 0.5667 | | MMLU | 0.836 | | SWE-Bench Verified | 0.6540 | ### 5.2 Speed Benchmark **Test Environment:** * Hardware: NVIDIA H200 (1×H200 for TP=1, 4×H200 for TP=4) * Model: `poolside/Laguna-XS.2` (BF16) * SGLang Version: `0.5.12.dev20260509+g096ad02b0` (nightly wheel containing the #24204 merge commit; same code path as the original PR runs) * Workload: `sglang.bench_serving --backend sglang --dataset-name random` (defaults: `--random-input-len 1024 --random-output-len 1024 --random-range-ratio 0.0`) * Server flags identical to the accuracy runs above. #### 5.2.1 Latency Benchmark (10 prompts, concurrency = 1) ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 10 --max-concurrency 1 ``` | Metric | TP=1 | TP=4 | | - | -: | -: | | Successful requests | 10 | 10 | | Output token throughput (tok/s) | 193.10 | 238.88 | | Total token throughput (tok/s) | 471.82 | 583.68 | | Mean TTFT (ms) | 35.32 | 24.17 | | Mean TPOT (ms) | 5.10 | 4.13 | | Median ITL (ms) | 5.14 | 4.14 | #### 5.2.2 Throughput Benchmark (1000 prompts, concurrency = 100) ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 1000 --max-concurrency 100 ``` | Metric | TP=1 | TP=4 | | - | -: | -: | | Successful requests | 1000 | 1000 | | Request throughput (req/s) | 7.32 | 14.61 | | Output token throughput (tok/s) | 3739.30 | 7465.18 | | Peak output token throughput (tok/s) | 4718.00 | 10133.00 | | Total token throughput (tok/s) | 7485.82 | 14944.81 | | Mean TTFT (ms) | 115.17 | 68.36 | | Mean TPOT (ms) | 25.51 | 12.71 | | Median ITL (ms) | 21.31 | 10.64 | TP=4 delivers roughly 2.0× total-token throughput and \~1.7× lower mean TTFT compared to TP=1 on the `cc=100` random workload. # Qwen2.5-VL Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen2.5-VL ## 1. Model Introduction **[Qwen2.5-VL](https://huggingface.co/collections/Qwen/qwen25-vl)** is a vision-language model series from the Qwen team, offering significant improvements over its predecessor in understanding, reasoning, and multi-modal processing. **Key Features:** * **Understand things visually**: Proficient in recognizing common objects such as flowers, birds, fish, and insects, and it is highly capable of analyzing texts, charts, icons, graphics, and layouts within images. * **More Agentic**: Play as a visual agent that can reason and dynamically direct tools, which is capable of computer use and phone use. * **Understanding long videos and capturing events**: Supports comprehending videos of over 1 hour, and this time it has a new ability of capturing event by pinpointing the relevant video segments. * **Capable of visual localization in different formats**: Accurately localize objects in an image by generating bounding boxes or points, and it can provide stable JSON outputs for coordinates and attributes. * **Generating structured outputs**: Supports structured outputs of the contents, benefiting usages in finance, commerce, etc for data like scans of invoices, forms, tables, etc. * **Dynamic Resolution and Frame Rate Training for Video Understanding**: Extend dynamic resolution to the temporal dimension by adopting dynamic FPS sampling, enabling the model to comprehend videos at various sampling rates. Accordingly, we update mRoPE in the time dimension with IDs and absolute time alignment, enabling the model to learn temporal sequence and speed, and ultimately acquire the ability to pinpoint specific moments. * **Multiple Sizes**: Available in 3B, 7B, 32B, and 72B variants to suit different deployment needs. * **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified). For more details, please refer to the [official Qwen2.5-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for AMD MI300X, MI325X and MI355X as well as Intel Xeon CPU hardware platforms and different use cases. ### 3.1 Basic Configuration The Qwen2.5-VL series offers models in various sizes. The following configurations have been verified on AMD MI300X, MI325X and MI355X GPUs as well as Intel Xeon CPUs. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model size. ### 3.2 Configuration Tips * **Memory Management**: For the 72B model on MI300X/MI325X/MI355X, we have verified successful deployment with `--context-length 128000`. Smaller context lengths can be used to reduce memory usage if needed. * **Multi-GPU Deployment**: Use Tensor Parallelism (`--tp`) to scale across multiple GPUs. For example, use `--tp 8` for the 72B model and `--tp 2` for the 32B model on MI300X/MI325X/MI355X. * **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Multi-Modal Inputs Qwen2.5-VL supports image inputs. Here's a basic example with single image input: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Read all the text in the image." } ] } ] start = time.time() response = client.chat.completions.create( model="Qwen/Qwen2.5-VL-7B-Instruct", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 2.31s Generated text: Auntie Anne's CINNAMON SUGAR 1 x 17,000 SUB TOTAL 17,000 GRAND TOTAL 17,000 CASH IDR 20,000 CHANGE DUE 3,000 ``` **Multi-Image Input Example:** Qwen2.5-VL can process multiple images in a single request for comparison or analysis: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg" } }, { "type": "image_url", "image_url": { "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg" } }, { "type": "text", "text": "Compare these two images and describe the differences in 100 words or less." } ] } ] start = time.time() response = client.chat.completions.create( model="Qwen/Qwen2.5-VL-7B-Instruct", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 13.79s Generated text: The first image shows a single red taxi driving on a street with a few other taxis in the background. The second image shows a large number of taxis parked in a lot, with some appearing to be in various states of repair. The first image has a single taxi with a visible license plate, while the second image has multiple taxis with different license plates. The first image has a clear view of the street and surrounding area, while the second image is taken from an elevated perspective, showing a wider view of the parking lot and the surrounding area. ``` **Note:** * You can also provide local file paths using `file://` protocol. * For larger images, you may need more memory, adjust `--mem-fraction-static` accordingly. ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X GPU (8x) * Model: Qwen2.5-VL-72B-Instruct * Tensor Parallelism: 8 * SGLang Version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen2.5-VL-72B-Instruct \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen2.5-VL-72B-Instruct \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen2.5-VL-72B-Instruct \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 37.99 Total input tokens: 24781 Total input text tokens: 821 Total input vision tokens: 23960 Total generated tokens: 4220 Total generated tokens (retokenized): 2365 Request throughput (req/s): 0.26 Input token throughput (tok/s): 652.26 Output token throughput (tok/s): 111.07 Peak output token throughput (tok/s): 128.00 Peak concurrent requests: 2 Total token throughput (tok/s): 763.34 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3797.61 Median E2E Latency (ms): 3140.90 P90 E2E Latency (ms): 6545.54 P99 E2E Latency (ms): 7939.56 ---------------Time to First Token---------------- Mean TTFT (ms): 504.45 Median TTFT (ms): 510.93 P99 TTFT (ms): 521.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.82 Median TPOT (ms): 7.82 P99 TPOT (ms): 7.84 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.07 Median ITL (ms): 7.90 P95 ITL (ms): 15.79 P99 ITL (ms): 15.93 Max ITL (ms): 23.60 ================================================== ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen2.5-VL-72B-Instruct \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 454.68 Total input tokens: 2481865 Total input text tokens: 85865 Total input vision tokens: 2396000 Total generated tokens: 510855 Total generated tokens (retokenized): 296466 Request throughput (req/s): 2.20 Input token throughput (tok/s): 5458.50 Output token throughput (tok/s): 1123.55 Peak output token throughput (tok/s): 5004.00 Peak concurrent requests: 106 Total token throughput (tok/s): 6582.05 Concurrency: 98.63 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 44844.92 Median E2E Latency (ms): 42866.15 P90 E2E Latency (ms): 82798.20 P99 E2E Latency (ms): 106306.30 ---------------Time to First Token---------------- Mean TTFT (ms): 4507.79 Median TTFT (ms): 1180.83 P99 TTFT (ms): 39975.22 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 80.26 Median TPOT (ms): 82.38 P99 TPOT (ms): 152.89 ---------------Inter-Token Latency---------------- Mean ITL (ms): 100.66 Median ITL (ms): 13.26 P95 ITL (ms): 428.45 P99 ITL (ms): 1393.35 Max ITL (ms): 31943.26 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 MMMU Benchmark You can evaluate the model's accuracy using the MMMU dataset: * Benchmark Command: ```shell Command theme={null} python3 benchmark/mmmu/bench_sglang.py \ --port 30000 \ --concurrency 64 ``` ```text Output theme={null} Benchmark time: 97.75084622902796 answers saved to: ./answer_sglang.json Evaluating... answers saved to: ./answer_sglang.json {'Accounting': {'acc': 0.633, 'num': 30}, 'Agriculture': {'acc': 0.5, 'num': 30}, 'Architecture_and_Engineering': {'acc': 0.367, 'num': 30}, 'Art': {'acc': 0.767, 'num': 30}, 'Art_Theory': {'acc': 0.9, 'num': 30}, 'Basic_Medical_Science': {'acc': 0.7, 'num': 30}, 'Biology': {'acc': 0.467, 'num': 30}, 'Chemistry': {'acc': 0.433, 'num': 30}, 'Clinical_Medicine': {'acc': 0.733, 'num': 30}, 'Computer_Science': {'acc': 0.567, 'num': 30}, 'Design': {'acc': 0.833, 'num': 30}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.467, 'num': 30}, 'Economics': {'acc': 0.767, 'num': 30}, 'Electronics': {'acc': 0.433, 'num': 30}, 'Energy_and_Power': {'acc': 0.467, 'num': 30}, 'Finance': {'acc': 0.533, 'num': 30}, 'Geography': {'acc': 0.633, 'num': 30}, 'History': {'acc': 0.7, 'num': 30}, 'Literature': {'acc': 0.867, 'num': 30}, 'Manage': {'acc': 0.633, 'num': 30}, 'Marketing': {'acc': 0.733, 'num': 30}, 'Materials': {'acc': 0.333, 'num': 30}, 'Math': {'acc': 0.533, 'num': 30}, 'Mechanical_Engineering': {'acc': 0.433, 'num': 30}, 'Music': {'acc': 0.367, 'num': 30}, 'Overall': {'acc': 0.62, 'num': 900}, 'Overall-Art and Design': {'acc': 0.717, 'num': 120}, 'Overall-Business': {'acc': 0.66, 'num': 150}, 'Overall-Health and Medicine': {'acc': 0.693, 'num': 150}, 'Overall-Humanities and Social Science': {'acc': 0.775, 'num': 120}, 'Overall-Science': {'acc': 0.553, 'num': 150}, 'Overall-Tech and Engineering': {'acc': 0.443, 'num': 210}, 'Pharmacy': {'acc': 0.833, 'num': 30}, 'Physics': {'acc': 0.7, 'num': 30}, 'Psychology': {'acc': 0.767, 'num': 30}, 'Public_Health': {'acc': 0.733, 'num': 30}, 'Sociology': {'acc': 0.767, 'num': 30}} eval out saved to ./val_sglang.json Overall accuracy: 0.62 ``` # Qwen3 Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3 ## 1. Model Introduction [Qwen3 series](https://github.com/QwenLM/Qwen3) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications. This generation delivers comprehensive upgrades across the board: * **Stronger general intelligence**: Significant improvements in instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage. * **Broader multilingual knowledge**: Substantial gains in long-tail knowledge coverage across multiple languages. * **More helpful & aligned responses**: Markedly better alignment with user preferences in subjective and open-ended tasks, enabling higher-quality, more useful text generation. * **Extended context length**: Enhanced capabilities in understanding and reasoning over 256K-token long contexts. * **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance. * **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions. For more details, please refer to the [official Qwen3 GitHub Repository](https://github.com/QwenLM/Qwen3). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The Qwen3 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities. ### 3.2 Configuration Tips * **Memory Management:** Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K. * **Expert Parallelism:** SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details. * **Kernel Tuning:** For MoE Triton kernel tuning on your specific hardware, refer to [fused\_moe\_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton). * **Qwen3-4B and Qwen3-8B on H200:** For concurrent requests with long prompts, benchmark `--enable-mixed-chunk` to mix decode requests into chunked-prefill batches. This can reduce median time to first token (TTFT), but may increase time per output token (TPOT) and end-to-end tail latency. Compare these metrics and throughput on your workload before enabling it. * **Speculative Decoding:** Using Speculative Decoding for latency-sensitive scenarios. * `--speculative-algorithm EAGLE3`: Speculative decoding algorithm * `--speculative-num-steps 3`: Number of speculative verification rounds * `--speculative-eagle-topk 1`: Top-k sampling for draft tokens * `--speculative-num-draft-tokens 4`: Number of draft tokens per step * `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan`](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan). * **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser Qwen3-235B-A22B supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-235B-A22B-Thinking-2507 \ --reasoning-parser qwen3 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="Qwen/Qwen3-235B-A22B-Thinking-2507", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= Okay, so I need to figure out what 15% of 240 is. Hmm, percentages can sometimes trip me up, but I think I remember some basics. Let me start by recalling that "percent" means "per hundred," so 15% is the same as 15 per 100, or 15/100. So, maybe I can convert 15% into a decimal first? Yeah, I think that's a common method. ... So conclusion: The answer is 36. =============== Content ================= To determine what 15% of 240 is, we can follow a systematic approach that involves converting the percentage to a decimal and then performing multiplication. Here's a step-by-step breakdown of the solution: .... ### Final Answer: $$ \boxed{36} $$ Thus, 15% of 240 is **36**. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.3 Tool Calling Qwen3 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-235B-A22B-Thinking-2507 \ --reasoning-parser qwen3 \ --tool-call-parser qwen25 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="Qwen/Qwen3-235B-A22B-Thinking-2507", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= Okay, the user is asking for the weather in Beijing. Let me check the tools available. There's a function called get_weather that takes location and unit parameters. The location is required, so I need to specify Beijing as the location. The unit is optional and can be either celsius or fahrenheit. Since the user didn't specify the unit, maybe I should default to a common one. In China, they usually use celsius, so I'll set unit to celsius. I'll call the get_weather function with location: Beijing and unit: celsius. That should get the current weather for them. =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="Qwen/Qwen3-235B-A22B-Thinking-2507", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The current weather in Beijing is **22°C** and **sunny**. A perfect day to enjoy outdoor activities! 🌞" ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (8x) * Model: Qwen3-235B-A22B-Instruct-2507 * Tensor Parallelism: 8 * sglang version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. #### 5.1.1 Standard Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --tp 8 ``` ##### 5.1.1.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 43.56 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4210 Total generated tokens (retokenized): 4206 Request throughput (req/s): 0.23 Input token throughput (tok/s): 140.07 Output token throughput (tok/s): 96.65 Peak output token throughput (tok/s): 100.00 Peak concurrent requests: 2 Total token throughput (tok/s): 236.72 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4353.63 Median E2E Latency (ms): 3475.79 ---------------Time to First Token---------------- Mean TTFT (ms): 99.03 Median TTFT (ms): 92.18 P99 TTFT (ms): 166.05 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.12 Median TPOT (ms): 10.12 P99 TPOT (ms): 10.15 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.13 Median ITL (ms): 10.12 P95 ITL (ms): 10.49 P99 ITL (ms): 10.70 Max ITL (ms): 13.45 ================================================== ``` ##### 5.1.1.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 48.95 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40725 Total generated tokens (retokenized): 40716 Request throughput (req/s): 1.63 Input token throughput (tok/s): 810.44 Output token throughput (tok/s): 832.04 Peak output token throughput (tok/s): 1151.00 Peak concurrent requests: 21 Total token throughput (tok/s): 1642.48 Concurrency: 13.61 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8326.72 Median E2E Latency (ms): 8827.86 ---------------Time to First Token---------------- Mean TTFT (ms): 215.70 Median TTFT (ms): 88.82 P99 TTFT (ms): 727.08 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 16.36 Median TPOT (ms): 16.12 P99 TPOT (ms): 24.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.96 Median ITL (ms): 14.52 P95 ITL (ms): 16.04 P99 ITL (ms): 67.69 Max ITL (ms): 457.52 ================================================== ``` ##### 5.1.1.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 92.07 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252162 Total generated tokens (retokenized): 251124 Request throughput (req/s): 5.43 Input token throughput (tok/s): 2713.46 Output token throughput (tok/s): 2738.78 Peak output token throughput (tok/s): 4400.00 Peak concurrent requests: 110 Total token throughput (tok/s): 5452.24 Concurrency: 90.50 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16665.09 Median E2E Latency (ms): 16060.10 ---------------Time to First Token---------------- Mean TTFT (ms): 260.55 Median TTFT (ms): 122.68 P99 TTFT (ms): 863.11 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 32.94 Median TPOT (ms): 34.04 P99 TPOT (ms): 41.19 ---------------Inter-Token Latency---------------- Mean ITL (ms): 32.59 Median ITL (ms): 23.54 P95 ITL (ms): 69.79 P99 ITL (ms): 119.09 Max ITL (ms): 577.70 ================================================== ``` #### 5.1.2 Reasoning Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --tp 8 ``` ##### 5.1.2.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 457.45 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 44452 Total generated tokens (retokenized): 44059 Request throughput (req/s): 0.02 Input token throughput (tok/s): 13.34 Output token throughput (tok/s): 97.17 Peak output token throughput (tok/s): 100.00 Peak concurrent requests: 2 Total token throughput (tok/s): 110.51 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 45742.42 Median E2E Latency (ms): 49266.87 ---------------Time to First Token---------------- Mean TTFT (ms): 110.60 Median TTFT (ms): 109.36 P99 TTFT (ms): 167.43 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.23 Median TPOT (ms): 10.24 P99 TPOT (ms): 10.32 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.27 Median ITL (ms): 10.26 P95 ITL (ms): 10.71 P99 ITL (ms): 10.97 Max ITL (ms): 15.79 ================================================== ``` ##### 5.1.2.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 340.17 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 318226 Total generated tokens (retokenized): 318104 Request throughput (req/s): 0.24 Input token throughput (tok/s): 116.61 Output token throughput (tok/s): 935.49 Peak output token throughput (tok/s): 1120.00 Peak concurrent requests: 19 Total token throughput (tok/s): 1052.10 Concurrency: 13.85 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 58885.30 Median E2E Latency (ms): 59238.70 ---------------Time to First Token---------------- Mean TTFT (ms): 169.71 Median TTFT (ms): 101.61 P99 TTFT (ms): 455.71 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.82 Median TPOT (ms): 14.91 P99 TPOT (ms): 15.20 ---------------Inter-Token Latency---------------- Mean ITL (ms): 14.76 Median ITL (ms): 14.63 P95 ITL (ms): 15.46 P99 ITL (ms): 16.62 Max ITL (ms): 104.94 ================================================== ``` ##### 5.1.2.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 544.83 Total input tokens: 158939 Total input text tokens: 158939 Total input vision tokens: 0 Total generated tokens: 1300705 Total generated tokens (retokenized): 1293015 Request throughput (req/s): 0.59 Input token throughput (tok/s): 291.72 Output token throughput (tok/s): 2387.34 Peak output token throughput (tok/s): 3008.00 Peak concurrent requests: 68 Total token throughput (tok/s): 2679.06 Concurrency: 56.35 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 95937.70 Median E2E Latency (ms): 99362.32 ---------------Time to First Token---------------- Mean TTFT (ms): 265.03 Median TTFT (ms): 129.11 P99 TTFT (ms): 823.85 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 23.66 Median TPOT (ms): 24.07 P99 TPOT (ms): 24.97 ---------------Inter-Token Latency---------------- Mean ITL (ms): 23.54 Median ITL (ms): 23.07 P95 ITL (ms): 25.92 P99 ITL (ms): 63.87 Max ITL (ms): 408.30 ================================================== ``` #### 5.1.3 Summarization Scenario Benchmark ##### 5.1.3.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 44.82 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4210 Total generated tokens (retokenized): 4210 Request throughput (req/s): 0.22 Input token throughput (tok/s): 935.86 Output token throughput (tok/s): 93.94 Peak output token throughput (tok/s): 99.00 Peak concurrent requests: 2 Total token throughput (tok/s): 1029.80 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4479.60 Median E2E Latency (ms): 3622.99 ---------------Time to First Token---------------- Mean TTFT (ms): 139.90 Median TTFT (ms): 114.85 P99 TTFT (ms): 225.17 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.31 Median TPOT (ms): 10.33 P99 TPOT (ms): 10.51 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.33 Median ITL (ms): 10.33 P95 ITL (ms): 10.73 P99 ITL (ms): 10.93 Max ITL (ms): 14.48 ================================================== ``` ##### 5.1.3.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 50.68 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41589 Total generated tokens (retokenized): 41578 Request throughput (req/s): 1.58 Input token throughput (tok/s): 5920.41 Output token throughput (tok/s): 820.69 Peak output token throughput (tok/s): 1200.00 Peak concurrent requests: 20 Total token throughput (tok/s): 6741.10 Concurrency: 13.90 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8805.54 Median E2E Latency (ms): 9368.79 ---------------Time to First Token---------------- Mean TTFT (ms): 284.29 Median TTFT (ms): 168.48 P99 TTFT (ms): 1027.21 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 16.81 Median TPOT (ms): 16.66 P99 TPOT (ms): 27.18 ---------------Inter-Token Latency---------------- Mean ITL (ms): 16.42 Median ITL (ms): 13.68 P95 ITL (ms): 17.23 P99 ITL (ms): 90.75 Max ITL (ms): 574.64 ================================================== ``` ##### 5.1.3.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 94.77 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 169680 Total generated tokens (retokenized): 169640 Request throughput (req/s): 3.38 Input token throughput (tok/s): 13441.86 Output token throughput (tok/s): 1790.43 Peak output token throughput (tok/s): 2687.00 Peak concurrent requests: 70 Total token throughput (tok/s): 15232.28 Concurrency: 58.63 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 17364.14 Median E2E Latency (ms): 17495.95 ---------------Time to First Token---------------- Mean TTFT (ms): 238.22 Median TTFT (ms): 203.27 P99 TTFT (ms): 510.48 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 32.50 Median TPOT (ms): 34.27 P99 TPOT (ms): 40.59 ---------------Inter-Token Latency---------------- Mean ITL (ms): 32.36 Median ITL (ms): 22.50 P95 ITL (ms): 97.81 P99 ITL (ms): 151.55 Max ITL (ms): 352.79 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * **Results**: * Qwen/Qwen3-235B-A22B-Instruct-2507 ```text Output theme={null} Accuracy: 0.945 Invalid: 0.000 Latency: 11.980 s Output throughput: 2358.105 token/s ``` # Qwen3-Coder Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3-Coder ## 1. Model Introduction [Qwen3-Coder](https://huggingface.co/collections/Qwen/qwen3-coder) is the latest code-focused large language model series from the Qwen team. Built on the foundation of Qwen3, Qwen3-Coder delivers exceptional performance in code generation, understanding, and reasoning tasks. **Key Features:** * **State-of-the-art Coding Performance**: Achieves top-tier results on HumanEval, MBPP, LiveCodeBench, and other major coding benchmarks. * **Tool Calling Support**: Native support for function calling and tool use, enabling seamless integration with external APIs and services. * **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents. * **Multilingual Code Support**: Proficient in Python, JavaScript, TypeScript, Java, C++, Go, Rust, and many other programming languages. * **MoE Architecture**: Efficient Mixture-of-Experts design for optimal performance-to-cost ratio. * **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified). * **NVIDIA GPU Support**: Compatible with NVIDIA GB200 and B200 GPUs via SGLang (verified). For more details, please refer to the [official Qwen3-Coder GitHub Repository](https://github.com/QwenLM/Qwen3-Coder). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations verified on AMD MI300X, MI325X, MI355X, NVIDIA B200, GB200, and Intel Xeon CPU hardware platforms. ### 3.1 Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and quantization method. ### 3.2 Configuration Tips **AMD (MI300X/MI325X/MI355X):** * **Memory Management**: We have verified successful deployment on MI300X/MI325X/MI355X with `--context-length 8192`. Larger context lengths may be supported but require additional memory. * **Expert Parallelism**: For 480B-A35B with FP8 quantization, `--ep 2` is required to satisfy the dimension alignment requirement. * **Page Size**: `--page-size 32` is recommended for MoE models to optimize memory usage. * **Environment Variable**: If you encounter aiter-related issues, try setting `SGLANG_USE_AITER=0`. **NVIDIA (B200/GB200):** * **GB200 Parallelism**: Use `--tp 4 --ep 4` on GB200. B200 uses the default NVIDIA settings generated above. * **NVFP4 Quantization**: Requires `--quantization modelopt_fp4` and uses a different model path (`nvidia/Qwen3-Coder-...`). * **DP Attention**: NVFP4 configuration supports DP attention (`--attn-dp-size `) for improved throughput. **Intel Xeon CPU:** * Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. **General:** * **Tool Use**: To enable tool calling capabilities, add `--tool-call-parser qwen3_coder` to the launch command. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Code Generation Example ```python Example theme={null} from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": "Write a Python function that implements binary search on a sorted list. Include docstring and type hints." } ] response = client.chat.completions.create( model="Qwen/Qwen3-Coder-480B-A35B-Instruct", messages=messages, max_tokens=2048, temperature=0.7 ) print(response.choices[0].message.content) ``` **Example Output:** ````text Output theme={null} ```python from typing import List, Optional, TypeVar T = TypeVar('T') def binary_search(arr: List[T], target: T) -> Optional[int]: """ Perform binary search on a sorted list to find the index of a target element. This function implements the binary search algorithm, which efficiently finds a target value in a sorted array by repeatedly dividing the search interval in half. Args: arr (List[T]): A sorted list of elements to search through. target (T): The element to search for in the list. Returns: Optional[int]: The index of the target element if found, None otherwise. Time Complexity: O(log n) where n is the number of elements in the array. Space Complexity: O(1) - iterative implementation uses constant extra space. Examples: >>> binary_search([1, 2, 3, 4, 5], 3) 2 >>> binary_search([1, 2, 3, 4, 5], 6) None >>> binary_search(['a', 'b', 'c', 'd'], 'b') 1 >>> binary_search([], 1) None """ if not arr: return None left: int = 0 right: int = len(arr) - 1 while left <= right: mid: int = (left + right) // 2 if arr[mid] == target: return mid elif arr[mid] < target: left = mid + 1 else: right = mid - 1 return None # Alternative recursive implementation def binary_search_recursive(arr: List[T], target: T, left: int = 0, right: Optional[int] = None) -> Optional[int]: """ Perform binary search recursively on a sorted list to find the index of a target element. Args: arr (List[T]): A sorted list of elements to search through. target (T): The element to search for in the list. left (int): Left boundary of the search range (inclusive). right (Optional[int]): Right boundary of the search range (inclusive). Returns: Optional[int]: The index of the target element if found, None otherwise. Time Complexity: O(log n) where n is the number of elements in the array. Space Complexity: O(log n) due to recursive call stack. Examples: >>> binary_search_recursive([1, 2, 3, 4, 5], 3) 2 >>> binary_search_recursive([1, 2, 3, 4, 5], 6) None """ if not arr: return None if right is None: right = len(arr) - 1 if left > right: return None mid: int = (left + right) // 2 if arr[mid] == target: return mid elif arr[mid] < target: return binary_search_recursive(arr, target, mid + 1, right) else: return binary_search_recursive(arr, target, left, mid - 1) ``` This implementation provides: 1. **Main function** (`binary_search`): An iterative implementation that's more memory-efficient 2. **Alternative function** (`binary_search_recursive`): A recursive implementation for educational purposes 3. **Type hints**: Using generics (`TypeVar`) to work with any comparable type 4. **Comprehensive docstring**: Including description, parameters, return value, complexity analysis, and examples 5. **Edge case handling**: Empty lists, elements not found, etc. 6. **Clear variable names**: Self-documenting code 7. **Examples**: Doctest-style examples in the docstring The function works with any sorted list of comparable elements (integers, strings, etc.) and returns the index of the target element if found, or `None` if not found. ```` #### 4.2.2 Tool Calling Example Qwen3-Coder supports tool calling capabilities. Enable the tool call parser during deployment. The following example uses 30B-A3B model: ```shell Command theme={null} SGLANG_USE_AITER=0 python -m sglang.launch_server \ --model Qwen/Qwen3-Coder-30B-A3B-Instruct \ --tp 1 \ --context-length 8192 \ --page-size 32 \ --tool-call-parser qwen3_coder ``` **Python Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) # Define available tools tools = [ { "type": "function", "function": { "name": "execute_code", "description": "Execute Python code and return the result", "parameters": { "type": "object", "properties": { "code": { "type": "string", "description": "The Python code to execute" } }, "required": ["code"] } } } ] response = client.chat.completions.create( model="Qwen/Qwen3-Coder-30B-A3B-Instruct", messages=[ {"role": "user", "content": "Calculate the factorial of 10 using Python"} ], tools=tools, temperature=0.7 ) # Check if the model wants to call a tool if response.choices[0].message.tool_calls: tool_call = response.choices[0].message.tool_calls[0] print(f"Tool: {tool_call.function.name}") print(f"Arguments: {tool_call.function.arguments}") else: # Model may return tool call in content format print(response.choices[0].message.content) ``` **Example Output:** ```text Output theme={null} Tool: execute_code Arguments: {"code": "def factorial(n):\n if n == 0 or n == 1:\n return 1\n else:\n return n * factorial(n-1)\n\nresult = factorial(10)\nresult"} ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X GPU (8x) * Model: Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 * Tensor Parallelism: 8 * Expert Parallelism: 2 * sglang version: 0.5.7 We use SGLang's built-in benchmarking tool to conduct performance evaluation with random dataset. #### 5.1.1 AMD Standard Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} SGLANG_USE_AITER=0 python -m sglang.launch_server \ --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ --tp 8 \ --ep 2 \ --context-length 8192 \ --page-size 32 \ --trust-remote-code ``` ##### 5.1.1.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 73.79 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4104 Request throughput (req/s): 0.14 Input token throughput (tok/s): 82.68 Output token throughput (tok/s): 57.19 Peak output token throughput (tok/s): 59.00 Peak concurrent requests: 2 Total token throughput (tok/s): 139.86 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7376.26 Median E2E Latency (ms): 5851.51 P90 E2E Latency (ms): 13351.89 P99 E2E Latency (ms): 16908.32 ---------------Time to First Token---------------- Mean TTFT (ms): 191.93 Median TTFT (ms): 126.06 P99 TTFT (ms): 662.15 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 17.06 Median TPOT (ms): 17.07 P99 TPOT (ms): 17.08 ---------------Inter-Token Latency---------------- Mean ITL (ms): 17.06 Median ITL (ms): 17.06 P95 ITL (ms): 17.14 P99 ITL (ms): 17.19 Max ITL (ms): 18.53 ================================================== ``` ##### 5.1.1.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 87.04 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40364 Request throughput (req/s): 0.92 Input token throughput (tok/s): 455.77 Output token throughput (tok/s): 468.83 Peak output token throughput (tok/s): 608.00 Peak concurrent requests: 20 Total token throughput (tok/s): 924.59 Concurrency: 13.76 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 14966.88 Median E2E Latency (ms): 15871.93 P90 E2E Latency (ms): 24983.41 P99 E2E Latency (ms): 29504.85 ---------------Time to First Token---------------- Mean TTFT (ms): 388.94 Median TTFT (ms): 157.49 P99 TTFT (ms): 1318.63 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 29.41 Median TPOT (ms): 29.22 P99 TPOT (ms): 43.48 ---------------Inter-Token Latency---------------- Mean ITL (ms): 28.64 Median ITL (ms): 26.42 P95 ITL (ms): 27.51 P99 ITL (ms): 131.63 Max ITL (ms): 995.11 ================================================== ``` ##### 5.1.1.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 177.82 Total input tokens: 158939 Total input text tokens: 158939 Total generated tokens: 170134 Total generated tokens (retokenized): 168387 Request throughput (req/s): 1.80 Input token throughput (tok/s): 893.84 Output token throughput (tok/s): 956.80 Peak output token throughput (tok/s): 1728.00 Peak concurrent requests: 70 Total token throughput (tok/s): 1850.64 Concurrency: 58.88 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 32716.53 Median E2E Latency (ms): 30896.37 P90 E2E Latency (ms): 65605.24 P99 E2E Latency (ms): 80970.63 ---------------Time to First Token---------------- Mean TTFT (ms): 372.97 Median TTFT (ms): 181.67 P99 TTFT (ms): 529.01 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 62.98 Median TPOT (ms): 50.44 P99 TPOT (ms): 204.24 ---------------Inter-Token Latency---------------- Mean ITL (ms): 60.95 Median ITL (ms): 37.87 P95 ITL (ms): 143.98 P99 ITL (ms): 148.02 Max ITL (ms): 36863.32 ================================================== ``` #### 5.1.2 NVIDIA (B200/GB200) Standard Scenario Benchmark The following runs use the same random dataset benchmark client commands as the AMD section. On B200, launch the server with the following command: ````bash theme={null} sglang serve --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 --tp 8 --ep 8 --context-length 8192 --page-size 32 --trust-remote-code ##### 5.1.2.1 FP8 Model - Low Concurrency: ```text Output ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 42.68 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4204 Request throughput (req/s): 0.23 Input token throughput (tok/s): 142.95 Output token throughput (tok/s): 98.88 Peak output token throughput (tok/s): 102.00 Peak concurrent requests: 2 Total token throughput (tok/s): 241.83 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4266.06 Median E2E Latency (ms): 3420.24 P90 E2E Latency (ms): 7717.19 P99 E2E Latency (ms): 9504.50 ---------------Time to First Token---------------- Mean TTFT (ms): 112.03 Median TTFT (ms): 112.70 P99 TTFT (ms): 115.35 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.87 Median TPOT (ms): 9.86 P99 TPOT (ms): 9.92 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.87 Median ITL (ms): 9.87 P95 ITL (ms): 10.06 P99 ITL (ms): 10.18 Max ITL (ms): 14.80 ================================================== ```` * Medium Concurrency: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 60.80 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40543 Request throughput (req/s): 1.32 Input token throughput (tok/s): 652.43 Output token throughput (tok/s): 671.13 Peak output token throughput (tok/s): 864.00 Peak concurrent requests: 20 Total token throughput (tok/s): 1323.57 Concurrency: 13.93 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 10587.26 Median E2E Latency (ms): 11486.18 P90 E2E Latency (ms): 17374.75 P99 E2E Latency (ms): 21107.18 ---------------Time to First Token---------------- Mean TTFT (ms): 155.27 Median TTFT (ms): 121.57 P99 TTFT (ms): 294.31 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 20.77 Median TPOT (ms): 21.13 P99 TPOT (ms): 23.62 ---------------Inter-Token Latency---------------- Mean ITL (ms): 20.49 Median ITL (ms): 18.73 P95 ITL (ms): 19.65 P99 ITL (ms): 98.85 Max ITL (ms): 536.87 ================================================== ``` * High Concurrency: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 100.07 Total input tokens: 158939 Total input text tokens: 158939 Total generated tokens: 170134 Total generated tokens (retokenized): 169119 Request throughput (req/s): 3.20 Input token throughput (tok/s): 1588.32 Output token throughput (tok/s): 1700.19 Peak output token throughput (tok/s): 2303.00 Peak concurrent requests: 71 Total token throughput (tok/s): 3288.51 Concurrency: 57.93 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 18114.01 Median E2E Latency (ms): 18279.15 P90 E2E Latency (ms): 30557.22 P99 E2E Latency (ms): 35889.84 ---------------Time to First Token---------------- Mean TTFT (ms): 346.40 Median TTFT (ms): 129.75 P99 TTFT (ms): 1370.20 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 33.76 Median TPOT (ms): 34.62 P99 TPOT (ms): 39.97 ---------------Inter-Token Latency---------------- Mean ITL (ms): 33.48 Median ITL (ms): 25.70 P95 ITL (ms): 99.36 P99 ITL (ms): 132.30 Max ITL (ms): 1132.39 ================================================== ``` ##### 5.1.2.2 NVFP4 Model * Low Concurrency: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 34.49 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4218 Request throughput (req/s): 0.29 Input token throughput (tok/s): 176.87 Output token throughput (tok/s): 122.34 Peak output token throughput (tok/s): 127.00 Peak concurrent requests: 2 Total token throughput (tok/s): 299.21 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3448.01 Median E2E Latency (ms): 2768.11 P90 E2E Latency (ms): 6225.73 P99 E2E Latency (ms): 7668.26 ---------------Time to First Token---------------- Mean TTFT (ms): 104.55 Median TTFT (ms): 105.38 P99 TTFT (ms): 105.63 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.94 Median TPOT (ms): 7.95 P99 TPOT (ms): 7.97 ---------------Inter-Token Latency---------------- Mean ITL (ms): 7.94 Median ITL (ms): 7.94 P95 ITL (ms): 8.05 P99 ITL (ms): 8.11 Max ITL (ms): 24.64 ================================================== ``` * Medium Concurrency: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 43.30 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 39975 Request throughput (req/s): 1.85 Input token throughput (tok/s): 916.16 Output token throughput (tok/s): 942.42 Peak output token throughput (tok/s): 1264.00 Peak concurrent requests: 21 Total token throughput (tok/s): 1858.57 Concurrency: 13.90 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7521.95 Median E2E Latency (ms): 8246.89 P90 E2E Latency (ms): 12370.93 P99 E2E Latency (ms): 15023.96 ---------------Time to First Token---------------- Mean TTFT (ms): 137.27 Median TTFT (ms): 109.59 P99 TTFT (ms): 208.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.69 Median TPOT (ms): 14.87 P99 TPOT (ms): 17.63 ---------------Inter-Token Latency---------------- Mean ITL (ms): 14.51 Median ITL (ms): 12.75 P95 ITL (ms): 13.33 P99 ITL (ms): 92.85 Max ITL (ms): 113.70 ================================================== ``` * High Concurrency: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 73.93 Total input tokens: 158939 Total input text tokens: 158939 Total generated tokens: 170134 Total generated tokens (retokenized): 168841 Request throughput (req/s): 4.33 Input token throughput (tok/s): 2149.98 Output token throughput (tok/s): 2301.42 Peak output token throughput (tok/s): 3497.00 Peak concurrent requests: 71 Total token throughput (tok/s): 4451.40 Concurrency: 58.28 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 13463.58 Median E2E Latency (ms): 13498.74 P90 E2E Latency (ms): 22957.10 P99 E2E Latency (ms): 26656.95 ---------------Time to First Token---------------- Mean TTFT (ms): 239.00 Median TTFT (ms): 113.42 P99 TTFT (ms): 713.87 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 25.13 Median TPOT (ms): 26.02 P99 TPOT (ms): 30.90 ---------------Inter-Token Latency---------------- Mean ITL (ms): 24.92 Median ITL (ms): 16.68 P95 ITL (ms): 93.33 P99 ITL (ms): 119.26 Max ITL (ms): 548.82 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. ##### AMD (MI300X/MI325X/MI355X) * **Results**: * Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 ``` Accuracy: 0.965 Invalid: 0.000 Latency: 23.084 s Output throughput: 1148.425 token/s ``` ##### NVIDIA (B200/GB200) For deployment commands, see [Section 3.1](#3-1-configuration). * Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 ``` Accuracy: 0.965 Invalid: 0.000 Latency: 14.870 s Output throughput: 1777.726 token/s ``` * nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP (NVFP4) ``` Accuracy: 0.960 Invalid: 0.000 Latency: 13.948 s Output throughput: 1988.548 token/s ``` # Qwen3-Coder-Next Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3-Coder-Next ## 1. Model Introduction [Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) is a cost-efficient code-focused language model from the Qwen team (Alibaba). With 80B total parameters but only 3B activated parameters, it achieves performance comparable to models with 10–20x more active parameters through its innovative hybrid architecture. **Key Features:** * **Hybrid Architecture**: Uses a 48-layer hybrid layout combining Gated DeltaNet and Gated Attention with Mixture-of-Experts (512 total experts, 10 activated, 1 shared), enabling exceptional efficiency. * **Tool Calling Support**: Advanced agentic capabilities with native support for function calling and tool use via the `qwen3_coder` parser. * **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents. * **Cost-Efficient Inference**: Only 3B parameters activated per token, making it ideal for local development and cost-effective deployment at scale. * **IDE Integration**: Compatible with Claude Code, Qwen Code, Cline, and other IDE platforms. For more details, please refer to the [Qwen3-Coder-Next model card](https://huggingface.co/Qwen/Qwen3-Coder-Next). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). **Note:** Qwen3-Coder-Next requires SGLang v0.5.8 or later. ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options. ### 3.2 Configuration Tips * **Context Length**: The model supports up to 256K tokens natively. If you encounter OOM issues, try `--context-length 32768`. * **Tool Use**: To enable tool calling capabilities, use the `--tool-call-parser qwen3_coder` flag. * **Sampling Parameters**: SGLang automatically applies the recommended sampling parameters from the model's `generation_config.json`. No manual configuration is needed. * **Mamba Radix Cache**: Qwen3-Coder-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). * **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation **Deployment Command:** ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Coder-Next \ --tp 2 \ --tool-call-parser qwen3_coder \ --host 0.0.0.0 \ --port 30000 ``` ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Code Generation Example ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3-Coder-Next", messages=[ {"role": "user", "content": "Write a Python function that implements binary search on a sorted list. Include type hints."} ], max_tokens=2048 ) print(response.choices[0].message.content) ``` **Example Output:** ````text Output theme={null} Here's a Python function implementing binary search on a sorted list, with comprehensive type hints: ```python from typing import Sequence, TypeVar, Optional T = TypeVar('T') def binary_search(sorted_list: Sequence[T], target: T) -> Optional[int]: """ Perform binary search on a sorted list to find the index of a target element. Args: sorted_list: A sequence (e.g., list, tuple) sorted in ascending order. target: The element to search for in the list. Returns: The index of the target element if found, or None if not found. Time Complexity: O(log n) Space Complexity: O(1) Note: The function assumes the list is sorted in ascending order. If the list contains duplicate elements, it returns the index of one of them. """ left = 0 right = len(sorted_list) - 1 while left <= right: mid = (left + right) // 2 mid_val = sorted_list[mid] if mid_val == target: return mid elif mid_val < target: left = mid + 1 else: right = mid - 1 return None ``` ### Example usage: ```python # Example 1: Finding an existing element numbers = [1, 3, 5, 7, 9, 11] print(binary_search(numbers, 7)) # Output: 3 # Example 2: Element not in the list print(binary_search(numbers, 4)) # Output: None # Example 3: Empty list print(binary_search([], 5)) # Output: None # Example 4: Single element print(binary_search([1], 1)) # Output: 0 print(binary_search([1], 2)) # Output: None ``` ### Key features: - Uses `TypeVar` to support generic types (as long as comparison operations are defined) - Returns `Optional[int]` to indicate either the index or no match found - Uses `Sequence[T]` to accept any sequence type (list, tuple, etc.) - Includes comprehensive docstring with time/space complexity - Implements standard iterative binary search for O(1) space complexity ```` #### 4.2.2 Streaming Example ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3-Coder-Next", messages=[ {"role": "user", "content": "Explain the difference between a stack and a queue in 3 sentences."} ], max_tokens=512, stream=True ) for chunk in response: if chunk.choices and chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) print() ``` **Example Output:** ```text Output theme={null} A **stack** follows the **Last In, First Out (LIFO)** principle, meaning the last element added is the first one removed—operations like `push` (add) and `pop` (remove) occur at the same end, called the *top*. In contrast, a **queue** follows the **First In, First Out (FIFO)** principle, where elements are added at the *back* (enqueue) and removed from the *front* (dequeue), preserving the order of insertion. This structural difference makes stacks ideal for tasks like function call management and expression evaluation, while queues suit scheduling, buffering, and breadth-first traversal. ``` #### 4.2.3 Tool Calling Example Qwen3-Coder-Next supports tool calling capabilities. Make sure `--tool-call-parser qwen3_coder` is included in the deployment command above. **Python Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "execute_code", "description": "Execute Python code and return the result", "parameters": { "type": "object", "properties": { "code": { "type": "string", "description": "The Python code to execute" } }, "required": ["code"] } } } ] response = client.chat.completions.create( model="Qwen/Qwen3-Coder-Next", messages=[ {"role": "user", "content": "Calculate the factorial of 10 using Python"} ], tools=tools ) # Check if the model wants to call a tool if response.choices[0].message.tool_calls: tool_call = response.choices[0].message.tool_calls[0] print(f"Tool: {tool_call.function.name}") print(f"Arguments: {tool_call.function.arguments}") else: print(response.choices[0].message.content) ``` **Example Output:** ```text Output theme={null} Tool: execute_code Arguments: {"code": "import math\nmath.factorial(10)"} ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (2x) * Model: Qwen/Qwen3-Coder-Next * Tensor Parallelism: 2 * sglang version: 0.5.8+ #### 5.1.1 Standard Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Coder-Next \ --tp 2 \ --host 0.0.0.0 \ --port 30000 ``` ##### 5.1.1.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 27.86 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4218 Request throughput (req/s): 0.36 Input token throughput (tok/s): 219.00 Output token throughput (tok/s): 151.48 Peak output token throughput (tok/s): 166.00 Peak concurrent requests: 2 Total token throughput (tok/s): 370.48 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2784.14 Median E2E Latency (ms): 2258.08 P90 E2E Latency (ms): 5044.43 P99 E2E Latency (ms): 6130.52 ---------------Time to First Token---------------- Mean TTFT (ms): 161.68 Median TTFT (ms): 168.09 P99 TTFT (ms): 183.26 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.19 Median TPOT (ms): 6.23 P99 TPOT (ms): 6.32 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.23 Median ITL (ms): 6.23 P95 ITL (ms): 6.51 P99 ITL (ms): 6.64 Max ITL (ms): 13.45 ================================================== ``` ##### 5.1.1.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 39.06 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40789 Request throughput (req/s): 2.05 Input token throughput (tok/s): 1015.62 Output token throughput (tok/s): 1044.73 Peak output token throughput (tok/s): 1664.00 Peak concurrent requests: 21 Total token throughput (tok/s): 2060.34 Concurrency: 14.16 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6910.97 Median E2E Latency (ms): 7248.27 P90 E2E Latency (ms): 11612.63 P99 E2E Latency (ms): 13933.91 ---------------Time to First Token---------------- Mean TTFT (ms): 183.48 Median TTFT (ms): 156.50 P99 TTFT (ms): 311.46 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 13.61 Median TPOT (ms): 13.59 P99 TPOT (ms): 21.11 ---------------Inter-Token Latency---------------- Mean ITL (ms): 13.22 Median ITL (ms): 9.76 P95 ITL (ms): 10.43 P99 ITL (ms): 158.04 Max ITL (ms): 394.39 ================================================== ``` ##### 5.1.1.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 102.81 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 252536 Request throughput (req/s): 4.86 Input token throughput (tok/s): 2429.99 Output token throughput (tok/s): 2457.53 Peak output token throughput (tok/s): 5299.00 Peak concurrent requests: 109 Total token throughput (tok/s): 4887.52 Concurrency: 94.28 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 19385.20 Median E2E Latency (ms): 17584.09 P90 E2E Latency (ms): 36762.15 P99 E2E Latency (ms): 42518.35 ---------------Time to First Token---------------- Mean TTFT (ms): 270.62 Median TTFT (ms): 159.65 P99 TTFT (ms): 938.90 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 38.57 Median TPOT (ms): 41.78 P99 TPOT (ms): 53.28 ---------------Inter-Token Latency---------------- Mean ITL (ms): 37.90 Median ITL (ms): 18.26 P95 ITL (ms): 167.82 P99 ITL (ms): 311.45 Max ITL (ms): 993.20 ================================================== ``` #### 5.1.2 Reasoning Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Coder-Next \ --tp 2 \ --host 0.0.0.0 \ --port 30000 ``` ##### 5.1.2.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 285.02 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 44462 Total generated tokens (retokenized): 44432 Request throughput (req/s): 0.04 Input token throughput (tok/s): 21.41 Output token throughput (tok/s): 156.00 Peak output token throughput (tok/s): 173.00 Peak concurrent requests: 2 Total token throughput (tok/s): 177.40 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 28499.54 Median E2E Latency (ms): 30424.65 P90 E2E Latency (ms): 49132.26 P99 E2E Latency (ms): 51075.28 ---------------Time to First Token---------------- Mean TTFT (ms): 95.51 Median TTFT (ms): 93.86 P99 TTFT (ms): 112.56 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.24 Median TPOT (ms): 6.30 P99 TPOT (ms): 6.60 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.39 Median ITL (ms): 6.34 P95 ITL (ms): 7.16 P99 ITL (ms): 7.42 Max ITL (ms): 12.48 ================================================== ``` ##### 5.1.2.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 237.77 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 318306 Total generated tokens (retokenized): 315646 Request throughput (req/s): 0.34 Input token throughput (tok/s): 166.83 Output token throughput (tok/s): 1338.72 Peak output token throughput (tok/s): 1727.00 Peak concurrent requests: 19 Total token throughput (tok/s): 1505.55 Concurrency: 13.88 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 41266.21 Median E2E Latency (ms): 41010.10 P90 E2E Latency (ms): 77574.22 P99 E2E Latency (ms): 82688.04 ---------------Time to First Token---------------- Mean TTFT (ms): 140.73 Median TTFT (ms): 84.52 P99 TTFT (ms): 365.86 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.32 Median TPOT (ms): 10.38 P99 TPOT (ms): 10.87 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.34 Median ITL (ms): 10.19 P95 ITL (ms): 10.75 P99 ITL (ms): 11.18 Max ITL (ms): 206.79 ================================================== ``` ##### 5.1.2.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 384.82 Total input tokens: 158939 Total input text tokens: 158939 Total generated tokens: 1301025 Total generated tokens (retokenized): 1299908 Request throughput (req/s): 0.83 Input token throughput (tok/s): 413.02 Output token throughput (tok/s): 3380.83 Peak output token throughput (tok/s): 4317.00 Peak concurrent requests: 69 Total token throughput (tok/s): 3793.85 Concurrency: 56.42 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 67847.54 Median E2E Latency (ms): 70724.38 P90 E2E Latency (ms): 120888.83 P99 E2E Latency (ms): 133234.48 ---------------Time to First Token---------------- Mean TTFT (ms): 212.24 Median TTFT (ms): 115.96 P99 TTFT (ms): 652.93 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 16.76 Median TPOT (ms): 16.99 P99 TPOT (ms): 18.18 ---------------Inter-Token Latency---------------- Mean ITL (ms): 16.64 Median ITL (ms): 15.83 P95 ITL (ms): 31.64 P99 ITL (ms): 90.85 Max ITL (ms): 576.60 ================================================== ``` #### 5.1.3 Summarization Scenario Benchmark ##### 5.1.3.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 29.42 Total input tokens: 41941 Total input text tokens: 41941 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.34 Input token throughput (tok/s): 1425.35 Output token throughput (tok/s): 143.42 Peak output token throughput (tok/s): 169.00 Peak concurrent requests: 3 Total token throughput (tok/s): 1568.77 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2941.19 Median E2E Latency (ms): 2411.84 P90 E2E Latency (ms): 5661.26 P99 E2E Latency (ms): 6497.45 ---------------Time to First Token---------------- Mean TTFT (ms): 139.46 Median TTFT (ms): 160.33 P99 TTFT (ms): 184.30 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.56 Median TPOT (ms): 6.65 P99 TPOT (ms): 7.29 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.65 Median ITL (ms): 6.68 P95 ITL (ms): 7.39 P99 ITL (ms): 7.51 Max ITL (ms): 16.34 ================================================== ``` ##### 5.1.3.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 41.62 Total input tokens: 300020 Total input text tokens: 300020 Total generated tokens: 41669 Total generated tokens (retokenized): 41664 Request throughput (req/s): 1.92 Input token throughput (tok/s): 7208.67 Output token throughput (tok/s): 1001.19 Peak output token throughput (tok/s): 1536.00 Peak concurrent requests: 21 Total token throughput (tok/s): 8209.86 Concurrency: 14.27 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7421.29 Median E2E Latency (ms): 7985.77 P90 E2E Latency (ms): 12122.09 P99 E2E Latency (ms): 14595.05 ---------------Time to First Token---------------- Mean TTFT (ms): 248.49 Median TTFT (ms): 179.25 P99 TTFT (ms): 915.90 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.13 Median TPOT (ms): 14.28 P99 TPOT (ms): 24.02 ---------------Inter-Token Latency---------------- Mean ITL (ms): 13.80 Median ITL (ms): 10.46 P95 ITL (ms): 11.00 P99 ITL (ms): 173.14 Max ITL (ms): 823.32 ================================================== ``` ##### 5.1.3.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 85.74 Total input tokens: 1273893 Total input text tokens: 1273893 Total generated tokens: 170000 Total generated tokens (retokenized): 169983 Request throughput (req/s): 3.73 Input token throughput (tok/s): 14858.12 Output token throughput (tok/s): 1982.80 Peak output token throughput (tok/s): 3734.00 Peak concurrent requests: 70 Total token throughput (tok/s): 16840.92 Concurrency: 59.75 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16008.12 Median E2E Latency (ms): 15460.65 P90 E2E Latency (ms): 27705.81 P99 E2E Latency (ms): 32874.74 ---------------Time to First Token---------------- Mean TTFT (ms): 476.99 Median TTFT (ms): 177.50 P99 TTFT (ms): 3014.39 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 29.81 Median TPOT (ms): 31.19 P99 TPOT (ms): 45.53 ---------------Inter-Token Latency---------------- Mean ITL (ms): 29.29 Median ITL (ms): 15.75 P95 ITL (ms): 173.94 P99 ITL (ms): 202.00 Max ITL (ms): 2783.23 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. * **Test Results:** ```text Output theme={null} Accuracy: 0.965 Invalid: 0.000 Latency: 26.407 s Output throughput: 929.132 token/s ``` #### 5.2.2 MMLU Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run mmlu \ --base-url http://127.0.0.1:30000/v1 \ --num-threads 64 --max-tokens 2048 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. * **Test Results:** ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.780 subject: anatomy, #q:135, acc: 0.807 subject: astronomy, #q:152, acc: 0.921 subject: business_ethics, #q:100, acc: 0.820 subject: clinical_knowledge, #q:265, acc: 0.860 subject: college_biology, #q:144, acc: 0.944 subject: college_chemistry, #q:100, acc: 0.590 subject: college_computer_science, #q:100, acc: 0.820 subject: college_mathematics, #q:100, acc: 0.800 subject: college_medicine, #q:173, acc: 0.803 subject: college_physics, #q:102, acc: 0.775 subject: computer_security, #q:100, acc: 0.880 subject: conceptual_physics, #q:235, acc: 0.936 subject: econometrics, #q:114, acc: 0.807 subject: electrical_engineering, #q:145, acc: 0.834 subject: elementary_mathematics, #q:378, acc: 0.854 subject: formal_logic, #q:126, acc: 0.802 subject: global_facts, #q:100, acc: 0.610 subject: high_school_biology, #q:310, acc: 0.971 subject: high_school_chemistry, #q:203, acc: 0.803 subject: high_school_computer_science, #q:100, acc: 0.920 subject: high_school_european_history, #q:165, acc: 0.891 subject: high_school_geography, #q:198, acc: 0.929 subject: high_school_government_and_politics, #q:193, acc: 0.969 subject: high_school_macroeconomics, #q:390, acc: 0.903 subject: high_school_mathematics, #q:270, acc: 0.689 subject: high_school_microeconomics, #q:238, acc: 0.962 subject: high_school_physics, #q:151, acc: 0.854 subject: high_school_psychology, #q:545, acc: 0.947 subject: high_school_statistics, #q:216, acc: 0.815 subject: high_school_us_history, #q:204, acc: 0.907 subject: high_school_world_history, #q:237, acc: 0.937 subject: human_aging, #q:223, acc: 0.821 subject: human_sexuality, #q:131, acc: 0.840 subject: international_law, #q:121, acc: 0.934 subject: jurisprudence, #q:108, acc: 0.870 subject: logical_fallacies, #q:163, acc: 0.847 subject: machine_learning, #q:112, acc: 0.812 subject: management, #q:103, acc: 0.922 subject: marketing, #q:234, acc: 0.923 subject: medical_genetics, #q:100, acc: 0.970 subject: miscellaneous, #q:783, acc: 0.941 subject: moral_disputes, #q:346, acc: 0.850 subject: moral_scenarios, #q:895, acc: 0.726 subject: nutrition, #q:306, acc: 0.915 subject: philosophy, #q:311, acc: 0.859 subject: prehistory, #q:324, acc: 0.889 subject: professional_accounting, #q:282, acc: 0.723 subject: professional_law, #q:1534, acc: 0.648 subject: professional_medicine, #q:272, acc: 0.923 subject: professional_psychology, #q:612, acc: 0.845 subject: public_relations, #q:110, acc: 0.782 subject: security_studies, #q:245, acc: 0.796 subject: sociology, #q:201, acc: 0.925 subject: us_foreign_policy, #q:100, acc: 0.950 subject: virology, #q:166, acc: 0.572 subject: world_religions, #q:171, acc: 0.883 Total latency: 208.985 Average accuracy: 0.834 ``` # Qwen3-Next Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3-Next ## 1. Model Introduction [Qwen3-Next](https://huggingface.co/collections/Qwen/qwen3-next) is an advanced large language model architecture developed by Alibaba's Qwen team, designed to enhance efficiency and performance in handling extensive contexts and large-scale parameters. It features advanced capabilities in reasoning, function calling, and multilingual understanding. Qwen3-Next introduces several groundbreaking innovations: * **Hybrid Attention Mechanism**: Replaces standard attention with a combination of **Gated DeltaNet** (linear attention) and **Full Attention**, enabling efficient processing of context lengths up to 262,144 tokens. This hybrid approach makes it ideal for analyzing lengthy documents such as entire books or contracts. * **Highly Sparse Mixture-of-Experts (MoE)**: Features an 80-billion parameter architecture where only 3 billion parameters are active during inference. This design reduces computational costs by up to 90% while maintaining high performance, drastically reducing FLOPs per token without compromising model capacity. * **Multi-Token Prediction (MTP)**: Enables generation of multiple tokens per inference step, significantly reducing latency and enhancing user experience in real-time applications. This innovation boosts both pretraining performance and inference speed. * **Multilingual Support**: Natively supports 119 languages, facilitating seamless cross-lingual tasks and making it versatile for global applications. * **Enterprise-Ready Deployment**: Released under the Apache 2.0 license, offering flexible deployment options including on-premises, virtual private cloud (VPC), and private cloud environments, ensuring security and compliance for enterprise use. * **Advanced Reasoning & Stability**: Demonstrates clear improvement in reasoning performance with support for tool use during inference. Includes stability optimizations such as **zero-centered** and **weight-decayed layernorm** for robust pre-training and post-training. For more details, please refer to the [official Qwen3-Next blog](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd\&from=research.latest-advancements-list). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The Qwen3-Next series comes in only one size but offers different thinking modes. Recommended starting configurations vary depending on hardware. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities. ### 3.2 Configuration Tips * `--max-mamba-cache-size`: Adjust `--max-mamba-cache-size` to increase mamba cache space and max running requests capability. It will decrease KV cache space as a trade-off. You can adjust it according to workload. * `--mamba-ssm-dtype`: `bfloat16` or `float32`, use `bfloat16` to save mamba cache size and `float32` to get more accurate results. The default setting is `float32`. * `--mamba-full-memory-ratio`: Adjust `--mamba-full-memory-ratio` to set the ratio of mamba state memory to full kv cache memory. The default setting is `0.9`. * **Mamba Radix Cache**: Qwen3-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). * **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser 1. **Streaming with Thinking Process:** Qwen3-Next-80B-A3B-Thinking only supports thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections. ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Thinking \ --reasoning-parser qwen3 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="Qwen/Qwen3-Next-80B-A3B-Thinking", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= Okay, let's see. I need to find 15% of 240. Hmm, percentages. Right, "percent" means per hundred, so 15% is 15 per 100, or 15/100. To find a percentage of a number, I think you multiply the number by the percentage as a decimal. So first, maybe convert 15% to a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Then, multiply that by 240. Let me check that. So 0.15 times 240. Let's calculate that. Maybe break it down. 10% of 240 is 24, because 10% is just moving the decimal one place left, so 240 becomes 24. Then 5% would be half of 10%, so half of 24 is 12. So 10% + 5% = 15%, so 24 + 12 = 36. Oh, that's another way to do it. Let me verify with the multiplication. 0.15 * 240. Let's do 240 * 0.1 = 24, 240 * 0.05 = 12, so 24 + 12 = 36. Yep, that works. Alternatively, 240 * 15 = 3600, then divide by 100, which is 36. Because 15% of 240 is (15/100)*240 = (15*240)/100. 15*240: 10*240=2400, 5*240=1200, so 2400+1200=3600. Then 3600/100=36. So that's 36. So the answer should be 36. Let me make sure. 15% of 240. If I take 240 and multiply by 0.15, 240*0.15. Let's compute 240*0.1=24, 240*0.05=12, so 24+12=36. Yep, that's right. So 15% of 240 is 36. =============== Content ================= To find **15% of 240**, follow these steps: --- ### **Step 1: Understand what "percent" means** - "Percent" means **per hundred**, so **15% = 15/100 = 0.15** in decimal form. --- ### **Step 2: Multiply the number by the decimal** - To find 15% of 240, multiply: $$ 240 \times 0.15 $$ --- ### **Step 3: Break it down for clarity (optional but helpful)** - **10% of 240** = $ 240 \times 0.1 = 24 $ - **5% of 240** = $ 240 \times 0.05 = 12 $ - Add them together: $$ 24 + 12 = 36 $$ --- ### **Step 4: Confirm with direct multiplication** - $ 240 \times 0.15 = 36 $ --- ### ✅ Final Answer: $$ \boxed{36} $$ ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. 2. **Turn off Thinking:** Qwen3-Next-80B-A3B-Instruct only supports instruct (non-thinking) mode. ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Instruct \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Turn off thinking process response = client.chat.completions.create( model="Qwen/Qwen3-Next-80B-A3B-Instruct", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True, extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} To find **15% of 240**, follow these steps: --- ### **Step 1: Understand what percentage means** "Percent" means "per hundred," so **15%** is the same as **15 per 100**, or the fraction: $$ \frac{15}{100} $$ --- ### **Step 2: Multiply the fraction by the number** To find 15% of 240, multiply: $$ \frac{15}{100} \times 240 $$ --- ### **Step 3: Simplify the multiplication** You can simplify this in a couple of ways. #### **Option A: Multiply first, then divide** $$ 15 \times 240 = 3600 $$ Then divide by 100: $$ \frac{3600}{100} = 36 $$ #### **Option B: Simplify the fraction first** $$ \frac{15}{100} = \frac{3}{20} \quad \text{(divided numerator and denominator by 5)} $$ Now multiply: $$ \frac{3}{20} \times 240 = \frac{3 \times 240}{20} = \frac{720}{20} = 36 $$ --- ### **Step 4: Final Answer** $$ \boxed{36} $$ So, **15% of 240 is 36**. ``` #### 4.2.2 Tool Calling Qwen/Qwen3-Next-80B-A3B-Instruct | Qwen/Qwen3-Next-80B-A3B-Thinking both support tool calling capabilities. Enable the tool call parser: **Python Example (without Thinking Process):** Start sglang server: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Instruct \ --tool-call-parser qwen \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="Qwen/Qwen3-Next-80B-A3B-Instruct", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"🔧 Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} {"name": "get_weather", "arguments": {"location": "Beijing"}} ``` **Python Example (with Thinking Process):** Start sglang server: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Thinking \ --reasoning-parser qwen3 \ --tool-call-parser qwen \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="Qwen/Qwen3-Next-80B-A3B-Thinking", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"🔧 Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= Okay, the user is asking for the weather in Beijing. Let me check the available tools. There's a get_weather function that requires location and optionally unit. The location is needed, so I need to provide Beijing as the location. The unit is optional, but the user didn't specify Celsius or Fahrenheit. Since the default might be Celsius, but maybe I should check if the parameters require unit. Wait, the required field is only location, so unit is optional. So I can just call get_weather with location "Beijing" and not include the unit. Let me confirm the parameters. The parameters for get_weather have location as required, and unit is an enum with celsius or fahrenheit, but not required. So the correct call is to send location as Beijing, and omit unit. So the tool call should be {"name": "get_weather", "arguments": {"location": "Beijing"}}. {"name": "get_weather", "arguments": {"location": "Beijing"}} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="Qwen/Qwen3-Next-80B-A3B-Thinking", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.3 Processing Ultra-Long Texts Qwen3-Next natively supports context lengths of up to 262,144 tokens. For conversations where the total length (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively. We have validated the model's performance on context lengths of up to 1 million tokens using the YaRN method. **Qwen3-Next-80B-A3B-Instruct** ```shell Command theme={null} SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Instruct --tp 8 --host 0.0.0.0 --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000 ``` **Qwen3-Next-80B-A3B-Thinking** ```shell Command theme={null} SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Thinking --reasoning-parser qwen3 --tp 8 --host 0.0.0.0 --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000 ``` #### 4.2.4 Multi-Token Prediction (NEXTN Speculative Decoding) Qwen3-Next ships built-in Multi-Token Prediction (MTP) layers and supports [EAGLE-style speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding) through the `NEXTN` algorithm. The MTP weights are bundled in the main checkpoint, so no separate draft model is required. ```shell Command theme={null} python3 -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Instruct \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp 4 ``` Tune `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens` for your workload with [bench\_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). See [PR #10233](https://github.com/sgl-project/sglang/pull/10233) for implementation details. ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (8x) * Tensor Parallelism: 8 * Model: Qwen/Qwen3-Next-80B-A3B-Instruct * sglang version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. #### 5.1.1 Latency-Sensitive Benchmark * Server Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Instruct \ --tp 8 ``` * Test Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --num-prompt 100 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 100 Benchmark duration (s): 146.52 Total input tokens: 33839 Total input text tokens: 33839 Total input vision tokens: 0 Total generated tokens: 21640 Total generated tokens (retokenized): 21619 Request throughput (req/s): 0.68 Input token throughput (tok/s): 230.95 Output token throughput (tok/s): 147.70 Peak output token throughput (tok/s): 164.00 Peak concurrent requests: 6 Total token throughput (tok/s): 378.65 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1464.81 Median E2E Latency (ms): 1077.48 ---------------Time to First Token---------------- Mean TTFT (ms): 127.88 Median TTFT (ms): 132.88 P99 TTFT (ms): 212.85 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.19 Median TPOT (ms): 6.17 P99 TPOT (ms): 6.64 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.21 Median ITL (ms): 6.16 P95 ITL (ms): 6.51 P99 ITL (ms): 6.71 Max ITL (ms): 10.07 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Server Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Instruct \ --tp 8 \ ``` * Test Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --num-prompt 1000 \ --max-concurrency 100 ``` **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 100.32 Total input tokens: 302118 Total input text tokens: 302118 Total input vision tokens: 0 Total generated tokens: 195775 Total generated tokens (retokenized): 195016 Request throughput (req/s): 9.97 Input token throughput (tok/s): 3011.69 Output token throughput (tok/s): 1951.60 Peak output token throughput (tok/s): 5909.00 Peak concurrent requests: 120 Total token throughput (tok/s): 4963.29 Concurrency: 93.05 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 9333.98 Median E2E Latency (ms): 6054.12 ---------------Time to First Token---------------- Mean TTFT (ms): 161.77 Median TTFT (ms): 137.94 P99 TTFT (ms): 503.29 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 50.87 Median TPOT (ms): 50.28 P99 TPOT (ms): 122.87 ---------------Inter-Token Latency---------------- Mean ITL (ms): 47.11 Median ITL (ms): 13.84 P95 ITL (ms): 195.33 P99 ITL (ms): 289.56 Max ITL (ms): 486.38 ================================================== ``` ### 5.2 Accuracy Benchmark ### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:8000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * **Results**: * Qwen3-Next-80B-A3B-Instruct ``` Accuracy: 0.960 Invalid: 0.000 Latency: 12.673 s Output throughput: 2538.255 token/s ``` * Qwen3-Next-80B-A3B-Thinking ``` Accuracy: 0.935 Invalid: 0.000 Latency: 9.912 s Output throughput: 3288.737 token/s ``` ### 5.2.2 MMLU Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run mmlu \ --base-url http://127.0.0.1:30000/v1 \ --num-threads 64 --max-tokens 2048 ``` The command evaluates all 57 subjects. Historical results below use the removed 10-subject harness and are not measurements of the sgl-eval command above. * **Results**: * Qwen3-Next-80B-A3B-Instruct ``` subject: abstract_algebra, #q:100, acc: 0.800 subject: anatomy, #q:135, acc: 0.807 subject: astronomy, #q:152, acc: 0.947 subject: business_ethics, #q:100, acc: 0.810 subject: clinical_knowledge, #q:265, acc: 0.894 subject: college_biology, #q:144, acc: 0.972 subject: college_chemistry, #q:100, acc: 0.680 subject: college_computer_science, #q:100, acc: 0.860 subject: college_mathematics, #q:100, acc: 0.780 subject: college_medicine, #q:173, acc: 0.861 Total latency: 10.098 Average accuracy: 0.856 ``` * Qwen3-Next-80B-A3B-Thinking ``` subject: abstract_algebra, #q:100, acc: 0.780 subject: anatomy, #q:135, acc: 0.815 subject: astronomy, #q:152, acc: 0.941 subject: business_ethics, #q:100, acc: 0.870 subject: clinical_knowledge, #q:265, acc: 0.894 subject: college_biology, #q:144, acc: 0.965 subject: college_chemistry, #q:100, acc: 0.670 subject: college_computer_science, #q:100, acc: 0.840 subject: college_mathematics, #q:100, acc: 0.770 subject: college_medicine, #q:173, acc: 0.861 Total latency: 10.236 Average accuracy: 0.855 ``` # Qwen3-VL Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3-VL ## 1. Model Introduction [Qwen3-VL series](https://github.com/QwenLM/Qwen3-VL) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications. This generation delivers comprehensive upgrades across the board: * **Superior text understanding & generation**: Qwen3-VL-235B-A22B-Instruct was ranked as the [#1 open model for text on lmarena.ai](https://x.com/arena/status/1973151703563460942) * **Deeper visual perception & reasoning**: Enhanced image and video understanding capabilities. * **Extended context length**: Supports up to 262K tokens for processing long documents and videos. * **Enhanced spatial and video dynamics comprehension**: Better understanding of spatial relationships and temporal dynamics. * **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance. * **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions. For more details, please refer to the [official Qwen3-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The Qwen3-VL series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs, as well as Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities. ### 3.2 Configuration Tips * **Multimodal attention backend** : Usually, `--mm-attention-backend` is default to `fa3` on H100/H200/A100 for better performance, but it is default to `triton_attn` on B200 for compatibility. * **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size \* number of images in current running requests.) * **Memory Management** : Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K. * **Expert Parallelism** : SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details. * **Kernel Tuning** : For MoE Triton kernel tuning on your specific hardware, refer to [fused\_moe\_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton). **Hardware-specific notes:** * **H100 (FP8):** Use the `Qwen/Qwen3-VL-235B-A22B-Instruct-FP8` checkpoint for best memory efficiency. * **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage. * **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing. **Additional multimodal server parameters:** * `--mm-feature-transport=cuda_ipc`: Keep multimodal feature tensors in a bounded GPU pool shared over CUDA IPC to avoid device-to-host memory copies, improving performance for high-frequency inference. **Example with full multimodal optimizations:** ```bash Command theme={null} SGLANG_USE_CUDA_IPC_TRANSPORT=1 \ SGLANG_VLM_CACHE_SIZE_MB=0 \ python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-235B-A22B-Instruct \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code \ --tp-size 8 \ --enable-cache-report \ --log-level info \ --max-running-requests 64 \ --mem-fraction-static 0.65 \ --chunked-prefill-size 8192 \ --attention-backend fa3 \ --mm-attention-backend fa3 \ --enable-metrics ``` * **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Multi-Modal Inputs Qwen3-VL supports both image and video inputs. Here's a basic example with image input: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Read all the text in the image." } ] } ] start = time.time() response = client.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Instruct", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 3.37s Generated text: Auntie Anne's CINNAMON SUGAR 1 x 17,000 17,000 SUB TOTAL 17,000 GRAND TOTAL 17,000 CASH IDR 20,000 CHANGE DUE 3,000 ``` **Multi-Image Input Example:** Qwen3-VL can process multiple images in a single request for comparison or analysis: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg" } }, { "type": "image_url", "image_url": { "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg" } }, { "type": "text", "text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image." } ] } ] start = time.time() response = client.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Instruct", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 10.18s Generated text: The two images present starkly different portrayals of Hong Kong’s iconic red taxis, contrasting a dynamic street-level moment with a static, large-scale gathering. The first image is a close-up, eye-level shot capturing a single red Toyota Crown taxi (license plate RX 5004) in motion or paused at an urban intersection. Its glossy red paint gleams under daylight, reflecting the vibrant, cluttered backdrop of a Hong Kong street — neon signs, glass-fronted shops displaying sunglasses, and Chinese characters. The taxi’s chrome grille, clear headlights, and black trim provide visual contrast. A green “4 SEATS” sticker and a “的士 TAXI” sign on the side reinforce its identity. The composition is intimate, focusing on the vehicle’s details — the texture of its paint, the slight reflections on the windows, and the crispness of its license plate. Other red taxis flank it, suggesting a bustling city rhythm, but the central taxi dominates the frame, conveying movement and immediacy. In contrast, the second image is an elevated, wide-angle shot of dozens of red taxis — along with a few green ones — parked in neat, grid-like rows on what appears to be a highway or staging area. The scene is static, almost ceremonial. Many taxis have their hoods open, suggesting maintenance, inspection, or protest. People are scattered among the vehicles, some inspecting engines, others conversing — adding a human, documentary element. The dominant color remains red, but the repetition creates a visual pattern rather than individual focus. The green taxis offer a subtle color contrast, hinting at different service zones (green for New Territories, red for urban areas). The setting is more utilitarian — concrete barriers, metal railings, and sparse vegetation — with an overpass looming in the background. The texture here is less about polished paint and more about the collective mass of vehicles, the asphalt, and the functional layout. Key contrasts emerge: the first image is kinetic and personal, emphasizing the taxi as a working vehicle in the city’s daily flow; the second is static and collective, portraying the taxis as a fleet, possibly for logistical or political purposes. The lighting in both is bright daylight, but the first has richer color saturation and depth due to its proximity and urban backdrop, while the second feels flatter, more documentary in tone. The first image invites you into the city’s pulse; the second invites you to observe a system — organized, perhaps even paused — from a distance. In essence, the first image celebrates the individual taxi in its natural habitat; the second reveals the scale and structure behind the fleet, transforming the familiar red icon into a symbol of coordination, maintenance, or collective action. Both are quintessentially Hong Kong, yet they offer vastly different narratives — one of motion and commerce, the other of assembly and purpose. ``` **Video Input Example:** Qwen3-VL supports video understanding by processing video URLs: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "video_url", "video_url": { "url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4" } }, { "type": "text", "text": "Describe what happens in this video." } ] } ] start = time.time() response = client.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Instruct", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Note:** * For video processing, ensure you have sufficient context length configured (up to 262K tokens) * Video processing may require more memory; adjust `--mem-fraction-static` accordingly * You can also provide local file paths using `file://` protocol **Example Output:** ```text Output theme={null} Response costs: 3.89s Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen. ``` #### 4.2.2 Reasoning Parser Qwen3-VL-Thinking supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-VL-235B-A22B-Thinking \ --reasoning-parser qwen3 \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Thinking", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.3 Tool Calling Qwen3-VL supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-VL-235B-A22B-Thinking \ --reasoning-parser qwen3 \ --tool-call-parser qwen \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Thinking", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Thinking", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (8x) * Model: Qwen3-VL-235B-A22B-Instruct * Tensor Parallelism: 8 * sglang version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-VL-235B-A22B-Instruct \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-VL-235B-A22B-Instruct \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 45.97 Total input tokens: 18348 Total input text tokens: 708 Total input vision tokens: 17640 Total generated tokens: 4220 Total generated tokens (retokenized): 3423 Request throughput (req/s): 0.22 Input token throughput (tok/s): 399.17 Output token throughput (tok/s): 91.81 Peak output token throughput (tok/s): 96.00 Peak concurrent requests: 2 Total token throughput (tok/s): 490.98 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4594.52 Median E2E Latency (ms): 3725.04 ---------------Time to First Token---------------- Mean TTFT (ms): 193.35 Median TTFT (ms): 196.32 P99 TTFT (ms): 222.75 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.44 Median TPOT (ms): 10.44 P99 TPOT (ms): 10.47 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.78 Median ITL (ms): 10.48 P95 ITL (ms): 21.01 P99 ITL (ms): 31.40 Max ITL (ms): 31.92 ================================================== ``` **Optimized Results (with CUDA IPC Transport):** For further TTFT optimization, enable CUDA IPC Transport for multimodal features by setting `SGLANG_USE_CUDA_IPC_TRANSPORT=1`. This significantly reduces TTFT by using CUDA IPC for transferring multimodal features. * Model Deployment Command: ```shell Command theme={null} SGLANG_USE_CUDA_IPC_TRANSPORT=1 python -m sglang.launch_server \ --model Qwen/Qwen3-VL-235B-A22B-Instruct \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-VL-235B-A22B-Instruct \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 100 \ --max-concurrency 1 ``` * **Test Results:** With `SGLANG_USE_CUDA_IPC_TRANSPORT=1`, TTFT improves significantly: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 100 Benchmark duration (s): 566.84 Total input tokens: 183667 Total input text tokens: 7267 Total input vision tokens: 176400 Total generated tokens: 52444 Total generated tokens (retokenized): 28702 Request throughput (req/s): 0.18 Input token throughput (tok/s): 324.02 Output token throughput (tok/s): 92.52 Peak output token throughput (tok/s): 96.00 Peak concurrent requests: 3 Total token throughput (tok/s): 416.54 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5667.50 Median E2E Latency (ms): 5830.00 ---------------Time to First Token---------------- Mean TTFT (ms): 191.16 Median TTFT (ms): 182.58 P99 TTFT (ms): 244.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.46 Median TPOT (ms): 10.46 P99 TPOT (ms): 10.48 ---------------Inter-Token Latency---------------- Mean ITL (ms): 13.91 Median ITL (ms): 10.56 P95 ITL (ms): 21.35 P99 ITL (ms): 31.55 Max ITL (ms): 42.36 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-VL-235B-A22B-Instruct \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-VL-235B-A22B-Instruct \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 584.65 Total input tokens: 1839015 Total input text tokens: 75015 Total input vision tokens: 1764000 Total generated tokens: 510855 Total generated tokens (retokenized): 284284 Request throughput (req/s): 1.71 Input token throughput (tok/s): 3145.50 Output token throughput (tok/s): 873.78 Peak output token throughput (tok/s): 2855.00 Peak concurrent requests: 107 Total token throughput (tok/s): 4019.29 Concurrency: 98.35 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 57502.05 Median E2E Latency (ms): 54301.08 ---------------Time to First Token---------------- Mean TTFT (ms): 5802.23 Median TTFT (ms): 1444.75 P99 TTFT (ms): 46675.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 100.22 Median TPOT (ms): 105.43 P99 TPOT (ms): 144.37 ---------------Inter-Token Latency---------------- Mean ITL (ms): 134.20 Median ITL (ms): 25.57 P95 ITL (ms): 558.14 P99 ITL (ms): 1449.01 Max ITL (ms): 33453.23 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 MMMU Benchmark You can evaluate the model's accuracy using the MMMU dataset with `lmms_eval`: * Benchmark Command: ```shell Command theme={null} uv pip install lmms_eval python3 -m lmms_eval \ --model openai_compatible \ --model_args "model=Qwen/Qwen3-VL-235B-A22B-Instruct,api_key=EMPTY,base_url=http://127.0.0.1:30000/v1/" \ --tasks mmmu_val \ --batch_size 128 \ --log_samples \ --log_samples_suffix "openai_compatible" \ --output_path ./logs \ --gen_kwargs "max_new_tokens=4096" ``` * **Test Results:** ```text Output theme={null}
Tasks Version Filter n-shot Metric Value Stderr
mmmu_val 0 none 0 mmmu_acc ↑ 0.6567 ± N/A
``` # Qwen3.5 Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.5 ## 1. Model Introduction [Qwen3.5-397B-A17B](https://huggingface.co/Qwen/Qwen3.5-397B-A17B) is the latest flagship model in the Qwen series developed by Alibaba, representing a significant leap forward with unified vision-language foundation, efficient hybrid architecture, and scalable reinforcement learning. Qwen3.5 features a Gated Delta Networks combined with sparse Mixture-of-Experts architecture (397B total parameters, 17B activated), delivering high-throughput inference with minimal latency. It supports multimodal inputs (text, image, video) and natively handles context lengths of up to 262,144 tokens, extensible to over 1M tokens. **Architecture details:** * **Hybrid Attention:** Gated Delta Networks (linear, O(n) complexity) combined with full attention every 4th layer — linear layers provide low-cost long-context processing while periodic full attention ensures high associative recall. * **MoE routing:** Top-10 active out of 512 routed experts plus a dedicated shared expert for universal features, keeping 17B parameters active from 397B total. * **Native multimodal:** DeepStack Vision Transformer with Conv3d temporal encoding for image and video understanding without separate visual encoders. **Key Features:** * **Unified Vision-Language Foundation**: Early fusion training on multimodal tokens achieves cross-generational parity with Qwen3 and outperforms Qwen3-VL models * **Efficient Hybrid Architecture**: Gated Delta Networks + sparse MoE (397B total / 17B active) for high-throughput inference * **Hybrid Reasoning**: Thinking mode enabled by default with step-by-step reasoning, can be disabled for direct responses * **Tool Calling**: Built-in tool calling support with `qwen3_coder` parser * **Multi-Token Prediction (MTP)**: Speculative decoding support for lower latency * **201 Language Support**: Expanded multilingual coverage across 201 languages and dialects **Available Models:**
Model BF16 (Full precision) FP8 (8-bit Quantized) FP4 (4-bit Quantized)
Qwen3.5-397B-A17B [Qwen/Qwen3.5-397B-A17B](https://huggingface.co/Qwen/Qwen3.5-397B-A17B) [Qwen/Qwen3.5-397B-A17B-FP8](https://huggingface.co/Qwen/Qwen3.5-397B-A17B-FP8) NVIDIA NVFP4: [nvidia/Qwen3.5-397B-A17B-NVFP4-V2](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4-V2)
AMD MXFP4: [amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2](https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2)
Qwen3.5-122B-A10B [Qwen/Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) [Qwen/Qwen3.5-122B-A10B-FP8](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8) -
Qwen3.5-35B-A3B [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) [Qwen/Qwen3.5-35B-A3B-FP8](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-FP8) -
Qwen3.5-27B [Qwen/Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B) [Qwen/Qwen3.5-27B-FP8](https://huggingface.co/Qwen/Qwen3.5-27B-FP8) -
Qwen3.5-9B [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) - -
Qwen3.5-4B [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) - -
Qwen3.5-2B [Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) - -
Qwen3.5-0.8B [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) - -
**License:** Apache 2.0 ## 2. SGLang Installation SGLang from the main branch is required for Qwen3.5. You can install from source or use a Docker image: ```bash Command theme={null} # Install from source uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' # Or use Docker (NVIDIA GPUs) docker pull lmsysorg/sglang:latest # Or use Docker (AMD MI300X/MI325X) docker pull lmsysorg/sglang-rocm:v0.5.15.post1-rocm720-mi30x-20260715 # Or use Docker (AMD MI355X) docker pull lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260927 ``` For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and capabilities. ### 3.2 Configuration Tips * Speculative decoding (MTP) can significantly reduce latency for interactive use cases. * **H100 / B200 / B300 FP8:** Add `--enable-symm-mem` to enable NCCL symmetric memory for faster collectives and better performance under multi-GPU settings. It only helps when tp > 1, so on B200/B300 it applies to the 397B-A17B FP8 recipe (tp=4); the 122B-A10B and 35B-A3B FP8 recipes are single-GPU there. * **AMD GPUs (MI300X / MI325X / MI355X):** Use `SGLANG_USE_AITER=1` and `SGLANG_USE_AITER_UNIFIED_ATTN=1` with `--attention-backend aiter`, which requires `--page-size 16` and can also enable `--enable-aiter-allreduce-fusion`. Additionally set `AITER_FLYDSL_FORCE=1` to force the AITER FlyDSL MoE kernels and `SGLANG_MAMBA_SSM_DTYPE=bfloat16` to store the Mamba SSM state in bfloat16 (instead of the default float32). For the **MXFP4 checkpoint on MI355X**, set `ROCM_QUICK_REDUCE_QUANTIZATION=INT4` to route multi-GPU collectives through INT4-quantized ROCm quick all-reduce, and drop `--enable-aiter-allreduce-fusion` (the two are mutually exclusive; quick all-reduce is preferred for this recipe). * **KV cache offloading (MXFP4 on MI355X):** Selecting **Host DRAM (HiCache)** adds a host-memory tier below the device KV cache, which lets long-context agentic workloads keep more prefix cached and reach higher concurrency than the device KV pool alone allows. It emits `--enable-hierarchical-cache --hicache-ratio 1.5 --hicache-write-policy write_through --hicache-io-backend kernel --hicache-mem-layout page_first`. HiCache extends the radix cache, so the generated command drops `--disable-radix-cache` — the two options are mutually exclusive and SGLang rejects them together at startup. `--hicache-ratio 1.5` sizes the host tier at 1.5x the device KV cache, so make sure the node has that much free CPU DRAM per server on top of the model weights. Leave this **Disabled** for fixed-length throughput serving, where the extra host traffic buys nothing. * **KV cache offloading (397B FP8 on MI355X):** A separate recipe from the MXFP4 one above, for long-context agentic serving of the FP8 checkpoint. Selecting **Host DRAM (HiCache)** emits `--tp 4` instead of the tp=2 that merely fits the weights, `--enable-hierarchical-cache --hicache-ratio 1.5 --hicache-write-policy write_through_selective --hicache-io-backend kernel --hicache-mem-layout page_first`, and `--kv-cache-dtype fp8_e4m3` to halve device KV bytes. `write_through_selective` keeps non-reusable blocks out of the host tier. This arm keeps `--enable-aiter-allreduce-fusion` and does not set `ROCM_QUICK_REDUCE_QUANTIZATION`; the MXFP4 recipe does the opposite. Read the FP8 KV cache warning below before using it for reasoning-heavy work. * **Watchdog timeout:** Increase `--watchdog-timeout` to `1200` or higher for this large model, as weight loading can take significant time. * **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs. * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). * The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload. * Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities. * To speed up weight loading for this large model, add `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'` to the launch command. * **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`. * **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200/B300. * **B200 (FP8)**: Add `--enable-flashinfer-allreduce-fusion` for optimized throughput on Blackwell. * **B200 / B300 (FP8 MoE)**: Add `--moe-runner-backend flashinfer_trtllm` to route the fused MoE through the trtllm-gen kernels on Blackwell. This is not picked up automatically — SGLang's auto-selection of `flashinfer_trtllm` is limited to the DeepSeek architecture family, so an FP8 Qwen3.5 MoE run otherwise falls back to the Triton MoE runner. It applies to the MoE sizes (397B-A17B / 122B-A10B / 35B-A3B); the dense sizes have no MoE layers. The NVFP4 recipes already set it. * **B200 / B300 (FP8)**: Add `--linear-attn-prefill-backend flashinfer` to run GDN (linear attention) prefill on the FlashInfer CuTe-DSL kernel. SGLang auto-selects it only inside a narrow validated domain (a chunked prefill size of at most 8192 and a bfloat16 mamba state pool, among other conditions), which these FP8 recipes fall outside of, so the override has to be explicit. Requires CUDA 13+ and flashinfer >= 0.6.14 — SGLang raises at startup on SM100 with an older CUDA. Applies to every Qwen3.5 size, since all of them are hybrid GDN models. * For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors. * Hardware requirements: * **BF16**: \~397B parameters require \~800GB of GPU memory for weights. * **H100 (80GB)** requires tp=16 (2 nodes) since each rank needs \~100GB at tp=8. * **H200 (141GB)** runs with tp=8. * **B200 (183GB)** runs with tp=8. * **B300 (275GB)** runs with tp=4. * **MI300X (192GB)** runs with tp=8. * **MI325X (256GB)** runs with tp=4. * **MI355X (288GB)** runs with tp=4. * **FP8**: The FP8 quantized model requires \~400GB for weights, cutting memory in half. * **H100 (80GB)** runs with tp=8. * **H200 (141GB)** runs with tp=4. * **B200 (183GB)** runs with tp=4. * **B300 (275GB)** runs with tp=2. * **MI300X (192GB)** runs with tp=4. * **MI325X (256GB)** runs with tp=2. * **MI355X (288GB)** runs with tp=2. * **FP4**: The FP4 quantized model requires \~250GB for weights, cutting memory by almost 4x. NVFP4 ([nvidia/Qwen3.5-397B-A17B-NVFP4-V2](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4-V2)) requires B200/B300 (Blackwell architecture); AMD provides an MXFP4 checkpoint with FP8 attention ([amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2](https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2)) for MI355X. * **B200 (183GB)** runs with tp=4 (tp=2 with expert parallelism 2 when MTP is enabled). (NVFP4) * **B300 (275GB)** runs with tp=2. (NVFP4) * **MI355X (288GB)** runs with tp=2 (use tp=4 for low concurrency). (MXFP4)
Hardware Memory BF16 TP FP8 TP FP4 TP
H100 80GB 16 8 N/A
H200 141GB 8 4 N/A
B200 183GB 8 4 4 / 2 + EP2 (MTP)
B300 275GB 4 2 2
MI300X 192GB 8 4 N/A
MI325X 256GB 4 2 N/A
MI355X 288GB 4 2 / 4 (agentic) 2 / 4 (MXFP4)
**FP8 KV Cache**: `--kv-cache-dtype fp8_e4m3` quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is emitted for the MXFP4-on-MI355X recipe and for the 397B FP8-on-MI355X agentic recipe (Host DRAM / HiCache selected), where it is part of the tuned configuration; for every other combination the generated commands leave it out, so add it manually only if memory constraints require the trade-off. * **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ### 3.3 Agentic Long-Context Deployment Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments of Qwen3.5-397B-A17B, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order. * **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks). * **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`. * **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image. ## 4. Model Invocation **NVIDIA:** Deploy Qwen3.5-397B-A17B with the following command (H200, all features enabled): ```shell Command theme={null} sglang serve \ --model-path Qwen/Qwen3.5-397B-A17B \ --tp 8 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --speculative-algo NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --mem-fraction-static 0.8 \ --host 0.0.0.0 \ --port 30000 ``` **AMD:** Deploy Qwen3.5-397B-A17B with the following command (MI300X/MI325X/MI355X): ```shell Command theme={null} SGLANG_USE_AITER=1 \ SGLANG_USE_AITER_UNIFIED_ATTN=1 \ AITER_FLYDSL_FORCE=1 \ SGLANG_MAMBA_SSM_DTYPE=bfloat16 \ python3 -m sglang.launch_server \ --model-path Qwen/Qwen3.5-397B-A17B \ --tp 8 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --trust-remote-code \ --page-size 16 \ --mem-fraction-static 0.8 \ --attention-backend aiter \ --enable-aiter-allreduce-fusion \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --watchdog-timeout 1200 \ --host 0.0.0.0 \ --port 30000 ``` > **Note:** To enable MTP (speculative decoding), append `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Vision Input Qwen3.5 supports image and video inputs as a unified vision-language model. Here is an example with an image: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3.5-397B-A17B", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/CI_Demo/mathv-1327.jpg" } }, { "type": "text", "text": "Describe this image in detail." } ] } ], max_tokens=2048, stream=True ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user wants a detailed description of the provided image. 1. **Identify the main components:** * There is a central square. * There are four circles of varying sizes. 2. **Analyze the arrangement and relationships:** * **The Square:** It's in the middle, oriented upright (sides are vertical and horizontal). It's drawn with a thin black line. * **The Circles:** * **Top Right:** A large circle. It overlaps the top right corner of the square. A significant portion of the circle is outside the square, but a chunk of it is inside. * **Bottom Left:** A large circle, roughly the same size as the top right one. It overlaps the bottom left corner of the square. Again, a chunk is inside, most is outside. * **Top Left:** A smaller circle. It is positioned near the top left corner of the square. It overlaps the corner slightly. * **Bottom Right:** A smaller circle, roughly the same size as the top left one. It is positioned near the bottom right corner of the square. It overlaps the corner slightly. 3. **Synthesize the description:** * Start with a general overview: A geometric line drawing. * Describe the central shape: A square. * Describe the surrounding shapes: Four circles. * Detail the positioning: * Two large circles are diagonally opposite (bottom-left and top-right). * Two smaller circles are diagonally opposite (top-left and bottom-right). * Describe the interaction/overlap: The shapes are not just touching; they intersect. The lines cross over each other. The square acts as a central anchor. The circles seem to be "orbiting" or clustered around the corners of the square. * Mention the style: Black lines on a white background. Minimalist. No shading or color. 4. **Refine the details:** * Let's look closer at the overlaps. * **Top Left Circle:** It sits mostly above and to the left of the square's top-left corner. The bottom-right part of this small circle cuts into the square. * **Top Right Circle:** This is the largest circle (or tied for largest). Its bottom-left quadrant overlaps significantly with the top-right quadrant of the square. * **Bottom Left Circle:** This is also a large circle. Its top-right quadrant overlaps significantly with the bottom-left quadrant of the square. * **Bottom Right Circle:** This is a small circle. Its top-left part overlaps with the bottom-right corner of the square. 5. **Final Polish:** Structure the description logically. Start with the big picture, then break down each element's position relative to the center. * *Drafting the description:* The image shows a black and white line drawing of geometric shapes. In the center is a square. Surrounding it are four circles of two different sizes. * *Adding detail:* There are two large circles and two small circles. * *Positioning:* * A large circle is in the bottom left. It overlaps the bottom-left corner of the square. * A large circle is in the top right. It overlaps the top-right corner of the square. * A small circle is in the top left. It overlaps the top-left corner of the square. * A small circle is in the bottom right. It overlaps the bottom-right corner of the square. * *Visual style:* The lines are thin and black. The background is white. The shapes are transparent (wireframe style), so you can see the lines of the square through the circles and vice versa where they overlap. 6. **Final Review against the image:** Does this match? Yes. The description is accurate. The "transparency" point is important because it's a line drawing, not solid shapes. The lines cross. * Wait, let me look really closely at the "Top Left" circle. It actually looks like it's touching the top edge and left edge. It overlaps the corner. * Let me look at the "Bottom Right" circle. It overlaps the bottom right corner. * The arrangement creates a sort of diagonal symmetry. Large-Large on one diagonal, Small-Small on the other. Let's write the final output based on these observations. =============== Content ================= Based on the image provided, here is a detailed description: **Overview** The image is a minimalist, black-and-white line drawing featuring geometric shapes. It consists of a central square surrounded by four circles of varying sizes. The lines are thin and black against a plain white background. The shapes are drawn in a "wireframe" style, meaning they are transparent outlines; where shapes overlap, the lines cross over each other rather than one blocking the other. **Detailed Breakdown** 1. **The Central Square:** * There is a single square positioned in the center of the composition. It is oriented upright with vertical and horizontal sides. 2. **The Circles:** * There are four circles arranged around the corners of the square. They appear in two distinct sizes: two large circles and two smaller circles. * **Top Right:** A large circle is positioned at the top right. It overlaps significantly with the top-right corner of the square. A portion of the circle's interior is inside the square's boundary. * **Bottom Left:** Another large circle (roughly the same size as the top right one) is positioned at the bottom left. It overlaps significantly with the bottom-left corner of the square. * **Top Left:** A smaller circle is positioned near the top left corner. It overlaps slightly with the top-left corner of the square. * **Bottom Right:** A smaller circle (roughly the same size as the top left one) is positioned near the bottom right corner. It overlaps slightly with the bottom-right corner of the square. **Composition** The arrangement creates a diagonal symmetry. The two largest circles are on a diagonal from bottom-left to top-right, while the two smallest circles are on a diagonal from top-left to bottom-right. The intersecting lines create a complex web of curves and angles in the center of the image. ``` ### 4.3 Advanced Usage #### 4.3.1 Reasoning Parser Qwen3.5 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response. To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time: * **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed. * **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process. **Example 1: Thinking Mode (Default)** Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Thinking mode is enabled by default, no extra parameters needed response = client.chat.completions.create( model="Qwen/Qwen3.5-397B-A17B", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= Thinking Process: 1. **Analyze the Request:** The user wants to solve a math problem: "What is 15% of 240?" and expects a step-by-step explanation. 2. **Identify the Core Operation:** The problem asks for a percentage of a number. The formula is: $\text{Percentage} \times \text{Number} = \text{Result}$. * Percentage: $15\%$ * Number: $240$ 3. **Determine the Steps:** * Step 1: Convert the percentage to a decimal or fraction. * Step 2: Multiply the converted value by the number. * Step 3: Calculate the final result. * Alternative Step (Mental Math): Break down 15% into 10% + 5%. 4. **Draft the Explanation (Method 1: Decimal Conversion):** * Convert $15\%$ to $0.15$. * Multiply $0.15 \times 240$. * Calculation: $15 \times 24 = 360$, adjust decimal places $\rightarrow 36.0$. 5. **Draft the Explanation (Method 2: Fraction Conversion):** * Convert $15\%$ to $\frac{15}{100}$ or $\frac{3}{20}$. * Multiply $\frac{3}{20} \times 240$. * $240 / 20 = 12$. * $3 \times 12 = 36$. 6. **Draft the Explanation (Method 3: Breaking it down - often easier for step-by-step):** * Find 10% of 240. * Find 5% of 240 (which is half of 10%). * Add them together. * 10% of 240 = 24. * 5% of 240 = 12. * $24 + 12 = 36$. 7. **Select the Best Presentation:** Method 3 is very intuitive for "step-by-step" requests as it shows logical reasoning without heavy multiplication. However, the standard decimal multiplication (Method 1) is the most formal. I will present the standard method clearly, perhaps mentioning the breakdown method as a check or alternative way to understand it. Let's stick to the standard multiplication method for clarity but explain the conversion clearly. Actually, providing the breakdown method is often more helpful for understanding *why*. I'll provide the standard calculation method as the primary steps. * Step 1: Convert percentage to decimal. * Step 2: Multiply. * Step 3: State answer. Let's refine the calculation for the final output. $15\% = 0.15$ $0.15 \times 240$ $= \frac{15}{100} \times 240$ $= 15 \times 2.4$ $= 36$ Or: $10\% = 24$ $5\% = 12$ $15\% = 36$ I will write out the solution using the decimal/fraction multiplication method as it's universally applicable, but I'll make the steps very clear. 8. **Final Review:** Does this answer the prompt accurately and step-by-step? Yes. * Step 1: Convert 15% to a decimal. * Step 2: Multiply by 240. * Step 3: Calculate. Let's write it out. Step 1: Convert the percentage to a decimal. $15\% = 0.15$ Step 2: Multiply the decimal by the number. $0.15 \times 240$ Step 3: Perform the multiplication. $15 \times 240 = 3600$ Adjust decimal (2 places): $36.00$ Result: 36. Wait, $0.15 \times 240$. $15 \times 24 = 360$. One zero from 240 cancels one decimal place? $0.15 \times 240 = 15 \times 2.4 = 36$. Or $\frac{15}{100} \times 240 = \frac{15 \times 240}{100} = \frac{3600}{100} = 36$. This looks correct. I will also include the "10% + 5%" method as an alternative way to think about it, as it's often easier for mental math. Plan: 1. Define the formula. 2. Convert percent to decimal/fraction. 3. Multiply. 4. Provide the alternative mental math method for completeness. 5. State final answer clearly. Actually, keeping it simple is better. Just one clear path. I'll use the fraction/decimal multiplication path. Step 1: Convert 15% to a decimal. Step 2: Multiply 240 by 0.15. Step 3: Final result. Let's go.cw =============== Content ================= Here is the step-by-step solution to find 15% of 240: **Step 1: Convert the percentage to a decimal.** To convert a percentage to a decimal, divide by 100. $$15\% = \frac{15}{100} = 0.15$$ **Step 2: Multiply the decimal by the number.** Now, multiply 0.15 by 240. $$0.15 \times 240$$ **Step 3: Calculate the result.** You can think of this as: $$15 \times 240 = 3600$$ Since there are two decimal places in 0.15, move the decimal point in the result two places to the left: $$3600 \rightarrow 36.00$$ **Alternative Method (Mental Math):** * Find 10% of 240: $240 \div 10 = 24$ * Find 5% of 240 (half of 10%): $24 \div 2 = 12$ * Add them together (10% + 5% = 15%): $24 + 12 = 36$ **Answer:** 15% of 240 is **36**. ``` **Example 2: Instruct Mode (Thinking Off)** To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Disable thinking mode via chat_template_kwargs response = client.chat.completions.create( model="Qwen/Qwen3.5-397B-A17B", messages=[ {"role": "user", "content": "What is 15% of 240?"} ], extra_body={"chat_template_kwargs": {"enable_thinking": False}}, max_tokens=2048, stream=True ) # In Instruct mode, the model responds directly without reasoning_content for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} To find 15% of 240, you can follow these steps: ### Step-by-Step Deduction 1. **Convert the percentage to a decimal **: To convert a percentage to a decimal, divide by 100. $$15\% = \frac{15}{100} = 0.15$$ 2. **Multiply the decimal by the number**: Multiply $0.15$ by $240$. $$0.15 \times 240$$ *Alternative Method (Mental Math)*: - Find 10% of 240: $240 \times 0.10 = 24$ - Find 5% of 240 (which is half of 10%): $24 / 2 = 12$ - Add them together ($10\% + 5\% = 15\%$): $24 + 12 = 36$ 3. **Calculation**: $$240 \times 0.15 = 36$$ ### Final Conclusion 15% of 240 is **36**. ``` #### 4.3.2 Tool Calling Qwen3.5 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`. **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="Qwen/Qwen3.5-397B-A17B", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I have access to a get_weather function that can provide current weather information for a location. Let me check the parameters: - location (required): "Beijing" - this is provided by the user - unit (optional): The user didn't specify a temperature unit, so I won't include this optional parameter I should call the get_weather function with Beijing as the location. =============== Content ================= Tool Call: get_weather Arguments: Tool Call: None Arguments: { Tool Call: None Arguments: "location": "Beijing" Tool Call: None Arguments: } ``` #### 4.3.3 Semantic Decisions Qwen3.5-35B-A3B can answer typed questions about an input with a probability for every option through `/v1/decisions`, without generating text. The server turns thinking off for these requests, so the answer is read after a closed think block, and the same server keeps serving chat traffic. No launch flag is needed. Until a release contains `/v1/decisions`, install a [nightly build](/docs/get-started/install#nightly-builds). With a server started by `sglang serve --model-path Qwen/Qwen3.5-35B-A3B --host 127.0.0.1 --port 30000`: ```python Example theme={null} import requests response = requests.post( "http://127.0.0.1:30000/v1/decisions", json={ "input": "I've been trying to connect my Stripe account for 3 days and the integration keeps failing.", "questions": [ { "id": "team", "type": "choice", "question": "Which team should handle this ticket?", "options": [{"name": "billing"}, {"name": "technical"}, {"name": "sales"}], }, {"id": "urgent", "type": "yes_no", "question": "The customer needs an answer today."}, ], }, timeout=60, ) response.raise_for_status() answers = response.json()["answers"] print(answers["team"]["choice"], answers["team"]["probabilities"]) print(answers["urgent"]["probabilities"]["yes"]) ``` See [Decision models](/docs/supported-models/decision_models) for the request reference, how the probabilities are computed, and the limits. ## 5. Benchmark ### 5.1 Accuracy Benchmark #### 5.1.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` Historical results below use the removed harness and are not measurements of the sgl-eval command above. * Test Result ```text Output theme={null} 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:31<00:00, 6.43it/s] Accuracy: 0.975 Invalid: 0.005 Latency: 31.784 s Output throughput: 998.166 token/s ``` #### 5.1.2 Full GSM8K with sgl-eval ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://localhost:30000/v1 \ --model Qwen/Qwen3.5-397B-A17B \ --num-examples 1319 --num-threads 256 --max-tokens 2048 --thinking ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. #### 5.1.3 MMMU Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/mmmu/bench_sglang.py --concurrency 128 --port 30000 --max-new-tokens 512 ``` * Test Result ```text Output theme={null} {'Accounting': {'acc': 1.0, 'num': 3}, 'Agriculture': {'acc': 1.0, 'num': 4}, 'Art': {'acc': 1.0, 'num': 9}, 'Art_Theory': {'acc': 1.0, 'num': 5}, 'Basic_Medical_Science': {'acc': 1.0, 'num': 2}, 'Biology': {'acc': 1.0, 'num': 1}, 'Chemistry': {'acc': 1.0, 'num': 1}, 'Computer_Science': {'acc': 1.0, 'num': 1}, 'Design': {'acc': 0.909, 'num': 11}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 1.0, 'num': 1}, 'Economics': {'acc': 1.0, 'num': 5}, 'Finance': {'acc': 1.0, 'num': 2}, 'Geography': {'acc': 1.0, 'num': 3}, 'History': {'acc': 1.0, 'num': 3}, 'Literature': {'acc': 0.938, 'num': 16}, 'Manage': {'acc': 1.0, 'num': 2}, 'Marketing': {'acc': 1.0, 'num': 5}, 'Math': {'acc': 1.0, 'num': 1}, 'Overall': {'acc': 0.978, 'num': 91}, 'Overall-Art and Design': {'acc': 0.96, 'num': 25}, 'Overall-Business': {'acc': 1.0, 'num': 17}, 'Overall-Health and Medicine': {'acc': 1.0, 'num': 7}, 'Overall-Humanities and Social Science': {'acc': 0.966, 'num': 29}, 'Overall-Science': {'acc': 1.0, 'num': 8}, 'Overall-Tech and Engineering': {'acc': 1.0, 'num': 5}, 'Pharmacy': {'acc': 1.0, 'num': 2}, 'Physics': {'acc': 1.0, 'num': 2}, 'Psychology': {'acc': 1.0, 'num': 4}, 'Public_Health': {'acc': 1.0, 'num': 2}, 'Sociology': {'acc': 1.0, 'num': 6}} eval out saved to ./val_sglang.json Overall accuracy: 0.978 ``` ### 5.2 Speed Benchmark **Test Environment:** * Hardware: H200 (8x) * Model: Qwen3.5-397B-A17B * Tensor Parallelism: 8 * SGLang Version: main branch Server Launch Command: ```bash Command theme={null} SGLANG_USE_CUDA_IPC_TRANSPORT=1 python -m sglang.launch_server \ --model Qwen/Qwen3.5-397B-A17B \ --tp 8 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --speculative-algo NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --mem-fraction-static 0.8 \ --host 0.0.0.0 \ --port 30000 ``` #### 5.2.1 Latency Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3.5-397B-A17B \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 18.94 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4211 Request throughput (req/s): 0.53 Input token throughput (tok/s): 322.16 Output token throughput (tok/s): 222.84 Peak output token throughput (tok/s): 289.00 Peak concurrent requests: 3 Total token throughput (tok/s): 545.00 Concurrency: 1.00 Accept length: 3.12 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1892.35 Median E2E Latency (ms): 1410.85 P90 E2E Latency (ms): 3749.34 P99 E2E Latency (ms): 4216.52 ---------------Time to First Token---------------- Mean TTFT (ms): 190.40 Median TTFT (ms): 208.46 P99 TTFT (ms): 261.27 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 3.96 Median TPOT (ms): 3.79 P99 TPOT (ms): 4.96 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.04 Median ITL (ms): 3.15 P95 ITL (ms): 6.65 P99 ITL (ms): 12.60 Max ITL (ms): 58.03 ================================================== ``` #### 5.2.2 Throughput Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3.5-397B-A17B \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 1000 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 283.04 Total input tokens: 502493 Total input text tokens: 502493 Total generated tokens: 500251 Total generated tokens (retokenized): 498222 Request throughput (req/s): 3.53 Input token throughput (tok/s): 1775.37 Output token throughput (tok/s): 1767.45 Peak output token throughput (tok/s): 3630.00 Peak concurrent requests: 108 Total token throughput (tok/s): 3542.82 Concurrency: 96.71 Accept length: 3.31 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 27372.05 Median E2E Latency (ms): 26660.21 P90 E2E Latency (ms): 39951.91 P99 E2E Latency (ms): 48405.51 ---------------Time to First Token---------------- Mean TTFT (ms): 14247.21 Median TTFT (ms): 14932.44 P99 TTFT (ms): 20998.45 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 26.16 Median TPOT (ms): 26.13 P99 TPOT (ms): 41.33 ---------------Inter-Token Latency---------------- Mean ITL (ms): 26.29 Median ITL (ms): 11.38 P95 ITL (ms): 72.10 P99 ITL (ms): 149.57 Max ITL (ms): 1220.68 ================================================== ``` ### 5.3 Vision Speed Benchmark We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. Each request has 128 input tokens, two 720p images, and 1024 output tokens. #### 5.3.1 Latency Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3.5-397B-A17B \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} TODO ``` #### 5.3.2 Throughput Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3.5-397B-A17B \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} TODO ``` # Qwen3.6 Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.6 ## 1. Model Introduction The Qwen3.6 series is developed by Alibaba. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, delivering substantial upgrades in agentic coding and thinking preservation. Two size/sparsity variants are released: * [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) — **Sparse MoE** (35B total, 3B active) on a Gated Delta Networks backbone. * [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) — **Dense** hybrid GDN; smaller weights footprint, single-GPU friendly. Both variants share the same hybrid reasoning, tool-calling, and multimodal interface and natively handle context lengths of up to 262,144 tokens, extensible to over 1M tokens. **Key Features:** * **Agentic Coding**: Handles frontend workflows and repository-level reasoning with greater fluency and precision * **Thinking Preservation**: New option to retain reasoning context from historical messages, streamlining iterative development * **Efficient Hybrid Architecture**: Gated Delta Networks backbone; sparse MoE (35B / 3B active) or dense 27B variant * **Hybrid Reasoning**: Thinking mode enabled by default with step-by-step reasoning, can be disabled for direct responses * **Tool Calling**: Built-in tool calling support with `qwen3_coder` parser * **Multi-Token Prediction (MTP)**: Speculative decoding support for lower latency; both MoE and Dense variants ship `mtp.safetensors` * **Multimodal**: Unified vision-language model supporting text, image, and video inputs **Available Models:**
Model Architecture Weights
Qwen3.6-35B-A3B (BF16) MoE 35B / 3B active [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)
Qwen3.6-35B-A3B (FP8) MoE 35B / 3B active [Qwen/Qwen3.6-35B-A3B-FP8](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8)
Qwen3.6-35B-A3B (NVFP4) MoE 35B / 3B active (Blackwell) [nvidia/Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4)
Qwen3.6-27B (BF16) Dense 27B [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B)
Qwen3.6-27B (FP8) Dense 27B [Qwen/Qwen3.6-27B-FP8](https://huggingface.co/Qwen/Qwen3.6-27B-FP8)
Qwen3.6-27B (NVFP4) Dense 27B (Blackwell) [nvidia/Qwen3.6-27B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4)
**License:** Apache 2.0 ## 2. SGLang Installation SGLang `>=0.5.10` is required for Qwen3.6. You can install from PyPI, from source, or use a Docker image: ```bash Command theme={null} # Install from PyPI uv pip install --prerelease=allow sglang # Or install from source uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' # Or use Docker (NVIDIA GPUs; also serves the NVFP4 variants) docker pull lmsysorg/sglang:latest ``` For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and capabilities. ### 3.2 Configuration Tips * Speculative decoding (MTP) can significantly reduce latency for interactive use cases. * **Mamba Radix Cache**: Qwen3.6's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. * The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload. * Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities. * **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`. * **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200/B300. * For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors. * Hardware requirements: * **35B-A3B BF16**: \~70GB for weights. TP=1 fits on all supported hardware. * **35B-A3B FP8**: \~35GB for weights. TP=1 fits on all supported hardware. * **35B-A3B NVFP4**: \~23GB for weights. TP=1 fits on B200/B300. * **27B BF16**: \~54GB for weights. TP=1 fits on all supported hardware. * **27B FP8**: \~27GB for weights. TP=1 fits on all supported hardware. * **27B NVFP4**: \~22GB for weights. TP=1 fits on B200/B300. All Qwen3.6 variants (MoE 35B-A3B and Dense 27B) fit on a single supported GPU. NVFP4 is available on B200/B300:
Hardware Memory BF16 TP FP8 TP NVFP4 TP
H100 80GB 1 1 —
H200 141GB 1 1 —
B200 183GB 1 1 1
B300 275GB 1 1 1
* **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation Deploy Qwen3.6 with the following command (H200, all features enabled). Swap `--model-path` to `Qwen/Qwen3.6-27B-FP8` for the dense 27B variant — all other flags carry over: ```shell Command theme={null} sglang serve \ --model-path Qwen/Qwen3.6-35B-A3B-FP8 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --mem-fraction-static 0.8 \ --host 0.0.0.0 \ --port 30000 ``` ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Vision Input Qwen3.6 supports image and video inputs as a unified vision-language model. **Image Input Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B-FP8", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/CI_Demo/mathv-1327.jpg" } }, { "type": "text", "text": "Describe this image in detail." } ] } ], max_tokens=2048, stream=True ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Video Input Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B-FP8", messages=[ { "role": "user", "content": [ { "type": "video_url", "video_url": { "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/video/N1cdUjctpG8.mp4" } }, { "type": "text", "text": "Describe what happens in this video." } ] } ], max_tokens=2048, stream=True ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` ### 4.3 Advanced Usage #### 4.3.1 Reasoning Parser Qwen3.6 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response. To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time: * **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed. * **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process. **Example 1: Thinking Mode (Default)** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B-FP8", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True ) has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Example 2: Instruct Mode (Thinking Off)** To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B-FP8", messages=[ {"role": "user", "content": "What is 15% of 240?"} ], extra_body={"chat_template_kwargs": {"enable_thinking": False}}, max_tokens=2048, stream=True ) for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` #### 4.3.2 Thinking Preservation Qwen3.6 has been trained to preserve and leverage thinking traces from historical messages. Enable this for agent scenarios where maintaining full reasoning context improves decision consistency: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B-FP8", messages=[ {"role": "user", "content": "Help me plan a web app architecture."} ], extra_body={"chat_template_kwargs": {"preserve_thinking": True}}, max_tokens=2048, stream=True ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` #### 4.3.3 Tool Calling Qwen3.6 supports tool calling capabilities. Enable the tool call parser during deployment. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] response = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B-FP8", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if hasattr(delta, 'tool_calls') and delta.tool_calls: if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") if delta.content: print(delta.content, end="", flush=True) print() ``` # Qwen3.8 Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8 Deploy Qwen3.8 with SGLang — day-0 recipes for Qwen's 2.4T-parameter (95B active) hybrid GDN/GQA Mixture-of-Experts model on NVIDIA and AMD. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. **NVIDIA GPUs** (H200 / B200 / B300 / GB300) — the launch image, since this is a day-0 model with no release cut yet: ```bash Command theme={null} docker pull lmsysorg/sglang:qwen38 ``` **AMD GPUs** — pinned `v0.5.17` builds. The two are **not** interchangeable: they target different GPU architectures *and* different ROCm versions, so pick the one matching your hardware or AITER's kernels won't load. MI350X / MI355X (CDNA4, gfx950 — ROCm 7.20): ```bash Command theme={null} docker pull lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260812 ``` MI300X (CDNA3, gfx942 — ROCm 7.00): ```bash Command theme={null} docker pull lmsysorg/sglang-rocm:v0.5.17-rocm700-mi30x-20260813 ``` For how to launch either image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with whatever the command generator below produces. Pick your hardware + quantization to generate the launch command. Three of the strategies are operating points on the throughput/latency curve; the fourth swaps the speculative decoder: * **Low Latency** — fastest reply for a single user. Pick for chat. * **Balanced** — good speed with several users at once. Use for typical multi-user serving. * **High Throughput** — most tokens per second across many users. Best for batch jobs. * **DSpark** — the trained [DSpark draft model](#3-3-dspark-and-replayssm-speculative-decoding) instead of the checkpoint's built-in MTP head. Only offered where DSpark's constraints allow it (see [3.3](#3-3-dspark-and-replayssm-speculative-decoding)). ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The knobs come in two flavors: * **Built-in SGLang features** — TP / DP-Attention, MoE backend + EP (including WideEP), reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers. * **Qwen3.8 specific features** — the **DSpark** speculative-decoding preset and its **ReplaySSM** opt-out, plus the GDN radix-cache-strategy and KV-cache-precision knobs (see [Configuration Tips](#2-configuration-tips) and [3.3 DSpark and ReplaySSM](#3-3-dspark-and-replayssm-speculative-decoding) below). ## 1. Model Introduction **Qwen3.8** (`Qwen3.8-2.4T-A95B`) is Qwen's largest open-weight model to date: **2.4T total parameters, 95B active per token**. It continues the hybrid-attention design of the Qwen3.5 / Qwen3.6 series, scaled up to 92 layers. * **Hybrid Attention** — 23 repeats of `3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE)`, so 69 linear-attention layers to 23 full-attention ones. Gated Attention runs 64 query heads over 4 KV heads at head dimension 256. This balances linear computational complexity against long-context modeling quality. * **GDN (Gated Delta Network)** — the linear-attention layers pair a State Space Model with causal convolution (CausalConv1d), 128 V heads and 16 QK heads at head dimension 128. A fixed-size recurrent state replaces the growing KV cache, so memory is `O(1)` per layer while compute stays `O(N)`. * **Sparse Mixture-of-Experts** — 512 experts, 10 routed plus 1 shared active per token, expert intermediate dimension 2048. Hidden dimension 8192, vocabulary 248,320. * **MTP** — the checkpoint ships multi-token-prediction weights trained with multiple steps. That is what the NEXTN speculative recipes on this page decode against. **License:** [Qwen3.8-2.4T-A95B](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B/blob/main/LICENSE). **Context length:** 262,144 native, extensible to 1,010,000 tokens. **Recommended generation:** `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=0.0`, `repetition_penalty=1.0`. Raising `presence_penalty` toward 2 curbs runaway repetition at some risk of language mixing. For agentic work Qwen suggests allowing 262,144 tokens of reasoning and 131,072 for the final response. **Resources:** Each precision is its own repo — [BF16](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B) · [FP8](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8) · [NVFP4, NVIDIA Blackwell (RadixArk)](https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-NVFP4) · [MXFP4, AMD CDNA4 (Qwen)](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8-MXFP4). Speculative-decoding draft model: [`RadixArk/Qwen3.8-2.4T-A95B-DSpark`](https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-DSpark) (see [3.3](#3-3-dspark-and-replayssm-speculative-decoding)). ## 2. Configuration Tips Four cells are marked **Not Verified** — **GB300 BF16** and the three GB300 **DSpark** strategies. They have launch recipes but no completed validation run; every other cell on the page has been run, including B300 NVFP4 DSpark. **Weight size decides the topology.** At 2.4T parameters BF16 is ≈4.8TB, FP8 ≈2.4TB, NVFP4 ≈1.2TB. FP8 fits no single node here — not even B300, whose 8 × 288GB = 2.30TB misses by a hair — so every FP8 recipe is multi-node. Single-node means FP4: NVFP4 on B300, MXFP4 on MI355X/MI350X. B200 NVFP4 would fit one node but pipelines two, because \~25GB per GPU after weights is too little to serve against. BF16 does not fit 16 GPUs either, so its one recipe is TP32 across 8 GB300 nodes — the only platform where a flat TP32 stays on rack-scale NVLink. **Two distinct FP4 checkpoints.** [NVFP4](https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-NVFP4) is Blackwell-only; [MXFP4](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8-MXFP4) is MI350X/MI355X-only and hybrid (MXFP4 experts, FP8 attention/dense). MI300X is CDNA3 with no hardware MX matmul, so it serves FP8. Leave `--moe-runner-backend` unset on both and the runner resolves from the checkpoint's own `quant_method` — except the NVFP4 wide-EP tier, which pairs `flashinfer_trtllm_routed` with `--moe-a2a-backend flashinfer` by hand because auto cannot resolve that combination. ### GDN state is the scarce resource, not KV Two thirds of the layers are Gated DeltaNet, and their recurrent state lives in its own pool. That pool, not KV, is usually what caps concurrency — and a request's cost depends on the caching strategy:
Strategy State slots per request
--disable-radix-cache 1
no\_buffer 3
extra\_buffer (this model's auto) 5, or 4 where PP disables the overlap scheduler
So `--max-mamba-cache-size` has to match the ratio in force, or it silently clamps `max_running_requests` to a fraction of the target — every cell leaves the pool to `--mamba-full-memory-ratio` except GB300 FP8 Balanced and Low Latency, whose pins are part of tuned capacity sets — Low Latency's `--max-mamba-cache-size 80` is exactly its 16 concurrent requests × 5 slots. And `extra_buffer` needs radix caching on: `mamba_extra_buffer_of()` requires `disable_radix_cache` false, so adding `--disable-radix-cache` makes the strategy inert and drops the budget to one slot. ### NEXTN caps concurrency at 48 MTP weights ship inside the checkpoint, so NEXTN needs no draft model and the 3/1/4 preset fills in automatically. But a speculative cell with no `--max-running-requests` gets **48** from the speculative hook rather than a memory-derived ceiling — pin it explicitly to serve more. `pp_size > 1` rules speculative decoding out entirely in aggregated serving, which is why the H200, B200/B300 FP8, B200 NVFP4 and MI300X recipes carry no MTP. ### Linear-attention backends differ by GPU generation `--mamba-ssm-dtype bfloat16` is load-bearing on SM100: the flashinfer GDN decode default is gated on it, and without it decode silently falls back to Triton. On SM90 the GDN default is Triton for *both* halves, which is why H200 is the one cell pinning `--linear-attn-decode-backend flashinfer` too. The flashinfer GDN prefill default only covers chunk sizes up to 8192, so any cell with a larger `--chunked-prefill-size` must state `--linear-attn-prefill-backend flashinfer` itself. `--attention-backend trtllm_mha` is SM100-only. On Blackwell cells that leave it unset, the model hook picks it together with `--page-size 64` — and returns early when the backend *is* named, so an explicit backend also drops that paired page size and nothing then depends on `--speculative-eagle-topk` to keep the backend off Triton. ### GB300 tiers The FP8 ladder spans the whole curve on 4 nodes × 4 GPUs:
Tier Shape Spec
Low Latency TP16 narrow EP NEXTN 3+1
Balanced DP4×TP4 + EP16 NEXTN 3+1
High Throughput DP4×TP4 + EP16 off
Balanced and High Throughput share one shape and differ only in capacity. Low Latency is the odd one out: narrow EP wins at low concurrency, where spending ranks on expert parallelism costs more than it returns. MTP is off at saturation because draft-plus-verify overhead outweighs the speedup. NVFP4 has only two tiers — its recipes span different GPU counts (8 vs 16) and the wide-EP one holds capacity fixed across its whole concurrency list, so no third operating point exists. Extra build requirements: FP8 Balanced and High Throughput need the **DeepEP v2 wheel** (`2.1.0+01dc3aa`), since their `--moe-a2a-backend deepep_v2` flags do not exist upstream. NVFP4 High Throughput needs `nvfp4_agg_wideep_dep16_flashinfer_setup.sh` run first, and its `SGLANG_FLASHINFER_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8192` is not optional — unset it falls back to 1024 and startup raises once `1024 × ep_size` no longer covers the largest CuteDSL MoE forward. ### GB300 PD disaggregation layouts The PD role selector adds the role and transfer flags to the selected base recipe. It does not resize the prefill and decode workers. Use separate workers with the layouts below for the measured GB300 operating points: | Checkpoint and operating point | Prefill workers | Decode worker | Capacity setting | | - | - | - | - | | FP8, high throughput | 2 × TP1 / PP16 | DP4-attention / TP4 / EP16, DeepEP v2 + EPLB | Keep frontend concurrency above the decode MRR so prefill remains queued | | NVFP4, high throughput | 2 × TP1 / PP6 | DP2-attention / TP4 / EP8, FlashInfer one-sided A2A | Prefill MRR 128 per worker; decode MRR 512; frontend concurrency 1536 | | NVFP4, low latency | TP4 / PP2 | TP16 | Decode MRR 1 and frontend concurrency 1 at the latency endpoint | For NVFP4 with NEXTN, use the 3/1/4 settings on both roles so the prefill worker transfers the draft state. Enable ReplaySSM on the decode role. The generated router command sets the main policy to `round_robin`, which applies to prefill, and keeps the decode policy explicit; the server-side `--load-balance-method` does not configure router worker selection. Round robin is the measured choice for the fixed-shape throughput runs above. For agentic workloads with substantial repeated-prefix reuse, consider cache-aware routing, especially on the prefill side, and validate the cache-locality versus load-balance tradeoff on the target workload. ### AllReduce fusion The four cells running one flat TP group — GB300 FP8 Low Latency, GB300 NVFP4 Low Latency, GB300 BF16, B300 NVFP4 — set `SGLANG_FLASHINFER_MNNVL_CUTEDSL_AR_FUSION`, the Qwen3.5 CuteDSL path whose single workspace fuses AllReduce + Residual + RMSNorm with the MoE finalize. Worth 6–13% over the legacy path. Don't pass `--flashinfer-allreduce-fusion-backend` alongside it — the env suppresses the flag with a warning. Nothing else can use it: the fusion needs DP-attention off and the built-in TP MoE, so the wide-EP tiers are out, and the pipelined cells put their cross-node traffic on IB rather than NVLink. The three GB300 cells also carry `NCCL_NVLS_ENABLE=1`, because SGLang forces NVLS collectives off when that variable is unset. ### AMD Recommended: **MI355X + MXFP4, single node, TP8**. MI350X emits the identical command (same gfx950, same 288GB, same `mi35x` image); MI300X is CDNA3, takes the `mi30x` image, and needs two nodes for the FP8 weights. * **`--mem-fraction-static` looks aggressive on purpose.** With the aiter backend above 8192 context SGLang multiplies it by **0.85** before allocating, so MI355X's `0.9` lands at ≈0.765 and MI300X's `1.0` at ≈0.85. Don't "fix" these downward. MI300X must stay at 1.0 or the weights stop fitting. * **`--disable-custom-all-reduce` belongs on every MI300X rank** — SGLang resolves it per process, so setting it on one node would leave the two pipeline stages reducing through different code paths. * MI300X runs `--kv-cache-dtype fp8_e4m3` with `--page-size 16`: at 8 × 192GB per node the shape is memory-bound. ### ReplaySSM A GDN layer's recurrent state overwrites itself every token, so speculative verify has to be rewindable. Snapshotting the whole K×V state per draft step costs 64 KiB per request, layer and head at K=V=128, times γ+1 steps — scratch taken out of the same budget as the persistent state pool. [ReplaySSM](https://tridao.me/blog/2026/replayssm/) stores each draft step's raw inputs `Sᵢ = (vᵢ, kᵢ, gᵢ, βᵢ)` instead, a few hundred bytes written by the verify kernel on its way through. Once the sampler fixes the accepted length, one fold kernel replays the accepted prefix from the committed checkpoint and advances it in place. The fold is a verbatim clone of the verify recurrence, so the rebuilt state is bit-identical to the recurrent baseline; draft-step scratch shrinks by roughly two orders of magnitude and is never allocated. Folding on every commit is what lets it compose with radix prefix caching over the mutable GDN state: every `--mamba-track-interval` tokens the state is handed to the radix tree, and under `extra_buffer` it goes to a second slot so the running request keeps mutating its own. It is off by default — see [3.3](#3-3-dspark-and-replayssm-speculative-decoding). ## 3. Advanced Usage The `model` argument in the examples below is the BF16 repo id. Every precision is a **separate repo**, so `model` has to be the checkpoint the server was actually launched with — `…-A95B-FP8`, `…-A95B-NVFP4`, or `…-A95B-FP8-MXFP4`. The Deploy panel's cURL snippet always shows the right id for the cell you have selected. ### 3.1 Reasoning Qwen3.8 **always** reasons — thinking cannot be turned off, and every response opens with a `…` block. The `qwen3` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) splits that block into `reasoning_content`, leaving `content` as the answer alone. Depth is tunable per request with `reasoning_effort` — `xhigh` (the default), `medium`, or `low`. `preserve_thinking` carries reasoning from earlier turns into context and is on by default. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="Qwen/Qwen3.8-2.4T-A95B", messages=[{"role": "user", "content": "What is 15% of 240?"}], reasoning_effort="xhigh", # xhigh (default) | medium | low ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Pending update — a sample transcript will be added here. ``` ### 3.2 Tool Calling Enable the `qwen3_coder` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] resp = client.chat.completions.create( model="Qwen/Qwen3.8-2.4T-A95B", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Pending update — a sample transcript will be added here. ``` ### 3.3 DSpark and ReplaySSM (Speculative Decoding) We trained a **DSpark** draft model for Qwen3.8 with SpecForge. Turn it on with the **DSpark** chip in the **Speculative Decoding** card of the [Playground above](#playground) — it emits: ```bash Command theme={null} --speculative-algorithm DSPARK \ --speculative-draft-model-path RadixArk/Qwen3.8-2.4T-A95B-DSpark ``` **ReplaySSM is a separate opt-in.** `--enable-linear-replayssm-spec` defaults off and DSpark does not turn it on, so add it with the **ReplaySSM (spec)** row in the flag-select list. With it on (see [Configuration Tips](#2-configuration-tips) above for how it works), the verify kernel stores each draft step's raw inputs instead of snapshotting the full K×V GDN state, and a single fold kernel replays the accepted prefix from the last committed checkpoint. It's a pure side channel behind a ring buffer — the verify output is bitwise unchanged — so there's no accuracy tradeoff, only a memory one. **DSpark does not compose with every cell.** `_handle_dspark` rejects the run outright rather than degrading, so check these before turning the chip on: * **`--pp-size` must be 1.** The H200, B200, B300 (FP8) and MI300X cells are all pipelined, so DSpark is unavailable on them. * **With DP-Attention it additionally requires `--enable-dp-lm-head`**, the built-in TP MoE (`--moe-a2a-backend none`), and no context parallel. That rules out the GB300 wide-EP tiers, which run DP attention over DeepEP v2 or FlashInfer A2A. * `--speculative-num-steps` is forced to 1, and an omitted `--speculative-draft-model-path` only works if the target checkpoint bundles the draft weights. The Playground greys the DSpark chip out on the combinations above. The **DSpark** strategy chip in the Deploy panel emits this substitution on the four hw × quantization combinations that clear those constraints: GB300 FP8 (on the low-latency shape — the balanced tier's DeepEP v2 a2a rules DSpark out), GB300 NVFP4, GB300 BF16 and B300 NVFP4. Everything else on the page is either pipelined or wide-EP. Because the draft model needs its own weights and KV, those cells run tighter than their NEXTN counterparts — the validated B300 recipe drops `--mem-fraction-static` to 0.80 and trims `--context-length` to 200000 to buy the room back. ReplaySSM composes with radix prefix caching, overlap scheduling, and PD decode, so DSpark speculative decoding runs alongside the rest of the stack (WideEP, PD disaggregation, HiCache) rather than requiring any of them to be turned off. # Qwen3.8-27B Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-27B Deploy Qwen3.8-27B with SGLang — dense hybrid GDN vision-language model with BF16/FP8/NVFP4 W4A4 checkpoints and in-checkpoint MTP, single-GPU on H200, RTX PRO 6000, RTX 5090 and DGX Spark. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your card + checkpoint precision to generate the launch command. The model runs single-GPU on every supported card — H200, RTX PRO 6000, RTX 5090 and DGX Spark — and ships one operating point. `--mamba-full-memory-ratio` is the one sizing flag that matters for throughput on hybrid GDN models: the default (0.9) over-provisions the KV pool and silently clamps concurrency. Set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator) below; everything else follows the panels, and the computed value is pinned into the command. Every cell above — RTX 5090, RTX PRO 6000 and DGX Spark, across all five checkpoints and every Speculative Decoding / Serving Strategy / SSM dtype combination — is measured on **v0.5.19**. That is 202 cells, each one served and scored on the full 1319-question GSM8K (93.18-95.15%). The serving envelope behind the pins is ISL 8192 / OSL 1024 at concurrency 1; throughput and acceptance-length numbers were not re-taken in that sweep. ### Mamba ratio calculator Hybrid GDN models split post-weight memory into a worst-case-reserved **GDN state pool** (sets the concurrency ceiling) and a paged **attention KV pool**, divided by `--mamba-full-memory-ratio`. Every parameter below except `L` and the target concurrency is read live from the Deploy panel and Playground selection; the balanced value is the per-request cost ratio: ```text Formula theme={null} ratio = (S + D) x state_bytes / (L x kv_bytes_per_token) ``` * `S` — state slots per running request: `extra_buffer=5` (default), `extra_buffer_lazy=4`, `no_buffer=3`, disabled radix cache `=1`. For the two `extra_buffer` strategies, `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` frees one slot, and `extra_buffer` frees one more with the overlap scheduler off; the calculator reads both knobs. * `D` — verify intermediate states under speculative decoding: `--speculative-num-draft-tokens` for EAGLE/MTP (4 at the recommended 3/1/4) and for DFLASH (8, DFlash2's block size); `--speculative-dspark-block-size + 1` for DSPARK, where the block size falls back to the draft checkpoint's `block_size` when the flag is omitted (7 for `RadixArk/Qwen3.8-27B-DSpark`, so `D = 8`); 0 with speculation off or with `--enable-linear-replayssm-spec`, which keeps the verify intermediates on a fixed ring instead of per-request slots. * `state_bytes` — one state slot, from the fixed geometry (48 GDN layers x 48 heads x 128 x 128 at `--mamba-ssm-dtype`, plus bf16 conv state): 153.9 MB at fp32, 78.4 MB at bf16. * `kv_bytes_per_token` — 16 attention layers x GQA 4 x 256 x K+V: 32.8 KB at fp8, 65.5 KB at bf16. * `L` — average total request length in tokens: input + output. `--max-mamba-cache-size = target_concurrency x S` is the equivalent explicit pin and overrides the ratio; the calculator emits it alongside. `D` is not a term here: the engine divides the state pool by `S` alone and sizes the speculative verify buffer separately, so folding `D` into the pin would over-provision the pool. After boot, verify with the `max_running_requests` line in the server log — it should not be capped below your target concurrency. ## Playground The Playground is where you experiment with **SGLang features beyond the recipes above**. The Deploy panel emits this model's documented launch recipes; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction **Qwen3.8-27B** is a dense hybrid Gated Delta Networks (GDN) **vision-language** model: a 27B causal language model paired with a vision encoder, with native image and video understanding alongside text. SGLang serves it through the Qwen3-VL path, so the vision tower is live on the recipes below. The language model is 64 layers, laid out as 16 repeats of *3 × (Gated DeltaNet → FFN)* followed by *1 × (Gated Attention → FFN)* — 48 linear-attention layers to 16 full-attention ones. Gated DeltaNet runs 48 value heads and 16 QK heads at head\_dim 128; Gated Attention is GQA 24/4 at head\_dim 256 with a 64-dim rotary slice. Hidden size is 5120 over a 17,408-dim FFN, and the checkpoint ships an MTP head trained with multiple steps. Context is 262,144 tokens natively, extensible to 1,000,000. The serving-relevant architecture is identical to Qwen3.6-27B. Thinking mode is on by default and can be disabled per request; reasoning depth is tunable with `reasoning_effort`, and `preserve_thinking` retains reasoning context from earlier messages.
Model Quantization Weights
Qwen3.8-27B BF16 Qwen/Qwen3.8-27B
Qwen3.8-27B-FP8 FP8 (blockwise) Qwen/Qwen3.8-27B-FP8
Qwen3.8-27B-NVFP4 (FP4 head) NVFP4 W4A4 + FP8 projections, `lm_head` packed to FP4 RadixArk/Qwen3.8-27B-NVFP4
Qwen3.8-27B-NVFP4 (BF16 head) Same body, `lm_head` left dense in BF16 RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead
Qwen3.8-27B-NVFP4 (NVIDIA) NVIDIA's ModelOpt export of the same W4A4 body, `lm_head` packed to FP4 nvidia/Qwen3.8-27B-NVFP4
The two RadixArk NVFP4 exports differ only in the `lm_head`: one packs it to FP4, the other leaves it dense in BF16. The dense head is \~1.7 GB larger on disk and \~3.2 GB larger at runtime, so it is the harder of the two to fit — every recipe on this page was measured against it, and the FP4-head cells reuse those pins unchanged. NVIDIA's own export is that same W4A4 body with that same FP4 head: identical quantized-layer map (FP8 attention and GDN projections, NVFP4 MLPs), identical tensor set, identical 21.9 GB on disk. On GB300, RTX PRO 6000 and DGX Spark its cells reuse the FP4-head pins unchanged, and both SM12x grids have been re-measured against this export on v0.5.19: all 16 overlay combinations per card serve and score 94.01-95.00% (RTX PRO 6000) and 94.16-95.07% (DGX Spark) on the full 1319-question GSM8K. The RTX 5090 is measured too — all 15 overlay combinations it offers serve and score 93.93-94.92% — and every winning launch command there is identical to the FP4-head export's, which is the strongest form of the claim above. What the 32GB card does need is the draft-model rows pinning their own pools: those recipes pin `--max-running-requests 1`, but nothing caps the pools to match, so the KV pool sizes itself for 127,332 tokens against the 9,216 one 8192-in/1024-out request needs, and the engine's default split then leaves the GDN state pool far short of the slots it needs once the draft model's weights are counted against `--mem-fraction-static`. The DSPARK row therefore pins `--max-total-tokens` and a measured `--mamba-full-memory-ratio`, as do DFLASH2 and MTP on the dense-lm\_head export at float32 state. Those pins override the calculator's live value for the selections that carry them. The no-speculation row needs none of it and runs at the pins shown. The two RadixArk checkpoints declare `kv_cache_quant_algo: FP8`, so SGLang's default `--kv-cache-dtype auto` already puts their KV pool in `fp8_e4m3`. The NVIDIA export ships no `kv_cache_scheme`, so `auto` would leave its pool in BF16 instead. Every recipe on this page pins `--kv-cache-dtype fp8_e4m3` explicitly, so all three run the same `fp8_e4m3` pool regardless; the difference only shows up if you switch the Playground's **KV Cache Precision** row back to Auto. ## 2. Configuration Tips * **SM120/SM121 (RTX PRO 6000 Blackwell, RTX 5090, DGX Spark)**: use `--attention-backend flashinfer`; `trtllm_mha` is SM100-only. MTP with the FlashInfer backend requires a FlashInfer build whose prefill `plan` accepts `uniform_q_len` (newer than 0.6.15.post1); otherwise run spec with `--attention-backend triton`. On DGX Spark the 128GB is unified memory shared with the host CPU, so all three checkpoints fit, and its cells reuse the RTX PRO 6000 recipe at `--mem-fraction-static 0.80` rather than a separate operating point. The one lower pin is the unified pool pricing the host's memory too: 0.85 of 128GB leaves \~8GB for the OS — exactly DGX OS earlyoom's SIGTERM threshold — and the first long prefill or boot-time graph capture dips under it and gets the scheduler killed with `exit code -15` and no traceback (`journalctl -u earlyoom` shows the kill). At 0.85, 15 of the 48 cells were killed that way, and which cells is margin noise; at 0.80 every cell served on every attempt. **Validated on SM121 / aarch64**: all 80 configurations (5 checkpoints x Speculative Decoding x Serving Strategy x Mamba SSM Dtype, DFLASH2 included) served on GB10 on `v0.5.19` at ISL 8192 / OSL 1024, concurrency 1, and each scored the full 1319-question GSM8K (93.18-95.15%); the float32 and bfloat16 halves ran on two separate GB10 boxes. No throughput or acceptance-length numbers were re-taken. The sweep exercises the FlashInfer `plan` / `uniform_q_len` path above, which raised no arity error on that build. Three host quirks when reproducing on GB10: docker GPU access is CDI-only (`--device nvidia.com/gpu=all`, as no `nvidia` runtime is registered); `nvidia-smi` reports `Not Supported` for memory because it is unified with the CPU — gate a relaunch on `MemAvailable` in `/proc/meminfo` instead; and the BF16 checkpoint takes \~6.5 minutes just to load its 18 shards from NVMe, so budget \~10 minutes to READY before calling a boot hung. * **H200 (SM90)**: BF16 and FP8 only — the card has no FP4 tensor cores, so an NVFP4 checkpoint's MLP would fall back to the Marlin W4A16 weight-only path, and all three NVFP4 cells are greyed out. The H200 recipes use 32768-token prefill chunks (SM90 prefill is fast enough that a big chunk barely stalls decode, unlike the SM120 guidance below), and the FlashInfer GDN prefill backend engages by default under them. `--attention-backend fa3` is a valid alternative, measured slightly faster at bs=1. * **MTP**: `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4` uses the in-checkpoint MTP head. (This recipe was originally documented with `NEXTN`, an alias of `EAGLE` — same algorithm.) * **DSpark**: the trained draft model is a separate checkpoint — add `--speculative-algorithm DSPARK --speculative-draft-model-path RadixArk/Qwen3.8-27B-DSpark` (the Playground's Speculative Decoding card emits this pair). DSpark does **not** take `--speculative-num-draft-tokens`: its verify window is `--speculative-dspark-block-size` (gamma) **+ 1**, and gamma is auto-inferred from the draft checkpoint when the flag is omitted (7 for this checkpoint, so D = 8). That `D` is a term in the balanced ratio — `r = (S + D) x token_equiv / L`, where `token_equiv` is the state slot expressed in KV tokens, `state_bytes / kv_bytes_per_token` (4698 at fp32 state / 2394 at bf16, over fp8 KV) — so DSpark needs a materially higher `--mamba-full-memory-ratio` than no-spec at the same `S`, and pinning a different gamma changes the ratio with it. MTP is the opposite case: with `--enable-linear-replayssm-spec` its draft intermediates move onto a fixed ring, so `D = 0` and the ratio returns to the no-spec value. The [calculator](#mamba-ratio-calculator) applies both rules. * **DFlash2**: a trained block-diffusion draft in a separate checkpoint — add `--speculative-algorithm DFLASH --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 --speculative-num-draft-tokens 8` (8 is the draft's block size, and it is the `D` term in the ratio, same value as DSpark's). Runs on Ascend NPUs as well ([#35629](https://github.com/sgl-project/sglang/pull/35629)): the selector verify falls back to argmax there, matching what EAGLE and first-generation DFlash drafts (such as `z-lab/Qwen3-8B-DFlash-b16`) already do on NPU. Therefore, NPU currently guarantees lossless verification only for greedy requests; use `temperature=0` and `top_k=1`. Non-greedy requests log a warning; both draft proposal and target verification fall back to greedy, so their requested sampling distribution is not preserved. The selector projects candidates through the target `lm_head`, including quantized heads, so it runs on the NVFP4 checkpoint (whose head is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head). The Ascend comparison in #35629 used an A3 Series device with BF16 target weights, `--tp-size 2 --attention-backend ascend --mamba-ssm-dtype bfloat16 --mamba-radix-cache-strategy extra_buffer`, and disabled RadixCache for both baseline and DFlash2 to exclude cache warm-up and prefix reuse. The DFlash2 run added the three flags shown above. That comparison's accuracy used zero-shot GSM8K with greedy sampling, `max_new_tokens=2048`, 128 examples, and concurrency levels 1, 2, 4, 8 and 16 — a different protocol from this page's own sweep below. Validation: every SM12x cell on this page is measured end to end on v0.5.19 — 202 cells over the five checkpoints, four speculative options, two serving tiers and two GDN state dtypes, full 1319-question GSM8K on each, 93.18-95.15%. The RTX PRO 6000 and DGX Spark recipes need no changes. On the 32GB RTX 5090 the panel applies the measured pins automatically: DFlash2 at `--mem-fraction-static 0.91` with `--chunked-prefill-size 1024` — at 0.91 the pools fit but a 2048-token chunk's activations do not — DSpark at 0.88 (bfloat16), 0.91 (float32) and 0.92 on the dense-lm\_head export, all three with their pools pinned and the last two also cutting the prefill chunk to 1024 and 512, EAGLE at 0.93 (bfloat16) and 0.94 (float32), and no-speculation at 0.90. Whether float32 is available with a draft model depends on the `lm_head`: on the BF16-head export it is greyed out for both DSpark and DFlash2, since the dense head's \~3.2 GB leave no fp32 state pool that also clears prefill graph capture. The FP4-head export frees that headroom back — DSpark serves at 0.91 and DFlash2 High-Throughput at 0.895 with `--mamba-full-memory-ratio 10` overriding the balanced value — and only DFlash2 Low-Latency stays out of reach, where five fp32 slots and a full request's KV never coexist. bfloat16 remains the faster choice regardless: DFlash2 posts 4.92 ms median TPOT at an accept length of 4.29, the best result on this card. * **Hardware fit**: FP8 weights \~28.5GB (not serviceable beyond bs≤2 on 32GB cards); NVFP4 weights \~16.5GB (recommended for RTX 5090-class GPUs). * `--mamba-radix-cache-strategy extra_buffer_lazy` lowers the state cost per request from 5 slots to 4 at no accuracy cost. On small-VRAM cards (RTX 5090 32GB) the state pool bounds concurrency long before KV does — prefer lowering `S` (lazy strategy, or `--disable-radix-cache` for S=1); the [calculator](#mamba-ratio-calculator) re-derives the ratio for the new `S`. The balanced ratio itself is VRAM-independent. * `--mamba-ssm-dtype`: the GDN state slot is **153.9 MB at `float32`** (the checkpoint's declared precision) and **78.4 MB at `bfloat16`**, so bf16 roughly halves the state pool and hands the difference to KV — measured on an RTX 5090 with no speculation, 97,280 KV tokens at bf16 against 68,588 at fp32. On 32GB cards it also decides whether a config fits at all: EAGLE needs `--mem-fraction-static 0.94` at fp32 but 0.92 at bf16. Speed is **not** a one-way trade — with speculative decoding fp32 sometimes wins (NVFP4 + EAGLE: 152.9 vs 144.5 tok/s/user) and sometimes loses (FP8 + EAGLE: 106.3 vs 116.1); measure both for your quantization. Treat `bfloat16` as an accuracy gate and validate it for your workload. On SM120 both precisions run the Triton linear-attn prefill path — the FlashInfer GDN prefill fast path gates on SM100, where its validated domain is in fact a bf16 state pool — so no dtype forces an extra flag here. One interaction to know: `--enable-linear-replayssm-spec` auto-selects fp32 state when `--mamba-ssm-dtype` is unset, and an explicit non-fp32 value logs a state-drift warning at boot. The SSM dtype row always emits the flag explicitly, so the bf16 + EAGLE cells run with that warning — accounted for in their validation. * `--chunked-prefill-size 2048`: decode steps stall behind each prefill chunk on hybrid GDN models, and 8192-token chunks stall them \~600ms at a time. 2048 keeps decode inter-token latency smooth under mixed load and also improves single-wave TTFT. ## 3. Agent Harnesses Agent harnesses drive the model through the OpenAI-compatible endpoint — or, for Claude Code, through SGLang's Anthropic-compatible one — so any of them works once three things line up. **The parsers ship in the command.** Every recipe above carries `--reasoning-parser qwen3 --tool-call-parser qwen3_coder`, because without them a harness receives tool calls as raw text instead of structured `tool_calls`. The **Parsers** card in the [Playground](#playground) is therefore an opt-out — both chips start on, and turning one off strips its flag. `qwen3_coder` is the right tool-call parser for this checkpoint: its chat template instructs the model to reply with an inner `` / `` block nested in ``, which is exactly what that parser decodes. The Hermes parser (`--tool-call-parser hermes`) reads a *different* payload — bare JSON inside `` — so pointing a Hermes-format harness at this model without switching the flag yields tool calls that never parse. `--reasoning-parser qwen3` matches the template's `enable_thinking` toggle, which defaults to on. **Endpoint and model id.** The base URL is `http://:30000/v1`. The `model` string a harness sends must equal the server's `--model-path` — the OpenAI `/v1/models` name defaults to it — unless you override it with `--served-model-name`, which is usually worth doing to keep harness configs short. SGLang also serves an Anthropic-compatible `/v1/messages`, which is what [§3.3](#3-3-claude-code) uses. It converts each request to the OpenAI shape, hands it to the same chat-serving path, and converts the response back — so the parser flags above apply there identically. **Auth.** `--api-key` is unset by default, so the server accepts unauthenticated requests. Harnesses that insist on a key can send any placeholder; set `--api-key` on the server if the endpoint is reachable beyond localhost. ### 3.1 OpenCode [OpenCode](https://opencode.ai/docs/providers/) reaches a self-hosted endpoint through a provider entry in `opencode.json`. Store the credential first — pick **Other**, give the provider an id, and enter any placeholder when the server has no `--api-key`: ```bash Command theme={null} opencode /connect ``` Then declare the provider in `opencode.json`: ```json Config theme={null} { "$schema": "https://opencode.ai/config.json", "provider": { "sglang": { "npm": "@ai-sdk/openai-compatible", "name": "SGLang (Qwen3.8-27B)", "options": { "baseURL": "http://localhost:30000/v1" }, "models": { "RadixArk/Qwen3.8-27B-NVFP4": { "name": "Qwen3.8-27B NVFP4" } } } } } ``` `npm` selects the transport — `@ai-sdk/openai-compatible` is the one for a plain OpenAI-shaped endpoint. `apiKey` is optional and takes a `"{env:VAR_NAME}"` reference rather than a literal. The `models` keys are the ids sent on the wire, so they must match the served model name. Confirm with `/models`. ### 3.2 Pi [Pi](https://pi.dev/docs/latest/custom-provider) (`@earendil-works/pi-coding-agent`) registers providers from an extension rather than a config file. ```javascript Extension theme={null} pi.registerProvider("sglang", { baseUrl: "http://localhost:30000/v1", api: "openai-completions", apiKey: "$SGLANG_API_KEY", models: [ { id: "RadixArk/Qwen3.8-27B-NVFP4", name: "Qwen3.8-27B", reasoning: true, input: ["text", "image"], cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, contextWindow: 262144, maxTokens: 32768, }, ], }); ``` `api: "openai-completions"` is what selects the OpenAI-compatible transport, and `apiKey` takes a `$ENV_VAR` reference rather than a literal. `contextWindow` is the checkpoint's native 262,144; set `maxTokens` to whatever output cap you want per turn. Confirm registration with `pi --list-models`. ### 3.3 Claude Code Claude Code speaks the Anthropic API, so it points at SGLang's `/v1/messages` rather than the OpenAI endpoint. Anthropic documents that routing Claude Code to non-Claude models through a gateway is **not supported**. The wiring below works because SGLang implements the Anthropic message format, but it sits outside what Claude Code is tested against — expect newer Claude Code features to degrade or fail. `ANTHROPIC_BASE_URL` is the server origin — Claude Code appends `/v1/messages` itself, so leave the `/v1` suffix off: ```bash Command theme={null} export ANTHROPIC_BASE_URL=http://localhost:30000 export ANTHROPIC_AUTH_TOKEN=placeholder ``` The two credential variables travel in different headers: `ANTHROPIC_AUTH_TOKEN` goes out as `Authorization: Bearer`, `ANTHROPIC_API_KEY` as `x-api-key`. Either satisfies a server started without `--api-key`; with `--api-key` set, pick the variable matching the header your server reads. A credential variable also takes precedence over a saved claude.ai login for that session. The same pair can live in a settings file instead, which persists across shells and wins over a shell export: ```json Config theme={null} { "env": { "ANTHROPIC_BASE_URL": "http://localhost:30000", "ANTHROPIC_AUTH_TOKEN": "placeholder" } } ``` Run `/status` in Claude Code to confirm which base URL and credential source the session picked up. ### 3.4 Hermes Agent [Hermes Agent](https://github.com/NousResearch/hermes-agent) (Nous Research, MIT) selects a self-hosted endpoint through its setup wizard or its config file. ```bash Command theme={null} hermes model # choose "Custom endpoint (self-hosted / VLLM / etc.)", then enter the # base URL, an API key (blank for a local server) and the model name ``` Equivalently, in `~/.hermes/config.yaml`: ```yaml Config theme={null} model: default: RadixArk/Qwen3.8-27B-NVFP4 provider: custom base_url: http://localhost:30000/v1 api_key: "" context_length: 262144 ``` For several endpoints at once, declare them under `providers:` and switch with `/model custom:` mid-session: ```yaml Config theme={null} providers: workstation: api: http://localhost:30000/v1 server: api: https://gpu-host.internal:30000/v1 key_env: SGLANG_API_KEY ``` ## 4. Semantic Decisions Qwen3.8-27B can answer typed questions about an input with a probability for every option through `/v1/decisions`, without generating text. The server turns thinking off for these requests, so the answer is read after a closed think block, and the same server keeps serving chat traffic. No launch flag is needed. Until a release contains `/v1/decisions`, install a [nightly build](/docs/get-started/install#nightly-builds). With a server started by `sglang serve --model-path Qwen/Qwen3.8-27B --host 127.0.0.1 --port 30000`: ```python Example theme={null} import requests response = requests.post( "http://127.0.0.1:30000/v1/decisions", json={ "input": "I've been trying to connect my Stripe account for 3 days and the integration keeps failing.", "questions": [ { "id": "team", "type": "choice", "question": "Which team should handle this ticket?", "options": [{"name": "billing"}, {"name": "technical"}, {"name": "sales"}], }, {"id": "urgent", "type": "yes_no", "question": "The customer needs an answer today."}, ], }, timeout=60, ) response.raise_for_status() answers = response.json()["answers"] print(answers["team"]["choice"], answers["team"]["probabilities"]) print(answers["urgent"]["probabilities"]["yes"]) ``` See [Decision models](/docs/supported-models/decision_models) for the request reference, how the probabilities are computed, and the limits. # Qwen3.8-Flash-Next Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next Deploy Qwen3.8-Flash-Next with SGLang — day-0 recipes for Qwen's 176B-parameter (6B active) GDN + QSA hybrid Mixture-of-Experts preview of the Qwen4 architecture, on NVIDIA and AMD. ## Deployment Use an SGLang build that includes Qwen3.8-Flash-Next support (v0.5.20 or later). ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` The deployment panel can render a complete `docker run` command for the selected hardware and options. See [Install SGLang with Docker](/docs/get-started/install#method-3-using-docker) for host setup. Pick your hardware + quantization to generate the launch command. The NVFP4 checkpoint is 126 GiB (78 GiB of experts and dense weights plus a 47.7 GiB FP8 N-gram table), so it does not fit one DGX Spark's 128 GB of unified memory, and `--ple-offload-embedding` does not help there: on GB10 the "offloaded" pinned-host table comes out of the same pool as the GPU weights. The two-node shape is TP=2 across two Sparks over the ConnectX-7 200GbE link, with `--no-ple-offload-embedding` keeping the table GPU-resident and sharded (\~65 GB of weights per node). A single Spark works only with the table file-backed on NVMe (see the Single Spark bullet below). * **Launch order.** Start rank 1 first, then rank 0 within a few seconds. When re-launching, stop both ranks and confirm nothing listens on the rendezvous port before starting again: a new rank 1 attaching to a stale rank 0 store fails with a gloo "Connection reset by peer". * **NCCL.** The cross-node decode CUDA-graph deadlock seen on an earlier DGX Spark stack was on NCCL 2.28.x; both builds these images load have been verified here for TP=2 across two Sparks: `dev-qwen38-next-local` runs its pip NCCL 2.29.7 (a system 2.28.3 is also present but not loaded), `qwen38flashnext` its 2.30.7. Confirm with the startup log line `sglang is using nccl==…`, which reports the library actually loaded. If your Sparks also have a slower management NIC, pin `NCCL_SOCKET_IFNAME` / `GLOO_SOCKET_IFNAME` to the 200GbE interface; with `--dist-init-addr` on the 200GbE address the verified runs picked it without pinning. * **Memory.** Both cells run `--mem-fraction-static 0.85`, which leaves \~8–12 GiB of host memory free per node under load (measured through GSM8K at full concurrency and a 100k-token prefill). Two precautions for long-context work beyond that: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` stops variable-shape chunked-prefill buffers from fragmenting the caching allocator at 200k+ contexts, and a host-side watchdog that kills the server when `MemAvailable` drops below a few GiB, because a unified-memory exhaustion can take the whole box down and needs a power cycle to recover. * **Concurrency.** The hybrid GDN/QSA model reserves mamba state slots per running request (5 with the default `extra_buffer` radix strategy, 4 with `extra_buffer_lazy`), and the scheduler caps `--max-running-requests` to what the mamba pool admits; read the effective value from the startup log, not from `/get_server_info`. The cells pin `--max-mamba-cache-size` to requests × slots (24 × 5 = 120 for low latency, 96 × 4 = 384 for high throughput); raising concurrency further takes memory from the KV pool one-for-one. * **Flags.** The cells are the model card's TP=2 recipe without `--mamba-track-interval 64`: the default of 256 tokens satisfies the constraints (a multiple of the 64-token page, at least the 4 draft tokens) and leaves a \~40% larger KV pool (1.48M tokens at 24 concurrent with MTP, 1.07M at 96 without), at the cost of coarser prefix-cache reuse of the recurrent state. `--trust-remote-code` is not needed; the architecture is native to SGLang. * **Measured.** 100k-token prefill at 2,400–2,840 tok/s; MTP accept length 3.5–3.7 of 4 draft tokens on non-thinking output (lower, \~2.5, on thinking output). * **Single Spark (file-backed PLE table).** One GB10 holds the checkpoint only if the 47.7 GiB FP8 N-gram table leaves memory entirely: with **PLE Offload = On (NVMe file)** (`--ple-offload-embedding --ple-offload-backend file`, from [#37068](https://github.com/sgl-project/sglang/pull/37068), merged into `qwen4-main-squashed`) SGLang creates a sparse 47.7 GiB file under `$SGLANG_CACHE_DIR/ple/` (relocate with `--ple-offload-dir`; put it on the local NVMe and mount that directory into the container), fills it on boot, and the gather kernel reads rows through the host page tables; the table's resident set stays at 0 while serving, with hot pages in page cache (capped at 8 GiB by `SGLANG_QWEN4_PLE_FILE_RSS_BUDGET_GB`). The 78.3 GiB of experts and dense weights stay resident (the server logs \~80 GB after load, which also counts the CUDA context and allocator overhead) and `--mem-fraction-static 0.85` leaves \~12–18 GB for the pools, so concurrency is pinned low: **8 requests with MTP** (40 fp32 mamba slots of \~113 MB each) or **24 without** (96 slots on `extra_buffer_lazy`). Measured on `4ccff141db`: MTP cell 27.5 tok/s single-stream (TPOT 33.6 ms) and 71.7 tok/s output at 8; no-MTP cell 15.9 tok/s single-stream and 83 tok/s output at 24; host memory never below 10 GiB. **Boot-time caveat:** every boot rewrites the whole table through the mapping; on an already-populated file that is a read-modify-write per 4 KiB page with readahead disabled (`MADV_RANDOM`), \~17 MB/s and \~55 minutes; on a fresh sparse file it fills at GB/s and the boot takes \~10 minutes. Until that is fixed upstream, delete the previous `ple_table_*.bin` before each boot. * **NVIDIA export (NVFP4 (NVDA)).** `nvidia/Qwen3.8-Flash-Next-NVFP4` is a ModelOpt MIXED\_PRECISION checkpoint (NVFP4 experts, FP8 N-gram table, FP8 block-scaled MTP experts) and needs the loader from [#38121](https://github.com/sgl-project/sglang/pull/38121), merged into `qwen4-main-squashed` (the branch the Python install path builds and the `lmsysorg/sglang:dev-qwen38-next-local` image ships); the `qwen38flashnext` image predates it and cannot load this export. Do not pass `--quantization` for it (it resolves to `modelopt_mixed`), and pass `--moe-runner-backend flashinfer_cutlass` explicitly: the mixed-precision auto-default picks `flashinfer_trtllm` on GB10, which the NVFP4 MoE method rejects at autotune. Its MTP experts are FP8 block-scaled with 128-wide blocks and cannot be split across two ranks (640 / 2 = 320), so the two-node low-latency cell reads the MTP draft from the RadixArk export, the same trained head kept in BF16 there. On a single Spark (TP=1, file-backed table) nothing is sharded, so the export's own MTP head loads directly. The single-Spark NVDA cells use the same pins as the RDXA ones (8 requests with MTP, 24 without); the smaller fp8 draft leaves a 174k-token KV pool with MTP, against 93k for the RadixArk cell. Measured on the `qwen4-main-squashed` tip `4ccff141db` (#38121 merged) at TP=2: 47.4 tok/s single-stream with MTP, 253 tok/s output at 96 concurrent without, the same as the RadixArk export. The NVFP4 checkpoint fits a single 96 GB RTX PRO 6000 Blackwell (SM120) only with the 47.7 GiB FP8 N-gram table in **CPU pinned memory** (`--ple-offload-embedding`, the forced setting of the PLE Offload row on this hardware): the other 78 GiB of the checkpoint loads onto the card. After the loader's temporaries are collected, 74.7 GiB stays resident without speculative decoding and 81.8 GiB with it (the draft head is 0.5 GiB; the rest is memory the loader still holds), leaving 19.4 GiB and 12.3 GiB of the 94.2 GiB the process can use. `--mem-fraction-static` keeps (1 − fraction) × 94.2 GiB of that as runtime slack and the pools take the rest: 6.6 GiB of slack and 12.8 GiB of pools at 0.93, 3.8 GiB and 8.3 GiB at 0.96. (The 81 GiB "mem usage" in the load log is the delta before that collection.) The host needs ≥ 64 GB of free RAM for the locked table (plus page cache for the checkpoint) and Docker needs `--ulimit memlock=-1`, or the pinned allocation fails. * **Concurrency.** The hybrid model reserves mamba state slots per running request, and the scheduler caps `--max-running-requests` to what the state pool admits: with the default `extra_buffer` strategy (5 fp32 slots per request at 0.109 GiB) that is 3 requests with MTP and 12 without on this card. The cells keep prefix caching on and get to 16 / 64 with three levers: `--mamba-radix-cache-strategy extra_buffer_lazy` (4 slots per request), `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` (3; a running request's prefix state is no longer pinned in the radix tree during decode, so it can be evicted, which trades cache retention, not numerics), and `--mamba-ssm-dtype bfloat16` (0.055 GiB per slot). `--max-mamba-cache-size` is pinned to requests × 3 (48 / 192). The ceilings from the pool arithmetic are \~20 requests with MTP (each request carries 4 intermediate draft states) and \~70 without; turning prefix caching off (`--disable-radix-cache`, 1 slot per request) reaches 24 / 96 on this card and was verified too, but re-prefills every prompt. * **Linear-attention kernels.** Left on auto. On SM120 the server resolves decode, prefill and verify to triton — the same GDN kernels the fp32 runs and the DGX Spark cells use; the bf16-state FlashInfer GDN auto-default applies only to SM100. An explicit `--linear-attn-decode-backend flashinfer` also runs on this card and measured the same TPOT (within 0.3 ms) and accuracy, so there is nothing to gain from pinning either. * **Memory headroom.** With the state pool pinned, the KV pool absorbs the rest of the static budget, so `--mem-fraction-static` is what sets the activation headroom. 4096-token prefill chunks of ShareGPT-length prompts peak 1.5–2.6 GB above the post-graph-capture level, and cells left with 2.4 GB free OOMed in the GDN short-conv during prefill. The cells keep ≥ 4 GB free after graph capture and ≥ 2.3 GB at the measured peak: 0.96 with MTP (78k-token KV pool, \~4.9k per request at 16) and 0.93 without (98k tokens, \~1.5k per request at 64; 0.94 gives 138k tokens with 2.3 GB at peak). `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is set on both. * **Accuracy.** Full GSM8K, 1,319 questions, on the `lmsysorg/sglang:dev-qwen38-next-local` image, in two protocols. Chat completions API with thinking off, greedy, an 8,192-token budget, answer parsed from a final "The answer is N" line (the protocol of the DGX Spark rows, and the figure on the benchmark card): 96.9% for the MTP cell and 96.9% for the no-MTP cell (RadixArk export); 97.3% and 97.0% for the NVIDIA export. `python -m sglang.test.run_eval --eval-name gsm8k --num-examples 1319 --max-tokens 16384` (5-shot, greedy, last-number scorer, chat template with thinking on): 97.72% / 97.79% (RadixArk), 97.41% / 97.72% (NVIDIA). Measured on the `4ccff141db` build, which carries the #36811 and #38290 router fixes. * **Measured (1024-in / 256-out random prompts, `ignore_eos`).** MTP vs no-MTP at 1 request: TTFT 115 vs 116 ms median, TPOT 5.9 vs 11.4 ms, 148 vs 83 tok/s. At 16: TPOT 19.3 vs 25.6 ms, 613 vs 524 tok/s. No-MTP at 64: TPOT 55 ms, 861 tok/s, 3.4 req/s. ShareGPT chat (thinking off, ≤ 512 output tokens): 785 tok/s at 16-way with MTP, 1,258 tok/s at 64-way without. MTP accept length 3.3 of 4 on GSM8K / random prompts, 2.9 on long-form ShareGPT answers. On the `dev-qwen38-next-local` image (`4ccff141db`): 6.0 / 11.44 ms TPOT at 1 request (MTP / no MTP), 685 / 561 tok/s at 16, 909 tok/s at 64 without MTP, accept length 3.16. * **NVIDIA export (NVFP4 (NVDA)).** `nvidia/Qwen3.8-Flash-Next-NVFP4` runs on this card with the same shape, pools and flags as the RadixArk cells on the `lmsysorg/sglang:dev-qwen38-next-local` image (it carries the loader from [#38121](https://github.com/sgl-project/sglang/pull/38121); `qwen38flashnext` cannot load this export). Do not pass `--quantization`: the checkpoint resolves to `modelopt_mixed`. Low latency keeps the in-checkpoint MTP head: at TP=1 its fp8 block-scaled experts need no sharding, and #38121 runs them on triton under the `flashinfer_cutlass` pin. The RadixArk BF16 draft measured the same here (accept 3.33 vs 3.31, TPOT 18.5 vs 19.1 ms at 16), so the cell stays single-checkpoint. Verified on that image (`4ccff141db`): full GSM8K in the Accuracy bullet above; 6.02 ms TPOT at 1 request with MTP, 675 output tok/s at 16; 906 output tok/s at 64 without, the same as the RadixArk export. The smaller fp8 draft leaves a \~170k-token KV pool at 16 concurrent. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## Agentic Long-Context Deployment Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order. * **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks). * **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`. * **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image. ## 1. Model Introduction **Qwen3.8-Flash-Next** is a multimodal Mixture-of-Experts model released as an early preview of the architecture Qwen4 is being built on — the same role Qwen3-Next played for Qwen3.5, whose hybrid Gated DeltaNet + Gated Attention design then carried through the Qwen3.5, Qwen3.6, Qwen3.7 and Qwen3.8 series. Qwen is publishing the architectural changes ahead of the full Qwen4 family so the community can evaluate them independently. It has **176B total parameters — 51B of which is an N-gram embedding table — and 6B active per token**. Against Qwen3.7-Plus it cuts both training and inference cost substantially (training takes roughly 1/9 as much) while holding comparable overall quality. It takes text and images in, and the hosted production variant is served as `qwen3.8-flash` on QwenCloud. The upgrades span four axes: * **Attention — GDN + QSA hybrid.** Three of every four layers use Gated DeltaNet, which compresses history into a fixed-size recurrent state; the fourth is global attention running **Qwen Sparse Attention (QSA)**. A lightweight indexer aggregates the sequence into micro-blocks, scores importance at block level, and selects the relevant regions — so the indexing overhead shrinks along with the attention itself. Unlike approaches that reuse indices across layers, QSA compresses independently per layer, which suits an architecture that interleaves GDN and attention. Qwen measures up to 10.2× prefill and 6.6× decode speedups for the QSA attention kernel at 1M tokens. * **Residual — Gated Residual (GR).** The single residual stream widens into four parallel branches, with an element-wise dynamic gate controlling how much each layer reads from and writes back to each branch. Qwen reports one branch naturally becoming a long-range bus. The gate also suppresses activation outliers, and the residual state can be held in FP8. * **Embedding — N-gram Embedding.** Lookups keyed on the local context (current token plus a few preceding ones) rather than a single token, adding 51B parameters at almost no extra per-token compute. Because lookup addresses are known in advance, the table can live in host memory and be prefetched asynchronously alongside model compute. The final model uses a single such layer near the start of the network. * **Optimization — Muon.** Muon for the genuine 2-D linear maps (attention, GDN and MoE expert weights), AdamW for embeddings, the MoE router and GR's low-rank parameters, with fused QKV / SwiGLU / GDN projections split before orthogonalization. The scaling law was refit for the new architecture, and batch-size warmup was dropped — it cost 18.8% more optimizer steps without improving the result. Carried over from Qwen3-Next and refined through the Qwen3.5–Qwen3.8 series: an ultra-sparse MoE (large expert pool, few routed experts per token plus one shared expert) with global load balancing; a **multi-step-trained MTP module**, whose own full-attention layers are QSA as well, which is what keeps speculative acceptance high in practice; and the training-stability set of zero-centered RMSNorm with weight decay on norm weights, attention output gating, and normalized MoE router initialization. **Context length:** 262,144 native, extensible to 1,000,000 tokens with YaRN. **License:** see [the model card's LICENSE](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE). **Recommended generation:** Qwen has not published sampling recommendations for this preview. SGLang applies the checkpoint's own `generation_config.json`, so leave `temperature` / `top_p` unset unless you have a measured reason not to. Each precision is its own repository:
Precision Repository Where it runs
BF16 Qwen/Qwen3.8-Flash-Next H200, B200, B300, GB300, MI350X, MI355X
FP8 Qwen/Qwen3.8-Flash-Next-FP8 H200, B200, B300, GB300, MI350X, MI355X
NVFP4 (RDXA) RadixArk/Qwen3.8-Flash-Next-NVFP4 B200, B300, GB300, RTX PRO 6000, 1x or 2x DGX Spark (Blackwell only)
NVFP4 (NVDA) nvidia/Qwen3.8-Flash-Next-NVFP4 (ModelOpt MIXED\_PRECISION) RTX PRO 6000, 1x or 2x DGX Spark (needs #38121: the dev-qwen38-next-local image or the Python install path)
**Resources:** [Qwen's announcement](https://qwen.ai/blog?id=qwen3.8-flash-next). ## 2. Advanced Usage The `model` argument in the examples below is the BF16 repo id. Every precision is a **separate repo**, so `model` has to be the checkpoint the server was actually launched with — `…-Flash-Next-FP8` or `…-Flash-Next-NVFP4`. The Deploy panel's cURL snippet always shows the right id for the cell you have selected. ### 2.1 Reasoning Qwen3.8-Flash-Next **always** reasons — thinking cannot be turned off. `--reasoning-parser auto` (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) lets SGLang pick the matching parser from the checkpoint's chat template, and splits the thinking into `reasoning_content`, leaving `content` as the answer alone. The resolved name is logged at startup if you want to pin it explicitly later. Depth is requested with `reasoning_effort`. Qwen documents `xhigh` (the default), `medium` and `low` for the hosted model; SGLang forwards whatever you pass into the checkpoint's chat template. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="Qwen/Qwen3.8-Flash-Next", messages=[{"role": "user", "content": "What is 15% of 240?"}], reasoning_effort="xhigh", # xhigh (default) | medium | low ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Pending update — a sample transcript will be added here once the weights are public. ``` ### 2.2 Tool Calling Add `--tool-call-parser auto` (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. As with the reasoning parser, SGLang resolves the concrete detector from the chat template at startup. No Deploy cell sets it, so this is an opt-in: add the flag to the generated command, or flip the chip in the Playground. Because this model always thinks, the final assistant turn can put text in `reasoning_content` rather than `content` — print both so a bare `None` doesn't mislead you. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] resp = client.chat.completions.create( model="Qwen/Qwen3.8-Flash-Next", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Pending update — a sample transcript will be added here once the weights are public. ``` # Dots3-Note Source: https://docs.sglang.io/cookbook/autoregressive/RedNote/Dots3-Note Deploy RedNote dots3.note with SGLang — a native multimodal omni model (MoE ViT + Whisper-derived audio encoder + native video flattening) on the dots3 hybrid MLA/SWA language model, with DSA and full-sharing MTP speculative decoding. ## Deployment dots3.note support is in [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829). Until that PR is included in a tagged SGLang release, install from a build that contains the PR. ```bash Command theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate git clone https://github.com/sgl-project/sglang.git cd sglang git fetch origin pull/33829/head && git checkout FETCH_HEAD uv pip install --prerelease=allow -e python ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev-dots3-note ``` This image packages SGLang with the dots3.note support from PR #33829 and is the recommended way to deploy until the PR lands in a tagged SGLang release — it saves you from building the branch yourself. For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick the hardware and the checkpoint precision. The recipe runs on a single 8-GPU Hopper node with DP8 attention × TP8 × EP8 and DeepEP as the MoE all-to-all transport. Blackwell is not supported yet. **Precision** — selects the MoE path, not just the weights. The BF16 cells pin `--moe-runner-backend deep_gemm` with BF16 DeepEP dispatch output (JIT DeepGEMM is enabled via `SGLANG_ENABLE_JIT_DEEPGEMM=1`). The FP8 cells leave both at `auto` and let SGLang resolve the runner from the checkpoint's quantization config. **Spec Decode** — NEXTN is on in every cell: 3 draft steps, 4 draft tokens per step, and the draft model path pointing at the target checkpoint itself. dots3's MTP layer is full-sharing — it carries the dots3 sliding-window attention geometry and reuses the target LM head — so no separate draft checkpoint is needed. Target verification and draft extension run on the paged, absorbed SWA-MLA FA3 path. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations signed off on this page; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction dots3.note is RedNote's native multimodal omni model, built on the dots3 language model. It accepts text, image, audio, and native video input. * **Native multimodality** — a custom MoE vision transformer and a Whisper-derived audio encoder run in-process with the language model, loaded from the same checkpoint directory. Image and audio placeholders are expanded by a model-specific processor. * **Native video pipeline** — the server jointly samples and interleaves frames, timestamps, and audio segments under a token budget, reproducing the training-time flattening algorithm. A generic uniform-frame video processor would silently change the modality ordering and token allocation (inference/training mismatch), so the pipeline is vendored into the serving path. * **Hybrid attention** — dots3 combines MLA with full-attention and sliding-window layers of different geometry, attention gates, and optional DSA indexing on full-attention layers. * **MTP speculative decoding** — a full-sharing MTP/NextN architecture exposes one recursively shared, SWA-shaped MTP layer and shares the target LM head. **Available checkpoints:** * **BF16**: [dots-studio/dots3-note-prev](https://huggingface.co/dots-studio/dots3-note-prev) * **FP8**: [dots-studio/dots3-note-prev-fp8](https://huggingface.co/dots-studio/dots3-note-prev-fp8) **Resources:** [Hugging Face (BF16)](https://huggingface.co/dots-studio/dots3-note-prev) · [Hugging Face (FP8)](https://huggingface.co/dots-studio/dots3-note-prev-fp8) · [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829) ## 2. Configuration Tips **Hybrid KV pool.** dots3 mixes full-attention and sliding-window layers, and its MTP draft layer is an ordinary SWA layer — not a full-attention one. SGLang sizes the pool accordingly, with `--swa-full-tokens-ratio 0.03` setting the ratio of SWA-layer KV tokens to full-layer KV tokens (`swa_tokens ≈ full_tokens × ratio`). Lower it when long full-attention contexts dominate and the full pool fills first; raise it when the SWA pool is the bottleneck. **MoE runner.** Leave the runner at the cell default: `deep_gemm` for BF16 checkpoints, `auto` for quantized ones. DeepEP is the all-to-all transport in every cell (`--moe-a2a-backend deepep`, dispatch tokens per rank tuned via `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128`). **Attention backend.** FA3 across the board: prefill, decode, and draft (`--prefill-attention-backend fa3 --decode-attention-backend fa3 --speculative-draft-attention-backend fa3`) with `--page-size 64`. MTP target verification uses FA3's absorbed SWA-MLA fallback, which consumes the same paged latent KV view as decode. **DSA.** DSA indexing on full-attention layers is on by default. To disable it, add `--json-model-override-args '{"index_topk":null}'`. **CUDA graphs.** The cells enable decode-side CUDA graphs only (`--cuda-graph-backend-decode full --cuda-graph-backend-prefill disabled`, max batch size 32) and are sized for GPUs with at least 120 GiB of memory. On smaller GPUs, switch to `--cuda-graph-backend-decode disabled` (and expect `--deepep-mode normal` to be the better fit). **Context length.** `--context-length 524288` is the model's window. Like other SGLang models, it bounds the longest accepted request; it does not size the KV pool. **Language-only mode.** Add `--language-only` to skip constructing the vision and audio towers entirely — the freed memory goes to the language model. This is also the language role of an encoder/LLM-disaggregated (EPD) deployment; see [EPD](#epd-disaggregation) below. ## 3. Advanced Usage ### 3.1 Native video input dots3.note accepts a native `video_url`. The server decodes the remote video in memory and applies the training-consistent flattening pipeline — interleaving timestamps, frames, and audio under a token budget, with a deterministic seed derived from the video and the question. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="dots3.note", messages=[ { "role": "user", "content": [ { "type": "video_url", "video_url": {"url": "https://example.com/sample.mp4"}, }, {"type": "text", "text": "Summarize what happens in this video."}, ], } ], extra_body={ "video_config": { "seq": 131072, "audio_cap": 0.5, "audio_sr": 16000, "k_mode": "eval_ek", }, }, ) print(response.choices[0].message.content) ``` ```text Output theme={null} Pending update... ``` Per-request video preprocessing controls are grouped under `video_config` in `extra_body`:
Field Default Purpose
seq 131072 Total sequence budget used by the video flattener.
audio\_cap 1.0 Maximum fraction of the input budget assigned to audio; 0 disables audio processing.
audio\_sr 16000 Audio sample rate.
k\_mode eval\_ek Deterministic evaluation/sampling mode of the flattener.
These controls are request-scoped so that evaluation jobs with different context budgets can share one server. For example: `extra_body={"video_config": {"seq": 131072, "audio_cap": 0.5}}`. The flattener reserves room for `max_new_tokens` inside the budget and falls back to visual-only processing if audio would exceed the configured token budget. A request may carry several videos, and videos can be mixed with image and audio parts. Each video is flattened independently under the same per-request budget, and the flattened frames and audio segments are spliced back at the position of their `video_url` part, so the modality ordering of the prompt is preserved. ### 3.2 Image and audio input Outside the native-video path, images and audio clips use the standard OpenAI multimodal message format. `--enable-multimodal` is in every cell; the vision and audio towers run in-process, so no extra server is needed. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="dots3.note", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": {"url": "https://example.com/sample.jpg"}, }, {"type": "text", "text": "Describe this image."}, ], } ], ) print(response.choices[0].message.content) ``` ```text Output theme={null} Pending update... ``` ### 3.3 Tool Calling Toggle **Tool Call Parser** (`--tool-call-parser dots`) and **Reasoning Parser** (`--reasoning-parser dots`) in the **Parsers** card of the [Playground above](#playground). Structured tool calls then surface via `message.tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }] resp = client.chat.completions.create( model="dots3.note", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) print(resp.choices[0].message.tool_calls) ``` ```text Output theme={null} Pending update... ``` ### 3.4 Encoder/LLM Disaggregation (EPD) `Dots3NoteForCausalLM` supports both roles of an encoder/LLM-disaggregated deployment: * **Encoder role** — serve with `--encoder-only`; the instance runs only the vision and audio towers. * **Language role** — serve with `--language-only`; the instance skips tower construction, leaving the memory to the language model. See the [EPD guide](../../../docs/advanced_features/epd_disaggregation) for how to wire the roles together. # Step-3.7-Flash (new) Source: https://docs.sglang.io/cookbook/autoregressive/StepFun/Step-3.7-Flash ## 1. Model Introduction [Step-3.7-Flash](https://huggingface.co/stepfun-ai/Step-3.7-Flash) is a 198B-parameter Mixture-of-Experts (MoE) vision-language model that combines a 196B-parameter language backbone with a 1.8B-parameter vision encoder for native image understanding. Engineered for high-frequency production workloads, it activates approximately 11B parameters per token and supports a 256k context window with three selectable reasoning levels (low, medium, and high). The model is available in multiple quantization formats (BF16, FP8, NVFP4). Step-3.7-Flash is built for developers who need to scale agentic workflows that combine perception, search, and reasoning — from parsing massive financial reports in one pass, to running multi-step search loops with cross-source verification, to operating concurrent coding agents in high-throughput pipelines. ## 2. SGLang Installation Step-3.7-Flash is currently available in SGLang via Docker image install. ### Docker (NVIDIA) ```bash Command theme={null} # Pull the docker image docker pull lmsysorg/sglang:latest # Launch the container docker run -it --gpus all \ --shm-size=32g \ --ipc=host \ --network=host \ lmsysorg/sglang:latest bash ``` ## 3. Model Deployment This section provides deployment configurations optimized for different use cases. ### 3.1 Basic Configuration The Step-3.7-Flash series comes in one size with multiple quantization options. Recommended starting configurations vary depending on hardware. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. ### 3.2 Configuration Tips * **Memory**: Requires GPUs with high VRAM capacity. Supported platforms: H200 (4x, TP=4), B200/B300 (4x, TP=4), GB200/GB300 (4x, TP=4). * **NVFP4 Quantization**: NVFP4 provides the smallest memory footprint. Requires `--quantization modelopt_fp4 --kv-cache-dtype fp8_e4m3 --moe-runner-backend flashinfer_trtllm`. * **Trust Remote Code**: All Step-3.7-Flash variants require `--trust-remote-code` due to the custom model architecture. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Multi-Modal Inputs Step-3.7-Flash supports image inputs alongside text. Here's a basic example: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Read all the text in the image." } ] } ] start = time.time() response = client.chat.completions.create( model="stepfun-ai/Step-3.7-Flash", messages=messages, max_tokens=2048, ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Multi-Image Input Example:** Step-3.7-Flash can process multiple images in a single request for comparison or analysis: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg" } }, { "type": "image_url", "image_url": { "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg" } }, { "type": "text", "text": "Compare these two images and describe the differences in 100 words or less." } ] } ] start = time.time() response = client.chat.completions.create( model="stepfun-ai/Step-3.7-Flash", messages=messages, max_tokens=2048, ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` #### 4.2.2 Reasoning Parser Step-3.7-Flash supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} sglang serve \ --model-path stepfun-ai/Step-3.7-Flash \ --tp 4 \ --trust-remote-code \ --reasoning-parser step3p5 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="stepfun-ai/Step-3.7-Flash", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` #### 4.2.3 Tool Calling Step-3.7-Flash supports tool calling capabilities. Enable the tool call parser: **Start sglang server:** ```shell Command theme={null} sglang serve \ --model-path stepfun-ai/Step-3.7-Flash \ --tp 4 \ --trust-remote-code \ --reasoning-parser step3p5 \ --tool-call-parser step3p5 ``` ```python Example theme={null} from openai import OpenAI import json client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # 1. define tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit"} }, "required": ["location"] } } } ] # 2. tool run def get_weather(location, unit="celsius"): return f"The weather in {location} is 22 {unit[0].upper()} and sunny." # 3. send first request print("--- Sending first request ---") response = client.chat.completions.create( model="stepfun-ai/Step-3.7-Flash", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=1.0, stream=False ) message = response.choices[0].message # 4. Handle Reasoning Content reasoning = getattr(message, 'reasoning_content', None) if reasoning: print("=============== Thinking =================") print(reasoning) print("==========================================") # 5. Handle Tool Calls if message.tool_calls: print("\nTool Calls detected:") history_messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, message ] for tool_call in message.tool_calls: print(f" Tool: {tool_call.function.name}") print(f" Args: {tool_call.function.arguments}") args = json.loads(tool_call.function.arguments) tool_result = get_weather(args.get("location"), args.get("unit", "celsius")) history_messages.append({ "role": "tool", "tool_call_id": tool_call.id, "content": tool_result }) print("\n--- Sending tool results ---") final_response = client.chat.completions.create( model="stepfun-ai/Step-3.7-Flash", messages=history_messages, temperature=1.0, stream=False ) print("=============== Final Content =================") print(final_response.choices[0].message.content) else: if message.content: print("=============== Content =================") print(message.content) ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation ## 5. Benchmark *Benchmark results will be added soon.* # Step3-VL-10B Source: https://docs.sglang.io/cookbook/autoregressive/StepFun/Step3-VL-10B ## 1. Model Introduction [Step3-VL-10B](https://huggingface.co/stepfun-ai/Step3-VL-10B) is a lightweight open-source multimodal model developed by StepFun, designed to redefine the trade-off between compact efficiency and frontier-level multimodal intelligence. Despite its compact 10B parameter footprint, Step3-VL-10B excels in visual perception, complex reasoning, and human-centric alignment. Key highlights of Step3-VL-10B include: * **STEM Reasoning**: Achieves 94.43% on AIME 2025 and 75.95% on MathVision (with PaCoRe), demonstrating exceptional complex reasoning capabilities that outperform models 10×–20× larger. * **Visual Perception**: Records 92.05% on MMBench and 80.11% on MMMU, establishing strong general visual understanding and multimodal reasoning. * **GUI & OCR**: Delivers state-of-the-art performance on ScreenSpot-V2 (92.61%), ScreenSpot-Pro (51.55%), and OCRBench (86.75%), optimized for agentic and document understanding tasks. * **Spatial Understanding**: Demonstrates emergent spatial awareness with 66.79% on BLINK and 57.21% on All-Angles-Bench, establishing strong potential for embodied intelligence applications. For more details, please refer to the [Step3-VL-10B model card on Hugging Face](https://huggingface.co/stepfun-ai/Step3-VL-10B). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration Step3-VL-10B is a compact 10B dense model that can run on a single GPU. Recommended starting configurations vary depending on hardware. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and quantization method. SGLang supports serving Step3-VL-10B on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs. ### 3.2 Configuration Tips * **Single GPU Deployment**: Step3-VL-10B fits comfortably on a single GPU with BF16 precision, no tensor parallelism required. * **Memory Management**: Set lower `--context-length` to conserve memory if needed. A value of `32768` is sufficient for most scenarios. * **FP8 Quantization**: Use FP8 quantization to further reduce memory usage while maintaining quality. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Multi-Modal Inputs Step3-VL-10B supports image inputs. Here's a basic example with image input: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Read all the text in the image." } ] } ] start = time.time() response = client.chat.completions.create( model="stepfun-ai/Step3-VL-10B", messages=messages, max_tokens=2048, extra_body={"top_k": -1} ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example output:** ```text Output theme={null} Response costs: 5.89s Generated text: Auntie Anne's CINNAMON SUGAR 1 × 17,000               17,000 SUB TOTAL                    17,000 GRAND TOTAL                 17,000 CASH IDR                    20,000 CHANGE DUE                 3,000 ``` **Multi-Image Input Example:** Step3-VL-10B can process multiple images in a single request for comparison or analysis: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg" } }, { "type": "image_url", "image_url": { "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg" } }, { "type": "text", "text": "Compare these two images and describe the differences in 100 words or less." } ] } ] start = time.time() response = client.chat.completions.create( model="stepfun-ai/Step3-VL-10B", messages=messages, max_tokens=2048, extra_body={"top_k": -1} ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 3.24s Generated text: First image: Single red Hong Kong taxi close - up, clear license plate (RX 5004), “4 SEATS” sticker, urban street with shops behind. Second image: Aerial view of many taxis (red, green) on a highway with a viaduct, some hoods open, dense arrangement. Differences: Scale (single vs many), perspective (close - up vs aerial), context (street shops vs highway), and taxi conditions (normal vs some open hoods). ``` #### 4.2.2 Reasoning Parser Step3-VL-10B supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model stepfun-ai/Step3-VL-10B \ --reasoning-parser deepseek-r1 \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="stepfun-ai/Step3-VL-10B", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True, extra_body={"top_k": -1} ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Example Output:** ```text Output theme={null} =============== Thinking ================= Okay, let's see. The problem is asking for 15% of 240. Hmm, I need to remember how to calculate percentages. So, percentage means "per hundred," right? So, 15% is the same as 15 per 100, or 15/100. To find a percentage of a number, I think you convert the percentage to a decimal and then multiply it by the number. Let me check that. First, 15% as a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Yeah, that's right. So 15% is 0.15 in decimal form. Then, to find 15% of 24 0, I need to multiply 0.15 by 240. Let me do that calculation. Let me write it out: 0.15 * 240. Let's compute that. Maybe break it down. 0.1 is 10%, and 0.05 is 5%, so 10% of 240 is 24, and 5% of 240 is 12. Then 10% + 5% is 15%, so 24 + 12 is 36. Oh, that's a good way to check. So 15% is 10% plus 5%, which adds up to 36. Let me verify with the decimal method. 0.15 * 240. Let's multiply 240 by 0.1 first: 24. Then 240 by 0.05: 12. Adding them gives 36. Yep, that matches. Alternatively, 240 * 15 = 3600, and then divide by 100 (since it's per hundred), so 3600 / 100 = 36. That's another way. So 15% of 240 is 36. Let me make sure I didn't make a mistake. Let's check with another method. 10% of 240 is 24, 20% would be 48, so 15% is halfway between 10% and 20%, which is (24 + 48)/2 = 36. Yep, that works too . So all methods point to 36. I think that's the answer. =============== Content ================= To solve the problem "What is 15% of 240?" step by step: --- ### **Step 1: Understand the concept of percentage** A percentage represents a portion of a whole. Specifically, "percent" means "per hundred." So, 15% means **15 out of 100**, or **15/100**. --- ### **Step 2: Convert the percentage to a decimal** To use percentages in calculations, convert them to decimals by dividing by 100: $$ 15\% = \frac{15}{100} = 0.15 $$ --- ### **Step 3: Multiply the decimal by the given number** Now, multiply 0.15 (the decimal form of 15%) by 240: $$ 0.15 \times 240 = 36 $$ --- ### **Alternative Verification Methods** #### **Method A: Break into parts** - 10% of 240 = $ 0.10 \times 240 = 24 $ - 5% of 240 = $ 0.05 \times 240 = 12 $ - Add them: $ 24 + 12 = 36 $ #### **Method B: Use direct multiplication** - $ 15\% \text{ of } 240 = \frac{15}{100} \times 240 = \frac{3600}{100} = 36 $ #### **Method C: Estimate using known percentages** - 20% of 240 = $ 0.20 \times 240 = 48 $ - 10% of 240 = $ 0.10 \times 240 = 24 $ - 15% is halfway between 10% and 20%: $ \frac{24 + 48}{2} = 36 $ --- ### **Final Answer** $$ \boxed{36} $$ ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.3 Tool Calling Step3-VL-10B supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model stepfun-ai/Step3-VL-10B \ --reasoning-parser deepseek-r1 \ --tool-call-parser hermes \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="stepfun-ai/Step3-VL-10B", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True, extra_body={"top_k": -1} ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Example Output:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I have a function called "get_weather" that can provide weather information for a location. Let me check the parameters: - location: required (string) - "Beijing" - unit: optional (string, enum: ["celsius", "fahrenheit"]) - not specified by the user, so I won't include it I should call the function with location="Beijing". =============== Content ================= Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="stepfun-ai/Step3-VL-10B", messages=messages, temperature=0.7, extra_body={"top_k": -1} ) print(final_response.choices[0].message.content) ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (1x) * Model: stepfun-ai/Step3-VL-10B * Tensor Parallelism: 1 * sglang version: 0.5.8+ We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model stepfun-ai/Step3-VL-10B \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model stepfun-ai/Step3-VL-10B \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 30.85 Total input tokens: 14120 Total input text tokens: 720 Total input vision tokens: 13400 Total generated tokens: 4220 Total generated tokens (retokenized): 4217 Request throughput (req/s): 0.32 Input token throughput (tok/s): 457.71 Output token throughput (tok/s): 136.79 Peak output token throughput (tok/s): 240.00 Peak concurrent requests: 2 Total token throughput (tok/s): 594.50 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3083.40 Median E2E Latency (ms): 2747.00 P90 E2E Latency (ms): 4574.50 P99 E2E Latency (ms): 5462.49 ---------------Time to First Token---------------- Mean TTFT (ms): 1327.69 Median TTFT (ms): 1341.01 P99 TTFT (ms): 1486.11 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.16 Median TPOT (ms): 4.17 P99 TPOT (ms): 4.18 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.17 Median ITL (ms): 4.18 P95 ITL (ms): 4.30 P99 ITL (ms): 4.38 Max ITL (ms): 8.24 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model stepfun-ai/Step3-VL-10B \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 976.52 Total input tokens: 1416949 Total input text tokens: 76949 Total input vision tokens: 1340000 Total generated tokens: 510855 Total generated tokens (retokenized): 510526 Request throughput (req/s): 1.02 Input token throughput (tok/s): 1451.02 Output token throughput (tok/s): 523.14 Peak output token throughput (tok/s): 20429.00 Peak concurrent requests: 103 Total token throughput (tok/s): 1974.16 Concurrency: 99.81 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 97463.22 Median E2E Latency (ms): 91872.75 P90 E2E Latency (ms): 118553.42 P99 E2E Latency (ms): 198445.56 ---------------Time to First Token---------------- Mean TTFT (ms): 94379.07 Median TTFT (ms): 87163.09 P99 TTFT (ms): 194871.41 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.89 Median TPOT (ms): 5.72 P99 TPOT (ms): 23.58 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.05 Median ITL (ms): 0.13 P95 ITL (ms): 0.56 P99 ITL (ms): 3.99 Max ITL (ms): 97551.06 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 MMMU Benchmark You can evaluate the model's accuracy using the MMMU dataset: * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model stepfun-ai/Step3-VL-10B \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code ``` * Benchmark Command: ```shell Command theme={null} python3 benchmark/mmmu/bench_sglang.py \ --port 30000 \ --concurrency 64 ``` * Result: ```text Output theme={null} Benchmark time: 934.6179109360091 answers saved to: ./answer_sglang.json Evaluating... answers saved to: ./answer_sglang.json {'Accounting': {'acc': 0.667, 'num': 30}, 'Agriculture': {'acc': 0.367, 'num': 30}, 'Architecture_and_Engineering': {'acc': 0.4, 'num': 30}, 'Art': {'acc': 0.467, 'num': 30}, 'Art_Theory': {'acc': 0.5, 'num': 30}, 'Basic_Medical_Science': {'acc': 0.367, 'num': 30}, 'Biology': {'acc': 0.3, 'num': 30}, 'Chemistry': {'acc': 0.467, 'num': 30}, 'Clinical_Medicine': {'acc': 0.567, 'num': 30}, 'Computer_Science': {'acc': 0.467, 'num': 30}, 'Design': {'acc': 0.567, 'num': 30}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.3, 'num': 30}, 'Economics': {'acc': 0.6, 'num': 30}, 'Electronics': {'acc': 0.567, 'num': 30}, 'Energy_and_Power': {'acc': 0.633, 'num': 30}, 'Finance': {'acc': 0.733, 'num': 30}, 'Geography': {'acc': 0.333, 'num': 30}, 'History': {'acc': 0.533, 'num': 30}, 'Literature': {'acc': 0.533, 'num': 30}, 'Manage': {'acc': 0.6, 'num': 30}, 'Marketing': {'acc': 0.767, 'num': 30}, 'Materials': {'acc': 0.6, 'num': 30}, 'Math': {'acc': 0.7, 'num': 30}, 'Mechanical_Engineering': {'acc': 0.333, 'num': 30}, 'Music': {'acc': 0.4, 'num': 30}, 'Overall': {'acc': 0.523, 'num': 900}, 'Overall-Art and Design': {'acc': 0.483, 'num': 120}, 'Overall-Business': {'acc': 0.673, 'num': 150}, 'Overall-Health and Medicine': {'acc': 0.513, 'num': 150}, 'Overall-Humanities and Social Science': {'acc': 0.492, 'num': 120}, 'Overall-Science': {'acc': 0.5, 'num': 150}, 'Overall-Tech and Engineering': {'acc': 0.481, 'num': 210}, 'Pharmacy': {'acc': 0.6, 'num': 30}, 'Physics': {'acc': 0.7, 'num': 30}, 'Psychology': {'acc': 0.467, 'num': 30}, 'Public_Health': {'acc': 0.733, 'num': 30}, 'Sociology': {'acc': 0.433, 'num': 30}} eval out saved to ./val_sglang.json Overall accuracy: 0.523 ``` # Step-3.5-Flash Source: https://docs.sglang.io/cookbook/autoregressive/StepFun/Step3.5 ## 1. Model Introduction [Step-3.5-Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) is StepFun's production-grade reasoning engine built to decouple elite intelligence from heavy compute, and cuts attention cost for low-latency, cost-effective long-context inference—purpose-built for autonomous agents in real-world workflows. The model is available in multiple quantization formats optimized for different hardware platforms. This generation delivers comprehensive upgrades across the board: * **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 3:1 ratio and an aggressive 128-token window. This hybrid approach ensures consistent performance across massive datasets or long codebases while significantly reducing the computational overhead typical of standard long-context models. * **Sparse Mixture-of-Experts**: Only 11B active parameters out of 196B parameters. * **Multi-Layer Multi-Token Prediction (MTP)**: Equipped with a 3-way Multi-Token Prediction (MTP-3). This allows for complex, multi-step reasoning chains with immediate responsiveness. ## 2.SGLang Installation Step-3.5-Flash is currently available in SGLang via Docker image install. ### Docker (NVIDIA) ```bash Command theme={null} # Pull the docker image docker pull lmsysorg/sglang:latest # Launch the container docker run -it --gpus all \ --shm-size=32g \ --ipc=host \ --network=host \ lmsysorg/sglang:latest bash ``` ### Docker (AMD ROCm) ```bash Command theme={null} # For MI300X/MI325X docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x # For MI350X/MI355X docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x docker run -it \ --device=/dev/kfd --device=/dev/dri \ --shm-size=32g \ --ipc=host \ --network=host \ --group-add video --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ lmsysorg/sglang:v0.5.9-rocm700-mi30x bash # or mi35x for MI350X/MI355X ``` ## 3.Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The Step-3.5-Flash series comes in only one sizes. Recommended starting configurations vary depending on hardware. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities. ### 3.2 Configuration Tips * **Memory**: Requires GPUs with high VRAM capacity. Supported platforms: H200 (4×, TP=4), MI300X/MI325X/MI350X/MI355X (4×, TP=4 EP=4). * **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X and `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X. * **AMD Expert Parallelism Required**: On AMD GPUs, always use `--ep 4` with `--tp 4`. Both BF16 and FP8 models require expert parallelism. Without EP, the MoE intermediate dimension is split across GPUs (N=320), which triggers an AITER CK GEMM incompatibility. With EP=4, each GPU handles 72 full experts (N=1280), which works correctly with cuda graph enabled. * **AITER JIT Compilation**: First inference on AMD may take 30-40 seconds for AITER kernel JIT compilation. Subsequent requests use cached kernels. ## 4.Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser Step-3.5-Flash only supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} sglang serve \ --model-path stepfun-ai/Step-3.5-Flash \ --tp 4 \ --ep 4 \ --reasoning-parser step3p5 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="stepfun-ai/Step-3.5-Flash", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= We are asked: "What is 15% of 240?" We need to solve step by step. Step 1: Understand that "15% of 240" means we need to calculate 15 percent of 240. In mathematical terms, it is (15/100) * 240. Step 2: Simplify the calculation. We can compute 15% of 240 by first finding 10% of 240 and then 5% of 240, and adding them. Alternatively, we can multiply directly. Method 1: 10% of 240 = 240 * 0.10 = 24. 5% is half of 10%, so 5% of 240 = 24 / 2 = 12. Then 15% = 10% + 5% = 24 + 12 = 36. Method 2: Direct multiplication: 15% = 15/100 = 0.15, so 0.15 * 240 = 36. We can also compute fractionally: (15/100)*240 = (15*240)/100. 15*240 = 3600, divided by 100 gives 36. Thus, the answer is 36. We'll present the solution step by step. =============== Content ================= To find 15% of 240, follow these steps: 1. **Convert the percentage to a decimal**: \( 15\% = \frac{15}{100} = 0.15 \) 2. **Multiply by the number**: \( 0.15 \times 240 = 36 \) Alternatively, break it down: - \( 10\% \text{ of } 240 = 240 \times 0.10 = 24 \) - \( 5\% \text{ of } 240 = \frac{24}{2} = 12 \) (since 5% is half of 10%) - \( 15\% = 10\% + 5\% = 24 + 12 = 36 \) **Answer:** 36 ``` #### 4.2.2 Tool Calling Step-3.5 supports tool calling capabilities. Enable the tool call parser: **Python Example:** Start sglang server: ```shell Command theme={null} sglang serve \ --model-path stepfun-ai/Step-3.5-Flash \ --tp 4 \ --ep 4 \ --reasoning-parser step3p5 \ --tool-call-parser step3p5 ``` ```python Example theme={null} from openai import OpenAI import json client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # 1. define tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit"} }, "required": ["location"] } } } ] # 2. tool run def get_weather(location, unit="celsius"): return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # 3. send first request print("--- Sending first request ---") response = client.chat.completions.create( model="stepfun-ai/Step-3.5-Flash", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=1.0, stream=False ) message = response.choices[0].message # 4. Handle Reasoning Content reasoning = getattr(message, 'reasoning_content', None) if reasoning: print("=============== Thinking =================") print(reasoning) print("==========================================") # 5. Handle Tool Calls if message.tool_calls: print("\n🔧 Tool Calls detected:") history_messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, message ] for tool_call in message.tool_calls: print(f" Tool: {tool_call.function.name}") print(f" Args: {tool_call.function.arguments}") args = json.loads(tool_call.function.arguments) tool_result = get_weather(args.get("location"), args.get("unit", "celsius")) history_messages.append({ "role": "tool", "tool_call_id": tool_call.id, "content": tool_result }) print("\n--- Sending tool results ---") final_response = client.chat.completions.create( model="stepfun-ai/Step-3.5-Flash", messages=history_messages, temperature=1.0, stream=False ) print("=============== Final Content =================") print(final_response.choices[0].message.content) else: if message.content: print("=============== Content =================") print(message.content) ``` **Output Example:** ```text Output theme={null} --- Sending first request --- =============== Thinking ================= The user is asking for the weather in Beijing. I should use the get_weather function with location="Beijing". The unit parameter is optional and the user didn't specify a preference, so I'll leave it out (the default should be fine). ========================================== 🔧 Tool Calls detected: Tool: get_weather Args: {"location": "Beijing"} --- Sending tool results --- =============== Final Content ================= The weather in Beijing is 22°C and sunny. ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA H200 GPU (4x) * Model: Step-3.5-Flash * Tensor Parallelism: 4 * Expert Parallelism: 4 * sglang version: 0.5.8 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. #### 5.1.1 Standard Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path stepfun-ai/Step-3.5-Flash \ --tp 4 \ --ep 4 ``` ##### 5.1.1.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model stepfun-ai/Step-3.5-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 35.30 Total input tokens: 6091 Total input text tokens: 6091 Total generated tokens: 4220 Total generated tokens (retokenized): 4212 Request throughput (req/s): 0.28 Input token throughput (tok/s): 172.57 Output token throughput (tok/s): 119.56 Peak output token throughput (tok/s): 124.00 Peak concurrent requests: 2 Total token throughput (tok/s): 292.14 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3527.94 Median E2E Latency (ms): 2884.72 P90 E2E Latency (ms): 6350.38 P99 E2E Latency (ms): 7858.53 ---------------Time to First Token---------------- Mean TTFT (ms): 107.53 Median TTFT (ms): 80.93 P99 TTFT (ms): 269.52 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.12 Median TPOT (ms): 8.13 P99 TPOT (ms): 8.14 ---------------Inter-Token Latency---------------- Mean ITL (ms): 8.12 Median ITL (ms): 8.11 P95 ITL (ms): 8.61 P99 ITL (ms): 8.91 Max ITL (ms): 20.77 ================================================== ``` ##### 5.1.1.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model stepfun-ai/Step-3.5-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 54.06 Total input tokens: 39588 Total input text tokens: 39588 Total generated tokens: 40805 Total generated tokens (retokenized): 40479 Request throughput (req/s): 1.48 Input token throughput (tok/s): 732.33 Output token throughput (tok/s): 754.84 Peak output token throughput (tok/s): 928.00 Peak concurrent requests: 21 Total token throughput (tok/s): 1487.17 Concurrency: 14.06 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 9501.23 Median E2E Latency (ms): 10010.71 P90 E2E Latency (ms): 15655.09 P99 E2E Latency (ms): 18803.63 ---------------Time to First Token---------------- Mean TTFT (ms): 198.34 Median TTFT (ms): 89.50 P99 TTFT (ms): 984.66 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 18.97 Median TPOT (ms): 18.80 P99 TPOT (ms): 35.67 ---------------Inter-Token Latency---------------- Mean ITL (ms): 18.27 Median ITL (ms): 17.48 P95 ITL (ms): 18.44 P99 ITL (ms): 62.47 Max ITL (ms): 460.85 ================================================== ``` ##### 5.1.1.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model stepfun-ai/Step-3.5-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 125.88 Total input tokens: 249331 Total input text tokens: 249331 Total generated tokens: 252662 Total generated tokens (retokenized): 251323 Request throughput (req/s): 3.97 Input token throughput (tok/s): 1980.77 Output token throughput (tok/s): 2007.23 Peak output token throughput (tok/s): 2500.00 Peak concurrent requests: 109 Total token throughput (tok/s): 3987.99 Concurrency: 92.25 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 23223.31 Median E2E Latency (ms): 22631.90 P90 E2E Latency (ms): 42269.38 P99 E2E Latency (ms): 47637.53 ---------------Time to First Token---------------- Mean TTFT (ms): 372.13 Median TTFT (ms): 127.26 P99 TTFT (ms): 1880.42 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 46.06 Median TPOT (ms): 47.61 P99 TPOT (ms): 51.34 ---------------Inter-Token Latency---------------- Mean ITL (ms): 45.31 Median ITL (ms): 39.86 P95 ITL (ms): 72.49 P99 ITL (ms): 117.05 Max ITL (ms): 1359.81 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * **Results**: * Step-3.5-Flash ``` Accuracy: 0.885 Invalid: 0.005 Latency: 9.986 s Output throughput: 1972.911 token/s ``` # Hy3 preview Source: https://docs.sglang.io/cookbook/autoregressive/Tencent/Hunyuan3-Preview ## 1. Model Introduction Hy3 preview is Tencent's preview of its third-generation flagship MoE language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving. **Key Features:** * **MoE Architecture**: 192 routed experts + 1 shared expert, 8 experts activated per token. \~276B total parameters with \~20B active, delivering dense-model quality at MoE inference cost. * **Hybrid Thinking**: Reasoning modes (`high`, `medium`, `low`, `none`) controllable via OpenAI-standard `reasoning_effort`, allowing the same weights to trade off latency and depth of reasoning. * **Native Tool Calling**: Trained on structured `` / `` / `` grammar. Pairs with SGLang's `hunyuan` tool-call parser for streaming OpenAI-compatible function-calling output. * **Long Context**: 256K token context window (262,144 positions) for repository-scale code and document reasoning. * **Multi-Token Prediction (MTP)**: Ships with a built-in MTP draft module enabling speculative decoding out of the box. **Available Models:** * [tencent/Hy3-preview](https://huggingface.co/tencent/Hy3-preview) — BF16 instruct * [tencent/Hy3-preview-Base](https://huggingface.co/tencent/Hy3-preview-Base) — BF16 base **Recommended Generation Parameters:**
Parameter Value
`temperature` 0.7
`top_p` 0.9
`reasoning_effort` `high` / `medium` / `low` (thinking) or `none` (instant)
**License:** TODO — verify on HuggingFace model card. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. **Docker Images by Hardware Platform:**
Hardware Platform Docker Image
NVIDIA H200 / B200 / B300 / GB300 `lmsysorg/sglang:latest`
`lmsysorg/sglang:latest` bundles the HYV3 model code, the `hunyuan` tool-call / reasoning parsers, and the MTP draft-module runtime. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization, and feature capabilities. ### 3.2 Configuration Tips **Key Parameters:**
Parameter Description Recommended Value
`--tool-call-parser` Tool call parser for function-calling support `hunyuan`
`--reasoning-parser` Reasoning parser for hybrid thinking modes `hunyuan`
`--trust-remote-code` Required for Hunyuan model loading Always enabled
`--mem-fraction-static` Static memory fraction (KV + activations) `0.9`
`--tp` Tensor parallelism size `2` / `4` / `8` depending on hardware
`--attention-backend` Attention backend (Blackwell only) `trtllm_mha`
`--speculative-algorithm` Speculative decoding via the bundled MTP draft `EAGLE` + `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`
**Hardware Requirements: NVIDIA BF16 (`Hy3-preview`, \~552GB weights)** * **H200 (141GB) / B200 (180GB)**: TP=8 (minimum for BF16 to fit single-node). * **B300 (275GB) / GB300**: TP=4. * **A100 / H100 (80GB)**: not supported single-node — BF16 requires multi-node TP=16+ on 80GB-class GPUs. **Blackwell (B200 / B300 / GB300):** Auto-selected attention backend can mis-route for HYV3 on Blackwell. Always pass `--attention-backend trtllm_mha` explicitly on Blackwell hardware (the config generator above enforces this). **Multi-Token Prediction (MTP):** The `Hy3-preview` release bundles an MTP draft module. SGLang runs it via its EAGLE speculative-decoding path — the draft module auto-loads from the same `--model-path`. Enable with the standard MTP flags: ```bash Command theme={null} sglang serve \ --model-path tencent/Hy3-preview \ --tp 8 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser hunyuan \ --tool-call-parser hunyuan \ --trust-remote-code \ --mem-fraction-static 0.85 ``` Toggle the "Speculative Decoding (MTP)" option in the generator above to add these flags automatically. Tune `num-steps` / `num-draft-tokens` based on acceptance rate in your workload. **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) **Deployment Command (H200 × 8, BF16 default):** ```bash Command theme={null} sglang serve \ --model-path tencent/Hy3-preview \ --tp 8 \ --reasoning-parser hunyuan \ --tool-call-parser hunyuan \ --trust-remote-code \ --mem-fraction-static 0.9 ``` **Testing Deployment:** After startup, you can test the SGLang OpenAI-compatible API with the following command: ```bash Command theme={null} curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "tencent/Hy3-preview", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"} ] }' ``` **Simple Completion Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="tencent/Hy3-preview", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"} ], max_tokens=1024 ) print("Reasoning:", response.choices[0].message.reasoning_content) print("Content: ", response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} Reasoning: None Content: The Los Angeles Dodgers won the 2020 World Series. They defeated the Tampa Bay Rays in six games (4-2). This was the Dodgers' first World Series championship since 1988. The series was notable for being played in a neutral-site bubble at Globe Life Field in Arlington, Texas, due to the COVID-19 pandemic. ``` When `reasoning_effort` is not set, the server defaults to instant mode (no thinking, `reasoning_content=None`). To opt into thinking, pass `reasoning_effort="high" / "medium" / "low"` on the request — see the Hybrid Thinking section below. ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser (Hybrid Thinking) Hy3-preview is a hybrid-thinking model. Control the thinking budget via the OpenAI-standard `reasoning_effort`: * `high` / `medium` / `low` — increasing amounts of chain-of-thought in `reasoning_content` * `none` — skip thinking entirely (instant responses, content-only) Enable the reasoning parser during deployment so that the thinking section (`...`) is separated into `reasoning_content`: ```bash Command theme={null} sglang serve \ --model-path tencent/Hy3-preview \ --tp 8 \ --reasoning-parser hunyuan \ --trust-remote-code \ --mem-fraction-static 0.9 ``` **Thinking Mode — High Effort:** ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="tencent/Hy3-preview", messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}], reasoning_effort="high", max_tokens=2048, ) msg = response.choices[0].message print("=============== Thinking =================") print(msg.reasoning_content) print("=============== Content =================") print(msg.content) ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= We need to solve: "What is 15% of 240?" Step by step. So we need to compute 15% of 240. The process: 15% means 15 per hundred, i.e., 15/100 = 0.15. Multiply 0.15 by 240. Or we can do: 10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. Or do multiplication: 15/100 * 240 = (15*240)/100 = (3600)/100 = 36. So answer is 36. We need to produce step-by-step explanation. The instruction: "Solve step by step: What is 15% of 240?" So we should provide a clear solution with steps. The final answer: 36. Also maybe include units? No units. We'll output the solution in a clear manner. =============== Content ================= To find 15% of 240, follow these steps: 1. **Understand that percent means "per hundred."** So, 15% = 15/100 or 0.15. 2. **Multiply the number (240) by the percentage in decimal form.** 0.15 × 240. Alternatively, you can use fractions: (15/100) × 240. 3. **Perform the multiplication.** 0.15 × 240 = 36. Or: (15 × 240) / 100 = 3600 / 100 = 36. 4. **Check using an alternative method:** - 10% of 240 = 24. - 5% of 240 = half of 10% = 12. - 15% = 10% + 5% = 24 + 12 = 36. Thus, **15% of 240 is 36**. ``` **Instant Mode — No Thinking:** ```python Example theme={null} response = client.chat.completions.create( model="tencent/Hy3-preview", messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}], reasoning_effort="none", max_tokens=256, ) print("Content:", response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe. ``` #### 4.2.2 Tool Calling Hy3-preview supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips thinking tokens before the tool-call parser runs: ```bash Command theme={null} sglang serve \ --model-path tencent/Hy3-preview \ --tp 8 \ --reasoning-parser hunyuan \ --tool-call-parser hunyuan \ --trust-remote-code \ --mem-fraction-static 0.9 ``` **Non-Streaming Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city.", "parameters": { "type": "object", "properties": { "city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["city"], }, }, } ] response = client.chat.completions.create( model="tencent/Hy3-preview", messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], tools=tools, ) msg = response.choices[0].message print("Reasoning:", msg.reasoning_content) print("Content: ", msg.content) for tc in msg.tool_calls or []: print(f"Tool Call: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` **Output Example:** ```text Output theme={null} Reasoning: None Content: I'll get the current weather for Beijing in Fahrenheit for you. Tool Call: get_weather Arguments: {"city": "Beijing", "unit": "fahrenheit"} ``` **Streaming Example (incremental argument deltas):** Hy3-preview's `hunyuan` tool-call parser emits tool names first, then argument JSON in incremental fragments — matching the OpenAI streaming contract: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") stream = client.chat.completions.create( model="tencent/Hy3-preview", messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], tools=tools, stream=True, ) tool_buffer = {} for chunk in stream: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) for tc in delta.tool_calls or []: buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""}) if tc.function and tc.function.name: buf["name"] += tc.function.name if tc.function and tc.function.arguments: buf["args"] += tc.function.arguments for idx, buf in tool_buffer.items(): print(f"\nTool[{idx}] {buf['name']}({buf['args']})") ``` **Output Example:** ```text Output theme={null} I'll check the current weather in Beijing for you using Fahrenheit. Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"}) ``` ## 5. Benchmark ### 5.1 Accuracy Benchmark **Test Environment:** * Hardware: 8× NVIDIA H200 (141GB) * Docker Image: `lmsysorg/sglang:hy3-preview` * Model: `tencent/Hy3-preview` (BF16) * Tensor Parallelism: 8 * SGLang version: latest `main` #### 5.1.1 GSM8K * Benchmark Method: GSM8K on 200 questions using sgl-eval * Benchmark Command: ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 \ --num-threads 64 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. * Test Results: ```text Output theme={null} TODO — replace with real GSM8K accuracy after benchmark run on Hy3-preview (BF16). ``` #### 5.1.2 MMLU * Benchmark Method: zero-shot Chat API evaluation via sgl-eval, all 57 subjects * Benchmark Command: ```bash Command theme={null} sgl-eval run mmlu \ --base-url http://127.0.0.1:30000/v1 \ --num-threads 64 --max-tokens 2048 ``` * Test Results: ```text Output theme={null} TODO — replace with real MMLU accuracy after benchmark run on Hy3-preview (BF16). ``` #### 5.1.3 Tool-Call Accuracy (MiniMax-Provider-Verifier) * Benchmark Tool: [MiniMax-Provider-Verifier](https://github.com/MiniMax-AI/MiniMax-Provider-Verifier) * Metric: function-call schema validity, argument match, and end-to-end response correctness * Test Results: ```text Output theme={null} TODO — replace with real tool-call accuracy after benchmark run on Hy3-preview (BF16). ``` ### 5.2 Speed Benchmark #### 5.2.1 Low Concurrency * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model tencent/Hy3-preview \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} TODO — replace with real low-concurrency output on Hy3-preview (BF16). ``` #### 5.2.2 High Concurrency * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model tencent/Hy3-preview \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} TODO — replace with real high-concurrency output on Hy3-preview (BF16). ``` # Hy3 Source: https://docs.sglang.io/cookbook/autoregressive/Tencent/Hy3 Deploy Tencent Hy3 with SGLang — verified launch commands and tuning for the BF16 Mixture-of-Experts model with hybrid thinking, native tool calling, 256K context, and MTP speculative decoding. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate # Install from source (main carries the suffix-aware `hunyuan` parser + the # HYV3 model code). Once a tagged release picks it up, # `uv pip install --prerelease=allow sglang` is enough. git clone https://github.com/sgl-project/sglang.git cd sglang uv pip install --prerelease=allow -e python ``` Run the **Python** output of the command panel below in that environment. ```bash Command theme={null} # The image bundles the HYV3 model code and the suffix-aware `hunyuan` parser. docker pull lmsysorg/sglang:dev ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator below produces. The `dev` image bundles the HYV3 model code, the suffix-aware `hunyuan` reasoning/tool-call parsers, and the MTP draft-module runtime. The same parsers serve both the preview (suffix-less) and the shipping (suffixed) Hy3 tokenizer — no per-model hard-coding. Pick your hardware + recipe to generate the launch command. * **Low-Latency** — fastest reply for a single user. Pick for chat. * **Balanced** — good speed with several users at once. Use for typical multi-user serving.

Panel controls (top of the command box):

  • Python / Docker — bare sglang serve … for an existing SGLang env, or a docker run … sglang serve … wrap against the per-hardware image from the Install SGLang panel above.
  • ⧉ Copy — copies the current command (with whichever framing is active) to your clipboard.
  • \$ cURL — a sample request against localhost:30000 to confirm the server is up.
  • ⚙ Env — edits the placeholders (HOST\_IP, PORT, HF\_TOKEN, NODE\_RANK, NODE0\_IP) the command and cURL share. Persists in localStorage across cookbooks.
  • Verified / Not Verified badge — green when the (hw, variant, quant, strategy, nodes) combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.
## Playground The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change. The knobs come in two flavors: * **Built-in SGLang features** — parallelism overrides (TP / DP-Attention), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers. * **Hy3 specific** — `--tool-call-parser auto` / `--reasoning-parser auto` (auto-detect Hy3's suffix-aware `hunyuan` parsers from the chat template; resolve the real special tokens from the tokenizer vocab at runtime). Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.

Panel controls reuse Python / Docker · ⧉ Copy · \$ cURL · ⚙ Env from the Deploy panel, plus one extra:

  • Submit ↗ — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says Not Verified; click it once you've actually run the command on your hardware and confirmed it works.
## 1. Model Introduction **Hy3** is Tencent's third-generation flagship Mixture-of-Experts language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving. **Key Features:** * **MoE Architecture**: 192 routed experts + 1 shared expert, top-8 activated per token. 295B total parameters with 21B active (+3.8B MTP layer), delivering dense-model quality at MoE inference cost. * **Hybrid Thinking**: Reasoning modes (`high`, `low`, `no_think`) controllable via OpenAI-standard `reasoning_effort`, allowing the same weights to trade off latency and depth of reasoning. * **Native Tool Calling**: Trained on a structured grammar. Pairs with SGLang's `hunyuan` tool-call parser for streaming OpenAI-compatible function-calling output. * **Long Context**: 256K token context window (262,144 positions) for repository-scale code and document reasoning. * **Multi-Token Prediction (MTP)**: Ships with a built-in MTP draft module enabling speculative decoding out of the box. **Available Model:** * [tencent/Hy3](https://huggingface.co/tencent/Hy3) — BF16 instruct * [tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8) — FP8 **Recommended Generation Parameters:**
Parameter Value
temperature 0.9
top\_p 1.0
reasoning\_effort high / low (thinking) or no\_think (instant)
**Special tokens.** The shipping Hy3 tokenizer appends a shared suffix to every special token (e.g. `` instead of the bare ``). SGLang's `hunyuan` parsers resolve the real token strings from the tokenizer vocab at runtime ([PR #29920](https://github.com/sgl-project/sglang/pull/29920)), so the same recipe serves both the preview (suffix-less) and the shipping (suffixed) tokenizer — no per-model hard-coding. This is why `--reasoning-parser hunyuan` / `--tool-call-parser hunyuan` work out of the box on the shipping model. ## 2. Configuration Tips **Hardware requirements (BF16, \~590GB weights):**
GPU VRAM TP Notes
H200 141GB 8 minimum single-node for BF16
B200 192GB 4 BF16 590GB → 148GB/GPU
B300 / GB300 288GB 4 BF16 590GB → 148GB/GPU; ample KV headroom
GB200 192GB 4 single-node 4×192GB = 768GB fits BF16 590GB
**Blackwell attention backend.** On SM100/SM103 (B200 / B300 / GB200 / GB300), SGLang auto-selects the `trtllm_mha` attention backend for HYV3's MHA architecture (no flag needed) — the launch commands above omit it for that reason. Override only if you have a specific kernel reason. **MTP (Multi-Token Prediction, EAGLE).** * `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1. * `balanced`: MTP disabled — keep the prefill batch moderate so chunked-prefill stays efficient. **`reasoning_effort` vs `thinking`.** The Hy3 chat template is driven by `reasoning_effort` (`high` / `low` / `no_think`), NOT by the `thinking` flag that some other families use. The default is `no_think` (instant). To opt into thinking, pass `reasoning_effort="high"` on the request (the OpenAI-standard field; sglang forwards it to the template). `reasoning_effort: max` is rejected by sglang — use `high`. For eval, sgl-eval's `--thinking` flag translates to `reasoning_effort="high"` for Hy3, so the benchmark commands below use it as-is. ## 3. Advanced Usage ### 3.1 Reasoning (Hybrid Thinking) Hy3 is a hybrid-thinking model. Control the thinking budget via `reasoning_effort`: * `high` / `low` — increasing amounts of chain-of-thought in `reasoning_content` * `no_think` — skip thinking entirely (instant responses, content-only) Enable the reasoning parser during deployment so the thinking section is separated into `reasoning_content`: ```bash Command theme={null} sglang serve \ --model-path tencent/Hy3 \ --tp 8 \ --reasoning-parser auto \ --tool-call-parser auto ``` ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="tencent/Hy3", messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}], reasoning_effort="high", max_tokens=2048, ) msg = response.choices[0].message print("=============== Thinking =================") print(msg.reasoning_content) print("=============== Content =================") print(msg.content) ``` ```text Output theme={null} =============== Thinking ================= We need to solve: "What is 15% of 240?" Step by step. 15% means 15/100 = 0.15. Multiply 0.15 by 240. 10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. So answer is 36. =============== Content ================= To find 15% of 240, follow these steps: 1. 15% = 15/100 or 0.15. 2. Multiply 240 by 0.15: 0.15 × 240 = 36. 3. Check: 10% of 240 = 24, 5% = 12, 15% = 36. Thus, 15% of 240 is 36. ``` ```python Example theme={null} response = client.chat.completions.create( model="tencent/Hy3", messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}], reasoning_effort="no_think", max_tokens=256, ) print("Content:", response.choices[0].message.content) ``` ```text Output theme={null} Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe. ``` ### 3.2 Tool Calling Hy3 supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips any thinking tokens before the tool-call parser runs: ```bash Command theme={null} sglang serve \ --model-path tencent/Hy3 \ --tp 8 \ --reasoning-parser auto \ --tool-call-parser auto ``` ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city.", "parameters": { "type": "object", "properties": { "city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["city"], }, }, } ] response = client.chat.completions.create( model="tencent/Hy3", messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], tools=tools, ) msg = response.choices[0].message print("Reasoning:", msg.reasoning_content) print("Content: ", msg.content) for tc in msg.tool_calls or []: print(f"Tool Call: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` ```text Output theme={null} Reasoning: None Content: I'll get the current weather for Beijing in Fahrenheit for you. Tool Call: get_weather Arguments: {"city": "Beijing", "unit": "fahrenheit"} ``` ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") stream = client.chat.completions.create( model="tencent/Hy3", messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], tools=tools, stream=True, ) tool_buffer = {} for chunk in stream: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) for tc in delta.tool_calls or []: buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""}) if tc.function and tc.function.name: buf["name"] += tc.function.name if tc.function and tc.function.arguments: buf["args"] += tc.function.arguments for idx, buf in tool_buffer.items(): print(f"\nTool[{idx}] {buf['name']}({buf['args']})") ``` ```text Output theme={null} I'll check the current weather in Beijing for you using Fahrenheit. Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"}) ``` # Hy4 preview Source: https://docs.sglang.io/cookbook/autoregressive/Tencent/Hy4-Preview Deploy Tencent Hy4 preview with SGLang — launch recipes for the 770B-parameter Mixture-of-Experts model with MLA, DeepSeek Sparse Attention (DSA), and MTP speculative decoding, in BF16 on H200/B200/B300/GB300 and MXFP8 on Blackwell GPUs. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](/docs/get-started/install). The Docker path below matches the **Docker** framing in the command panel. ```bash Command theme={null} docker pull lmsysorg/sglang:hy4-preview ``` The `hy4-preview` image bundles the HYV4 model code, the suffix-aware `hunyuan` reasoning/tool-call parsers, and the NEXTN MTP runtime. Switch to `:latest` once a tagged release picks them up. For how to launch the image, see [Install → Method 3: Using Docker](/docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + recipe to generate the launch command. The two serving strategies map to whether the MTP (NextN) draft layer is on: * **Low-Latency** — MTP speculative decoding on (steps=3, draft-tokens=4). Fastest reply for a single user; pick for chat. * **High-Throughput** — MTP off. At saturation the draft+verify overhead outweighs the speedup; best for batch jobs.

Panel controls (top of the command box):

  • Python / Docker — bare sglang serve … for an existing SGLang env, or a docker run … sglang serve … wrap against the per-hardware image from the Install SGLang panel above.
  • ⧉ Copy — copies the current command (with whichever framing is active) to your clipboard.
  • \$ cURL — a sample request against localhost:30000 to confirm the server is up.
  • ⚙ Env — edits the placeholders (HOST\_IP, PORT, HF\_TOKEN, NODE\_RANK, NODE0\_IP) the command and cURL share. Persists in localStorage across cookbooks.
  • Badge — single-node recipes show green Verified (run end-to-end on real hardware); the 2-node BF16 recipes show In Progress and flip to Verified as their multi-node verification lands.
## Playground The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change. The knobs come in two flavors: * **Built-in SGLang features** — parallelism overrides (TP / DP-Attention), the MoE a2a backend (DeepEP, which sets EP = TP), reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers. * **Hy4 specific** — `--reasoning-parser auto` / `--tool-call-parser auto` resolve to the suffix-aware Hunyuan parsers and read the real structural-token strings from the tokenizer vocab at runtime. Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the base recipe but stripped by an override. Parallelism combinations beyond the listed recipes (DP-Attention, DeepEP, other TP degrees) are experimentation territory — any override flips the badge to **Not Verified** until the configuration is run end-to-end and submitted back.

Panel controls reuse Python / Docker · ⧉ Copy · \$ cURL · ⚙ Env from the Deploy panel, plus one extra:

  • Submit ↗ — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says Not Verified; click it once you've actually run the command on your hardware and confirmed it works.
## 1. Model Introduction **Hy4 preview** is Tencent's next-generation flagship Mixture-of-Experts language model: 770B total parameters with 49B active per token, pairing a DeepSeek-style MLA + sparse-attention stack with Hunyuan's own residual control, MoE routing, and attention gating. It is a **text-only** model (vocab 120,832; the endpoint rejects image input by design), released under **Apache-2.0**. **Key architecture:** * **MoE**: 78 layers — layer 0 is a dense MLP, the remaining 77 are sparse MoE with 256 routed experts + 1 shared expert, top-8 sigmoid-scored routing (routed scaling 2.827), expert intermediate size 2048, and bounded SwiGLU (clamp 10.0). * **MLA + DSA on every layer**: Multi-head Latent Attention (`q_lora_rank` 2048, `kv_lora_rank` 512, 192 nope + 64 rope head dims, `v_head_dim` 256) under DeepSeek Sparse Attention — indexer top-k 2048 with 32 index heads, indexers alternating full/shared (one full indexer every 4 layers), and an FP8 index cache. * **iHC residual control**: Hunyuan's own inter-layer residual gating (`enable_ihc`, `hc_mult` 4) with pre- and post-residual gate groups. Semantically distinct from DeepSeek-V4's mHC (no combination step, no Sinkhorn) — the implementations are not interchangeable. * **Gated MLA + attention sink**: element-wise attention output gating evaluated in fp32, plus a learnable per-head attention sink propagated through all attention backends. * **MTP (NextN)**: one built-in multi-token-prediction draft layer (`model.mtp_layers.0`, 10B parameters with \~0.7B active, present in both checkpoints) enabling speculative decoding out of the box. * **Long context**: 1M max positions (RoPE theta 1e7). Size `--context-length` to your KV budget — the sizing table in §2 suggests 262,144 (131,072 on H200). **Available models:** * [tencent/Hy4-preview](https://huggingface.co/tencent/Hy4-preview) — BF16 (\~1.5TB weights) * [tencent/Hy4-preview-FP8](https://huggingface.co/tencent/Hy4-preview-FP8) — MXFP8 (ModelOpt recipe, UE8M0 group-32 weight scales, dynamic activations; \~760GB weights) **Recommended generation:** `temperature=0.9`, `top_p=1.0` (informational — SGLang applies the checkpoint's `generation_config.json` defaults; don't hardcode sampling parameters in client code). Thinking depth is controlled per request via `reasoning_effort` (defaults to `high`; `no_think` for direct responses — see §3.1). **Resources:** [HuggingFace](https://huggingface.co/tencent/Hy4-preview) · [GitHub](https://github.com/Tencent-Hunyuan/Hy4-preview). **Special tokens.** The Hy4 preview tokenizer's structural tokens are suffix-bearing (``, ``, ``, ``, ``). SGLang's Hunyuan reasoning/tool-call parsers resolve the real token strings from the tokenizer vocab at runtime, so `--reasoning-parser auto --tool-call-parser auto` work out of the box. ## 2. Configuration Tips **Hardware sizing.** BF16 weights are \~1.5TB and MXFP8 \~760GB; the MLA KV cache (compressed `kv_lora` 512 + rope 64, plus the DSA FP8 indexer cache, ≈95KB/token) is replicated per TP rank, so the per-rank pool left after weights sets the context ceiling:
GPU VRAM MXFP8 (\~760GB) BF16 (\~1.5TB)
H200 141GB Not supported — the MXFP8 kernel path requires SM100 (Blackwell) TP16, 2×8 nodes · 131K context
B200 192GB TP8, single node · 262K context TP16, 2×8 nodes · 262K context (8×192GB ≈ the weights alone)
B300 288GB TP4, single node · 262K context (\~190GB/rank) TP8, single 8-GPU node · 262K context
GB300 288GB TP4, single node · 262K context TP8, 2×4 nodes · 262K context (GB300 hosts carry 4 GPUs)
On H200 (BF16, TP16 → \~95GB of weights per rank) the per-rank pool holds roughly 260K tokens of KV — set `--context-length 131072` there (the model default is 1M positions, far beyond the pool); the 192GB+ parts comfortably support 262144. **DSA attention backend.** Every layer runs DeepSeek Sparse Attention; SGLang auto-selects the DSA backend for HYV4 (`--attention-backend dsa` with `flashmla_sparse` prefill/decode and an FP8 indexer cache over a bf16 KV pool), so the recipes don't pass attention flags. Override only with a kernel-specific reason. **MXFP8 kernel stack.** The MXFP8 checkpoint self-describes via its ModelOpt `hf_quant_config` (dynamic activations, UE8M0 group-32 weight scales) — no `--quantization` flag needed; the recipes pin `--moe-runner-backend deep_gemm --fp8-gemm-backend deep_gemm`, the validated HYV4 MXFP8 path (the runtime also defaults both to `deep_gemm` for HYV4 when the flags are left unset). The MXFP8 kernel path requires SM100+ (Blackwell); H200 (SM90) cannot serve the MXFP8 checkpoint — use BF16 there. **CUDA graph decode vs eager.** Decode CUDA-graph capture is on by default; long-duration soak validation of the graph path on Hy4 is still in progress. If you hit instability under long mixed agentic workloads, pass `--disable-cuda-graph` to fall back to eager decode (a restart recovers cleanly either way). **MTP (NextN) speculative decoding.** Both checkpoints ship one draft layer; the preset is `--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-num-draft-tokens 4` (top-k 1). Speculative decoding reserves 4 draft-token slots per request, so the effective request budget is `prompt_tokens + max_tokens + 4 ≤ context length` — requests at the exact context boundary are rejected with the reservation accounted for. **Fail-fast guardrails.** The model rejects pipeline parallelism and `--enable-prefill-cp` before allocation. The recipes run pure TP; DP-Attention, DeepEP (which sets EP = TP at runtime), and other TP degrees are Playground experimentation territory. **Multi-node BF16.** The BF16 weights don't fit a single H200/B200/GB300 host, so those cells are 2-node TP recipes — run the generated command on every node (the panel injects `--nnodes 2 --node-rank --dist-init-addr`) and keep the weights on storage shared across ranks. **Text-only.** Image input is rejected with HTTP 400 by design — don't route vision traffic to this endpoint. **Large prefills under concurrency.** First-prefill latency on very large prompts can exceed 30 s under high concurrency; use a client timeout of 300 s (and moderate concurrency) for long-context agentic workloads instead of the common 30 s default. ## 3. Advanced Usage ### 3.1 Reasoning (`reasoning_effort`) Hy4 preview is a hybrid-thinking model: reasoning defaults to `high` (deep chain-of-thought — suits math, coding, and agentic tasks). The top-level OpenAI `reasoning_effort` field accepts the standard tiers (e.g. `high`); for direct responses, pass the model-specific `no_think` through `extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}}`. The Deploy recipes enable the reasoning parser (`--reasoning-parser auto`) so thinking is separated into `reasoning_content` and the final answer into `content`: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="tencent/Hy4-preview-FP8", messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}], reasoning_effort="high", max_tokens=2048, ) msg = response.choices[0].message print("=============== Thinking =================") print(msg.reasoning_content) print("=============== Content =================") print(msg.content) ``` ```text Output theme={null} Pending update — will be captured verbatim from a live Hy4 preview server. ``` ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="tencent/Hy4-preview-FP8", messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}], extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}}, max_tokens=256, ) print("Content:", response.choices[0].message.content) ``` ```text Output theme={null} Pending update — will be captured verbatim from a live Hy4 preview server. ``` ### 3.2 Tool Calling Hy4 preview emits tool calls through suffixed structural tokens with an `arg_key` / `arg_value` argument format; SGLang's Hunyuan tool-call parser reassembles them into OpenAI-compatible `message.tool_calls` with schema-aware type coercion, for both streaming and non-streaming requests. The Deploy recipes enable both parsers together (`--reasoning-parser auto --tool-call-parser auto`) — the reasoning parser strips thinking tokens before the tool-call parser runs. Tool-call output is parsed, not grammar-constrained: `tool_choice: "required"` / named-function forcing is not enforced with structural-tag guided decoding on the current implementation. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city.", "parameters": { "type": "object", "properties": { "city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["city"], }, }, } ] response = client.chat.completions.create( model="tencent/Hy4-preview-FP8", messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], tools=tools, ) msg = response.choices[0].message print("Reasoning:", msg.reasoning_content) print("Content: ", msg.content) for tc in msg.tool_calls or []: print(f"Tool Call: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` ```text Output theme={null} Pending update — will be captured verbatim from a live Hy4 preview server. ``` # Inkling Source: https://docs.sglang.io/cookbook/autoregressive/ThinkingMachines/Inkling Deploy Inkling with SGLang — verified launch commands, tuning, and multimodal / reasoning / tool-calling usage for Thinking Machines' 975B Mixture-of-Experts model with 1M-token context. ## Deployment For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). Inkling has merged to `main` but isn't in a `pip` release yet — install from source: ```bash Command theme={null} pip install --upgrade pip pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' ``` Then run the **Python** output of the command panel below. The Inkling images are being published to [`lmsysorg/sglang`](https://hub.docker.com/r/lmsysorg/sglang/tags) — watch the tag list for status. There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU: ```bash Command theme={null} docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13 docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 (final build; lane retired) docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware to generate the launch command. Each platform ships a **Balanced** recipe plus **MTP** and **DSpark** (speculative decoding) tiers and a **Long Context (MXFP8 KV)** tier where validated; the **LoRA** variant serves adapters on top of the frozen base model. Set `MAX_LORAS` to the number of distinct adapters you serve (1 is fastest for single-adapter serving).

Panel controls (top of the command box):

  • ⧉ Copy — copies the current command to your clipboard.
  • \$ cURL — a sample request against localhost:30000 to confirm the server is up.
  • ⚙ Env — edits the placeholders (HOST\_IP, PORT, NODE\_RANK, NODE0\_IP) the command and cURL share.
  • Verified / Not Verified badge — green when the (hw, variant, quant, strategy, nodes) combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.
## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change. Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end. ## 1. Model Introduction **Inkling** is a Mixture-of-Experts model from Thinking Machines — **975B** total parameters, **41B** active per token, with a **1M-token** context window and **open weights** (BF16 and NVFP4 checkpoints below). It handles text, image, and audio inputs natively, and exposes a **variable reasoning-effort** control to trade latency and cost against answer quality. This page covers serving Inkling on SGLang, including its **MTP** speculative-decoding path and long-context prefix caching (unified radix cache + HiCache). **Resources:** HuggingFace — [Inkling](https://huggingface.co/thinkingmachines/Inkling) (BF16) · [Inkling-NVFP4](https://huggingface.co/thinkingmachines/Inkling-NVFP4). ## 2. Configuration Tips **Multimodal.** The recipes pass `--enable-multimodal` so the server accepts image and audio inputs alongside text — drop it for text-only serving. **Memory pool ratios.** `--swa-full-tokens-ratio` and `--mamba-full-memory-ratio` (both default `0.1`) size the SWA and Mamba/sconv state pools; tune them to your workload's usage. **MTP needs `--enable-multi-layer-eagle`.** The MTP recipe drives Inkling's multi-layer draft head; without this flag the standard EAGLE worker runs against it and outputs garbage. **Reasoning effort.** Pass `reasoning_effort` as one of the named levels below; requests that omit it default to `high`, and `max` is the strongest. Each level maps to an internal effort value (max at `0.99`):
reasoning\_effort value
none0.0
minimal0.1
low0.2
medium0.7
high0.9
xhigh0.99
max0.99
## 3. Advanced Usage ### 3.1 Reasoning Enable the `inkling` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="thinkingmachines/Inkling-NVFP4", messages=[{"role": "user", "content": "What is 17 times 24?"}], extra_body={"chat_template_kwargs": {"thinking": True}}, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Reasoning: The user is asking for the product of 17 and 24. Let me calculate that. 17 × 24 I can break this down: 17 × 20 = 340 17 × 4 = 68 340 + 68 = 408 Alternatively: 24 × 10 = 240 24 × 7 = 168 240 + 168 = 408 So the answer is 408. Answer: 17 times 24 is **408**. Here's a quick breakdown: - 17 × 20 = 340 - 17 × 4 = 68 - 340 + 68 = **408** ``` ### 3.2 Tool Calling Enable the `inkling` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": {"location": {"type": "string", "description": "The city name"}}, "required": ["location"], }, }, } ] resp = client.chat.completions.create( model="thinkingmachines/Inkling-NVFP4", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Reasoning: The user is asking for the weather in Beijing. I have a tool called `get_weather` that can get the current weather for a location. Let me call it with "Beijing" as the location. Content: Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_98f772f3a0044f45b80c5ba5', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)] ``` ### 3.3 Multimodal Input (Image + Audio) Inkling is multimodal: a single user message can mix **text**, **images**, and **audio**. Pass each media item as its own content part — `image_url` for images, `audio_url` for audio — with the `url` set to either an HTTP(S) link or a base64 `data:` URI. Audio can also be sent as OpenAI's `input_audio` part, carrying the base64 bytes in `data` alongside a `format` of `wav` or `mp3`. The server must be started with `--enable-multimodal` (already included in every recipe above). ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("image.png", "rb") as f: image_b64 = base64.b64encode(f.read()).decode() with open("audio.wav", "rb") as f: audio_b64 = base64.b64encode(f.read()).decode() resp = client.chat.completions.create( model="thinkingmachines/Inkling-NVFP4", messages=[ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}}, {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}}, {"type": "text", "text": "Describe the image, then transcribe the audio."}, ], } ], max_tokens=1024, ) print(resp.choices[0].message.content) ``` Images and audio can be sent as public HTTP(S) URLs instead of base64 — e.g. `{"type": "image_url", "image_url": {"url": "https://.../photo.jpg"}}`. Use one content part per media item; mix as many as the context budget allows. ### 3.4 LoRA (Serving Adapters) The **LoRA** deploy variant serves adapters on top of the frozen base model. Its launch command adds `--enable-lora --lora-paths lora0={{ADAPTER_PATH}} --max-loras-per-batch {{MAX_LORAS}}` — each adapter is registered under the **name** to the left of `=` (here `lora0`). Adapters can also be added/removed at runtime via the `POST /load_lora_adapter` endpoint. To serve several adapters, pass multiple `--lora-paths name=path` at launch and reference each by its name. Pick the adapter per request by that name — either in the `model` field with `base-model:adapter` syntax (recommended), or explicitly via `lora_path` in `extra_body`: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") # Option A (recommended): ":" in the model field resp = client.chat.completions.create( model="thinkingmachines/Inkling-NVFP4:lora0", messages=[{"role": "user", "content": "Summarize the changelog."}], ) # Option B: explicit lora_path via extra_body resp = client.chat.completions.create( model="thinkingmachines/Inkling-NVFP4", messages=[{"role": "user", "content": "Summarize the changelog."}], extra_body={"lora_path": "lora0"}, ) print(resp.choices[0].message.content) ``` One adapter per request — omit the `:adapter` suffix (and `lora_path`) to hit the base model. Different requests **in the same batch** may use different adapters; the number of *distinct* adapters co-resident in a batch is capped by `--max-loras-per-batch` (the `MAX_LORAS` field, default `1`). If both `model:adapter` and `lora_path` are supplied, the `model` suffix takes precedence. ### 3.5 HiCache (Hierarchical KV Caching) Inkling serves on SGLang's **unified radix cache**: the historically separate full-attention, SWA, and Mamba/sconv caches are combined into one radix tree with typed components, and native HiCache offloads cold prefix pages across tiers (GPU HBM → host DRAM → disk / remote). This expands effective prefix-cache capacity for multi-turn and long-context workloads. To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**, then pick a storage backend (`file` / `mooncake` / `nixl`) for the L3 tier. The Write policy defaults to `write_through`. ### 3.6 Long Context (MXFP8 KV) The **Long Context** deploy strategy adds `--kv-cache-dtype mxfp8` on top of the Balanced recipe. KV entries are stored as block-scaled MXFP8 instead of BF16, so the SWA + Mamba/sconv memory pool holds roughly 2x as many tokens on the same GPU. Use it when you're context-bound or concurrency-bound. **Blackwell only.** MXFP8 KV cache requires Blackwell (B200 / B300 / GB200 / GB300), it's not offered on Hopper (H200). The tradeoff is not just a \~5% decode latency penalty from the extra quantize/dequantize work versus BF16 KV — storing KV in MXFP8 also introduces some accuracy loss at long context lengths. Treat it as a capacity lever, not a speed one — stay on **Balanced** if you have headroom in the memory pool and just want lower latency or maximum output quality. To try it, select the **Long Context** strategy in the Deploy panel above for any NVFP4 cell; the panel regenerates the launch command with `--kv-cache-dtype mxfp8` inserted. Verified end-to-end on B200, B300, and GB300. ### 3.7 DSpark (Speculative Decoding) The **DSpark** deploy strategy is the second speculative-decoding path for Inkling. Unlike **MTP**, which drives Inkling's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-DSpark-Preview` — served unquantized alongside the NVFP4 target. DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12 — a final build, since that lane is retired), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4). # Inkling-Small Source: https://docs.sglang.io/cookbook/autoregressive/ThinkingMachines/Inkling-Small Deploy Inkling-Small with SGLang — launch commands, tuning, and multimodal / reasoning / tool-calling usage for Thinking Machines' Inkling-Small Mixture-of-Experts model. ## Deployment For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). Inkling-Small has merged to `main` but isn't in a `pip` release yet — install from source: ```bash Command theme={null} pip install --upgrade pip pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' ``` Then run the **Python** output of the command panel below. The Inkling-Small images are being published to [`lmsysorg/sglang`](https://hub.docker.com/r/lmsysorg/sglang/tags) — watch the tag list for status. There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU. DGX Spark (GB10) uses a dedicated arm64 CUDA 13 image: ```bash Command theme={null} docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13 docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 (final build; lane retired) docker pull lmsysorg/sglang:dev-inkling-small-dgx-spark # DGX Spark (GB10 / SM121) docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware to generate the launch command. Each platform ships a **Balanced** recipe plus **MTP** and **DSpark** (speculative decoding) tiers and a **Long Context (MXFP8 KV)** tier where validated; the **LoRA** variant serves adapters on top of the frozen base model. Set `MAX_LORAS` to the number of distinct adapters you serve (1 is fastest for single-adapter serving).

Panel controls (top of the command box):

  • ⧉ Copy — copies the current command to your clipboard.
  • \$ cURL — a sample request against localhost:30000 to confirm the server is up.
  • ⚙ Env — edits the placeholders (HOST\_IP, PORT, NODE\_RANK, NODE0\_IP) the command and cURL share.
  • Verified / Not Verified badge — green when the (hw, variant, quant, strategy, nodes) combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.
## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change. Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end. ## 1. Model Introduction **Inkling-Small** is a Mixture-of-Experts model from Thinking Machines with **open weights** (BF16 and NVFP4 checkpoints below), in the same architecture family as Inkling. It handles text, image, and audio inputs natively, and exposes a **variable reasoning-effort** control to trade latency and cost against answer quality. This page covers serving Inkling-Small on SGLang, including its **MTP** speculative-decoding path and long-context prefix caching (unified radix cache + HiCache). **Resources:** HuggingFace — [Inkling-Small](https://huggingface.co/thinkingmachines/Inkling-Small) (BF16) · [Inkling-Small-NVFP4](https://huggingface.co/thinkingmachines/Inkling-Small-NVFP4). ## 2. Configuration Tips **Multimodal.** The recipes pass `--enable-multimodal` so the server accepts image and audio inputs alongside text — drop it for text-only serving. **DGX Spark (2× GB10).** The verified cell runs NVFP4 with TP=2 across two Sparks over ConnectX-7 (1 GPU per node). Use the `dev-inkling-small-dgx-spark` image, Triton attention + Marlin FP4/MoE, and `--disable-prefill-cuda-graph`. The Docker command already carries the ConnectX-7 flags `--ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`. **Memory pool ratios.** `--swa-full-tokens-ratio` and `--mamba-full-memory-ratio` (both default `0.1`) size the SWA and Mamba/sconv state pools; tune them to your workload's usage. **MTP needs `--enable-multi-layer-eagle`.** The MTP recipe drives Inkling-Small's multi-layer draft head; without this flag the standard EAGLE worker runs against it and outputs garbage. **Reasoning effort.** Pass `reasoning_effort` as one of the named levels below; requests that omit it default to `high`, and `max` is the strongest. Each level maps to an internal effort value (max at `0.99`):
reasoning\_effort value
none0.0
minimal0.1
low0.2
medium0.7
high0.9
xhigh0.99
max0.99
## 3. Advanced Usage ### 3.1 Reasoning Enable the `inkling` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="thinkingmachines/Inkling-Small-NVFP4", messages=[{"role": "user", "content": "What is 17 times 24?"}], extra_body={"chat_template_kwargs": {"thinking": True}}, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Reasoning: The user is asking for the product of 17 and 24. Let me calculate that. 17 × 24 I can break this down: 17 × 20 = 340 17 × 4 = 68 340 + 68 = 408 Alternatively: 24 × 10 = 240 24 × 7 = 168 240 + 168 = 408 So the answer is 408. Answer: 17 times 24 is **408**. Here's a quick breakdown: - 17 × 20 = 340 - 17 × 4 = 68 - 340 + 68 = **408** ``` ### 3.2 Tool Calling Enable the `inkling` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": {"location": {"type": "string", "description": "The city name"}}, "required": ["location"], }, }, } ] resp = client.chat.completions.create( model="thinkingmachines/Inkling-Small-NVFP4", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Reasoning: The user is asking for the weather in Beijing. I have a tool called `get_weather` that can get the current weather for a location. Let me call it with "Beijing" as the location. Content: Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_98f772f3a0044f45b80c5ba5', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)] ``` ### 3.3 Multimodal Input (Image + Audio) Inkling-Small is multimodal: a single user message can mix **text**, **images**, and **audio**. Pass each media item as its own content part — `image_url` for images, `audio_url` for audio — with the `url` set to either an HTTP(S) link or a base64 `data:` URI. Audio can also be sent as OpenAI's `input_audio` part, carrying the base64 bytes in `data` alongside a `format` of `wav` or `mp3`. The server must be started with `--enable-multimodal` (already included in every recipe above). ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("image.png", "rb") as f: image_b64 = base64.b64encode(f.read()).decode() with open("audio.wav", "rb") as f: audio_b64 = base64.b64encode(f.read()).decode() resp = client.chat.completions.create( model="thinkingmachines/Inkling-Small-NVFP4", messages=[ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}}, {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}}, {"type": "text", "text": "Describe the image, then transcribe the audio."}, ], } ], max_tokens=1024, ) print(resp.choices[0].message.content) ``` Images and audio can be sent as public HTTP(S) URLs instead of base64 — e.g. `{"type": "image_url", "image_url": {"url": "https://.../photo.jpg"}}`. Use one content part per media item; mix as many as the context budget allows. ### 3.4 LoRA (Serving Adapters) The **LoRA** deploy variant serves adapters on top of the frozen base model. Its launch command adds `--enable-lora --lora-paths lora0={{ADAPTER_PATH}} --max-loras-per-batch {{MAX_LORAS}}` — each adapter is registered under the **name** to the left of `=` (here `lora0`). Adapters can also be added/removed at runtime via the `POST /load_lora_adapter` endpoint. To serve several adapters, pass multiple `--lora-paths name=path` at launch and reference each by its name. Pick the adapter per request by that name — either in the `model` field with `base-model:adapter` syntax (recommended), or explicitly via `lora_path` in `extra_body`: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") # Option A (recommended): ":" in the model field resp = client.chat.completions.create( model="thinkingmachines/Inkling-Small-NVFP4:lora0", messages=[{"role": "user", "content": "Summarize the changelog."}], ) # Option B: explicit lora_path via extra_body resp = client.chat.completions.create( model="thinkingmachines/Inkling-Small-NVFP4", messages=[{"role": "user", "content": "Summarize the changelog."}], extra_body={"lora_path": "lora0"}, ) print(resp.choices[0].message.content) ``` One adapter per request — omit the `:adapter` suffix (and `lora_path`) to hit the base model. Different requests **in the same batch** may use different adapters; the number of *distinct* adapters co-resident in a batch is capped by `--max-loras-per-batch` (the `MAX_LORAS` field, default `1`). If both `model:adapter` and `lora_path` are supplied, the `model` suffix takes precedence. ### 3.5 HiCache (Hierarchical KV Caching) Inkling-Small serves on SGLang's **unified radix cache**: the historically separate full-attention, SWA, and Mamba/sconv caches are combined into one radix tree with typed components, and native HiCache offloads cold prefix pages across tiers (GPU HBM → host DRAM → disk / remote). This expands effective prefix-cache capacity for multi-turn and long-context workloads. To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**, then pick a storage backend (`file` / `mooncake` / `nixl`) for the L3 tier. The Write policy defaults to `write_through`. ### 3.6 Long Context (MXFP8 KV) The **Long Context** deploy strategy adds `--kv-cache-dtype mxfp8` on top of the Balanced recipe. KV entries are stored as block-scaled MXFP8 instead of BF16, so the SWA + Mamba/sconv memory pool holds roughly 2x as many tokens on the same GPU. Use it when you're context-bound or concurrency-bound. **Blackwell only.** MXFP8 KV cache requires Blackwell (B200 / B300 / GB200 / GB300), it's not offered on Hopper (H200). The tradeoff is not just a \~5% decode latency penalty from the extra quantize/dequantize work versus BF16 KV — storing KV in MXFP8 also introduces some accuracy loss at long context lengths. Treat it as a capacity lever, not a speed one — stay on **Balanced** if you have headroom in the memory pool and just want lower latency or maximum output quality. To try it, select the **Long Context** strategy in the Deploy panel above for any NVFP4 cell; the panel regenerates the launch command with `--kv-cache-dtype mxfp8` inserted. Verified end-to-end on B200, B300, and GB300. ### 3.7 DSpark (Speculative Decoding) The **DSpark** deploy strategy is the second speculative-decoding path for Inkling-Small. Unlike **MTP**, which drives Inkling-Small's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-Small-DSpark` — served unquantized alongside the NVFP4 target. DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12 — a final build, since that lane is retired), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4). # MiMo-V2-Flash Source: https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2-Flash ## Introduction XiaomiMiMo/MiMo-V2-Flash, with 309B total parameters and 15B activated parameters, is a new inference-centric model designed to maximize decoding efficiency created by XiaomiMiMo Team explicitly co-designed for real-world serving workloads, enabling flexible tradeoffs between throughput and latency on different hardware. This model creates a new balance between long-context modeling capability and inference efficiency. Key features include: * **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and an aggressive 128-token window. This reduces KV-cache storage by nearly 6x while maintaining long-context performance via learnable attention sink bias. * **Multi-Token Prediction (MTP)**: Equipped with a lightweight MTP module (0.33B params/block) using dense FFNs. This triples output speed during inference and will be good to accelerates rollout in RL training. * **Efficient Pre-Training**: Trained on 27T tokens using FP8 mixed precision and native 32k seq length. The context window supports up to 256k length. * **Agentic Capabilities**: Post-training utilizes Multi-Teacher On-Policy Distillation (MOPD) and large-scale agentic RL, achieving superior performance on SWE-Bench and complex reasoning tasks. ## Installation MiMo-V2-Flash is currently available in SGLang via Docker image and pip install. ### Docker ```bash Command theme={null} # Pull the docker image docker pull lmsysorg/sglang:latest # Launch the container docker run -it --gpus all \ --shm-size=32g \ --ipc=host \ --network=host \ lmsysorg/sglang:latest bash ``` ### Pip Installation ```bash Command theme={null} # On a machine with SGLang dependencies installed or inside a SGLang nightly container # Start an SGLang nightly container docker run -it --gpus all \ --shm-size=32g \ --ipc=host \ --network=host \ lmsysorg/sglang:latest bash # If you already have SGLang installed, uninstall the current SGLang version pip uninstall sglang -y # Install the PyPI Package pip install sglang==0.5.6.post2.dev8005+pr.15207.g39d5bd57a \ --extra-index-url https://sgl-project.github.io/whl/pr/ ``` ## Model Deployment Use the configuration selector below to automatically generate the appropriate deployment command. MI355X (ROCm) is validated in the selector above with `--tp-size 4`, Triton attention, and `--disable-custom-all-reduce`. `--tp-size 8` hit a QKV sharding error during validation. EAGLE speculative decoding is still WIP on MI355X. ## Testing the deployment Once the server is running, test it with a chat completion request in another terminal: ```bash Command theme={null} curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "XiaomiMiMo/MiMo-V2-Flash", "messages": [ {"role": "user", "content": "Hello! What can you help me with?"} ], "temperature": 0.7, "max_tokens": 100 }' ``` **Expected response:** ```json Config theme={null} { "id": "...", "object": "chat.completion", "model": "XiaomiMiMo/MiMo-V2-Flash", "choices": [{ "message": { "role": "assistant", "content": "Hello! I can help you with..." } }] } ``` ## Troubleshooting **DeepGEMM Timeout Error** Occasionally DeepGEMM timeout errors occur during first launch. Simply rerun the server command in the same container - the compiled kernels are cached and subsequent launches will be fast. **ROCm MI355X Attention Backend** If you see an error such as `AiterAttnBackend.forward_decode() got an unexpected keyword argument 'sinks'` on MI355X, use the `MI355X` + `Performance Optimizations` command from the selector above, which switches to Triton attention and keeps `--disable-custom-all-reduce`. # MiMo-V2.5 Source: https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2.5 ## 1. Model Introduction [MiMo-V2.5-Pro](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro) and [MiMo-V2.5](https://huggingface.co/XiaomiMiMo/MiMo-V2.5) are next-generation Mixture-of-Experts models from the XiaomiMiMo Team.
Variant Total params Active (MoE) Modalities
MiMo-V2.5-Pro 1.02T 42B Text (multimodal planned)
MiMo-V2.5 310B 15B Text, Image, Video, Audio
**Key Features:** * **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) for reduced KV cache while preserving long-context capability. * **Multi-Token Prediction (MTP)**: 3-layer MTP module accelerates decoding. Both variants support EAGLE speculative decoding with MTP weights. * **1M-Token Context**: Both variants support up to 1 million token context windows. * **Agentic Capabilities**: Post-training with large-scale agentic RL achieves strong performance on coding, reasoning, and tool-use benchmarks. * **MiMo-V2.5 Multimodal** (V2.5 only): Native omnimodal architecture with a 729M-param ViT Vision Encoder (28 layers: 24 SWA + 4 Full) and a 261M-param Audio Transformer (24 layers: 12 SWA + 12 Full); supports image, video, and audio understanding via standard OpenAI-compatible multimodal API. **License:** Apache 2.0 ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install). **Docker Image:** All variants (MiMo-V2.5 310B and MiMo-V2.5-Pro 1.02T) use `lmsysorg/sglang:latest`, which ships CUDA 13.0 and runs on both Hopper (H100 / H200) and Blackwell (B200 / GB300). **TPU (sgl-jax):** MiMo-V2.5-Pro can also be served on TPU via the JAX-based [sgl-jax](https://github.com/sgl-project/sglang-jax) runtime. The container image and `pip install` steps are listed in [§3.3 TPU Deployment](#3-3-tpu-deployment-mimo-v2-5-pro-sgl-jax). ## 3. Model Deployment ### 3.1 Basic Configuration Use the selector below to generate the deployment command for your variant and hardware. ### 3.2 Configuration Tips **MiMo-V2.5-Pro (1.02T):** * **B200**: single node, TP=8 (verified). Uses `--attention-backend fa4` + `--moe-runner-backend flashinfer_trtllm` + `--mem-fraction-static 0.8`. Set `--swa-full-tokens-ratio 0.1` to keep KV-cache footprint within 192 GB HBM. * **GB300**: 2 nodes, TP=8 (verified). Same Blackwell stack as B200; multi-node interconnect requires `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1`. Default SWA ratio is fine. * **H100/H200**: 2 nodes × 8 GPUs (TP=16, not yet verified). Uses the Hopper stack (`fa3` + DeepEP + EAGLE multi-layer); fits with `--mem-fraction-static 0.7` and `--swa-full-tokens-ratio 0.3`. DeepEP dispatch tuning: `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256` avoids memory spikes during prefill. * EAGLE speculative decoding (3 steps, topk=1) typically yields a 2–3× decode speedup. Requires `--enable-multi-layer-eagle` (both Hopper and Blackwell). See §5.4 for acceptance-rate behavior on natural text vs random prompts. **MiMo-V2.5 (310B):** * The checkpoint has a TP=4-interleaved fused `qkv_proj`; attention-TP per DP group **must** be 4. Use `--attn-dp-size = TP / 4`; for TP > 4 this means DP-attention is on. Total GPUs must be a multiple of 4. A bare `--tp 8` without `--attn-dp-size 2` will fail to load with `MiMoV2 fused qkv_proj checkpoint is TP=4-interleaved; got attention tp_size=8`. * Single-node deployments: H100/H200 8× GPUs (`--tp 8 --attn-dp-size 2`), B200 4× GPUs (`--tp 4`, attention DP size 1, no DP-attn flag needed), GB300 4× GPUs (`--tp 4`, single NVL4 node). FP8 quantization. * On Blackwell, pass `--attention-backend fa4`: MiMoV2's asymmetric KV (`head_dim` 192 / `v_head_dim` 128) fails on the SM100 default `trtllm_mha`, which requires equal K/V widths. * On Blackwell, pass `--mm-attention-backend fa4` for the V2.5 vision encoder. The checkpoint config requests FlashAttention-3 internally, but SGLang rejects FA3 on Blackwell and expects FA4 for multimodal attention. * On Blackwell, pass `--moe-runner-backend flashinfer_trtllm`; the default `auto` falls through to the triton fused-MoE runner, \~12% slower at bs=1 decode. * `--enable-dp-lm-head` and `--mm-enable-dp-encoder` are required whenever `--attn-dp-size` is greater than 1, to keep LM head and encoder sharding consistent. * EAGLE MTP uses the checkpoint's MTP weights. Enable with `--speculative-algorithm EAGLE` and `--enable-multi-layer-eagle` (both Hopper and Blackwell). * **Multimodal**: Supports image, video, and audio understanding; see Section 4.3 for invocation examples. **DeepEP (optional toggle, Hopper-only):** * DeepEP replaces the default MoE all-to-all dispatch with a fused [DeepEP](https://github.com/deepseek-ai/DeepEP) backend; it lowers expert dispatch latency and memory traffic, so it pays off under **high concurrency / throughput-bound workloads** on H100/H200. Under concurrency=1 / latency-bound workloads the gain is negligible — leave it off. * Enabling adds `--moe-a2a-backend deepep` + `--moe-dense-tp-size 1` (and `--ep ` for Pro) plus `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256` env to cap the dispatch buffer. Requires `pip install deep_ep` (not part of the default sglang install). * On Blackwell (B200, GB300) the verified MoE backend is `flashinfer_trtllm`; the DeepEP toggle is a no-op there. ### 3.3 TPU Deployment (MiMo-V2.5-Pro, sgl-jax) MiMo-V2.5-Pro can also be served on TPU via [sgl-jax](https://github.com/sgl-project/sglang-jax). The runtime is a separate JAX-based stack (`sgl_jax.launch_server`); pick **TPU v7x** or **TPU v6e** in the panel above to generate the launch command. Verified topologies: | TPU Type | Topology | Chips/Node | Nodes | Total Chips | JAX Devices/Chip | Total JAX Devices (= `--tp-size`) | | - | - | - | - | - | - | - | | **v7x** | 2×2×4 | 4 | 4 | 16 | 2 | 32 | | **v6e** | 4×4×4 | 4 | 16 | 64 | 1 | 64 | > v7x exposes **2 logical JAX devices per chip**, so `--tp-size = 16 chips × 2 = 32`. v6e exposes 1 device per chip, so `--tp-size = 64`. Always set `--tp-size` to the total JAX device count across all nodes, not the chip count. All nodes must sit in the same TPU slice and reach each other on the JAX init port (`20000`) and the TPU process port (`8471`). **Step 1 — Launch the JAX TPU container on every node:** ```shell Command theme={null} docker run -it --privileged \ --shm-size=32g \ --ipc=host \ --network=host \ -v /dev:/dev \ us-docker.pkg.dev/cloud-tpu-images/jax-ai-image/tpu:jax0.8.1-rev1 bash ``` > The image is pinned to `jax0.8.1-rev1` to keep the JAX runtime aligned with sgl-jax's TPU extras. **Step 2 — Clone and install sgl-jax (inside the container):** ```shell Command theme={null} git clone https://github.com/sgl-project/sglang-jax.git cd sglang-jax pip install -e "python[tpu]" ``` ## 4. Model Invocation ### 4.1 Basic Usage See [Basic API Usage](../../../docs/basic_usage/send_request). ### 4.2 Reasoning Output Both variants support hybrid thinking mode. Thinking content is separated via the reasoning parser. **Thinking Mode (default):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="XiaomiMiMo/MiMo-V2.5", messages=[ {"role": "user", "content": "Which is larger, 9.11 or 9.9? Think carefully."} ] ) print("====== Reasoning ======") print(response.choices[0].message.reasoning_content) print("====== Answer ======") print(response.choices[0].message.content) ``` **Output Example (MiMo-V2.5):** ```text theme={null} ====== Reasoning ====== Comparing 9.11 and 9.9. The integer parts are both 9. Now compare the decimal parts: 0.11 vs 0.9. 0.9 = 0.90, which is greater than 0.11. So 9.9 > 9.11. ====== Answer ====== **9.9 is larger than 9.11.** Here's the reasoning: When comparing decimals, line them up to the same number of decimal places: - 9.11 - 9.90 Both have a **9** in the ones place, but in the tenths place, **9 > 1**, so 9.90 > 0.11. **9.9 > 9.11** ``` **Thinking Off (instant mode):** ```python Example theme={null} response = client.chat.completions.create( model="XiaomiMiMo/MiMo-V2.5", messages=[ {"role": "user", "content": "Which is larger, 9.11 or 9.9? Think carefully."} ], extra_body={"chat_template_kwargs": {"thinking": False}} ) print(response.choices[0].message.content) ``` **Output Example (MiMo-V2.5):** ```text theme={null} ## Comparing 9.11 and 9.9 **9.9 is larger.** The key is to compare them place by place. It helps to write them with the same number of decimal places: - **9.11** → 9.11 - **9.9** → 9.90 Both have **9** in the ones place, but in the tenths place: **9** (in 9.90) is greater than **1** (in 9.11). So **9.90 > 9.11**. ``` ### 4.3 Multimodal Invocation (V2.5 only) **Image Understanding:** ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="XiaomiMiMo/MiMo-V2.5", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sgl-test-files/refs/heads/main/images/man_ironing_on_back_of_suv.png"}}, {"type": "text", "text": "Describe this image in detail."} ] }] ) print(response.choices[0].message.content) ``` **Output Example:** ```text theme={null} Based on the image provided, here is a detailed description: The image captures a whimsical or surreal scene set on a busy city street, likely in New York City given the iconic yellow cabs. In the center foreground, a man is sitting on a folding chair, casually crossing his legs. He is wearing a bright yellow hoodie with a graphic on the front and blue jeans. He is intently focused on ironing a white dress shirt that rests on an ironing board set up directly on the asphalt. Behind him, a yellow SUV taxi cab is stopped or moving slowly, angled slightly away from the camera. To his left, another yellow taxi sedan is captured in motion blur, indicating it is driving past him. The background features tall city buildings with glass windows and storefronts. There are banners hanging from streetlights, and some greenery is visible in the distance. The overall impression is one of incongruity—performing a domestic chore like ironing in the middle of a chaotic urban environment. ``` **Video Understanding:** ```python Example theme={null} response = client.chat.completions.create( model="XiaomiMiMo/MiMo-V2.5", messages=[{ "role": "user", "content": [ {"type": "video_url", "video_url": {"url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"}}, {"type": "text", "text": "Summarize what happens in this video."} ] }] ) print(response.choices[0].message.content) ``` **Output Example:** ```text theme={null} A person wearing blue protective gloves is shown operating a microscope in a close-up shot. The individual is adjusting a knob on the side of the microscope, which moves the stage holding a glass slide, likely focusing the lens on the specimen. ``` > Video decoding requires `decord` (`pip install decord`); SGLang's MiMo-V2.5 multimodal processor uses `decord.VideoReader` for frame extraction. **Audio Understanding:** ```python Example theme={null} response = client.chat.completions.create( model="XiaomiMiMo/MiMo-V2.5", messages=[{ "role": "user", "content": [ {"type": "audio_url", "audio_url": {"url": "https://raw.githubusercontent.com/sgl-project/sgl-test-files/refs/heads/main/audios/Trump_WEF_2018_10s.mp3"}}, {"type": "text", "text": "Transcribe and summarize this audio."} ] }] ) print(response.choices[0].message.content) ``` **Output Example:** ```text theme={null} **Transcript:** "Thank you Klaus very much. It's a privilege to be here at this forum where leaders in business, science, art, diplomacy and world affairs have gathered for..." **Summary:** The speaker thanks Klaus for the introduction and expresses their honor at attending a forum. They highlight that the event has brought together high-level leaders from various sectors, including business, science, art, and diplomacy. ``` ### 4.4 Tool Calling ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]} }, "required": ["location"] } } } ] response = client.chat.completions.create( model="XiaomiMiMo/MiMo-V2.5", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools ) msg = response.choices[0].message if msg.reasoning_content: print("=== Reasoning ===") print(msg.reasoning_content) if msg.tool_calls: print("=== Tool Calls ===") for tc in msg.tool_calls: print(f" Function: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` **Output Example (MiMo-V2.5):** ```text theme={null} === Reasoning === The user wants to know the weather in Beijing. I have a function available called "get_weather" that can retrieve current weather for a location. Let me call that function with Beijing as the location. === Tool Calls === Function: get_weather Arguments: {"location": "Beijing"} ``` ## 5. Benchmark Accuracy numbers come from `sglang.test.run_eval` (GSM8K standard 5-shot, MMMU validation split). Speed numbers come from `sglang.bench_serving` with generated random prompts; text runs use 1024 input tokens and 1024 output tokens per request, and the image run uses 2 random 720p images per request. ### 5.1 Accuracy Benchmark #### 5.1.1 GSM8K Standard 5-shot, `temperature=0`, `max_tokens=4096`, model defaults to thinking-on (responses contain `...` and the eval extracts the trailing number via regex). Server launch: see [Section 3](#3-model-deployment). **Benchmark Command:** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --model XiaomiMiMo/MiMo-V2.5 \ --num-examples 200 \ --num-threads 8 \ --max-tokens 4096 \ --temperature 0.0 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. > `run_eval.py` automatically appends `/v1` to `--base-url`; pass the bare `host:port` URL (without trailing `/v1`), otherwise requests resolve to `/v1/v1/chat/completions` and 404. * **Test Results:** * MiMo-V2.5-Pro (FP8, 8× B200) ``` Score: 0.965 (193 / 200) Latency: 253.90 s Output throughput: 461.78 tok/s ``` * MiMo-V2.5 (FP8, 8× H200) ``` Score: 0.980 (196 / 200) Latency: 477.52 s Output throughput: 88.9 tok/s ``` #### 5.1.2 MMMU (V2.5 only) `MMMU/MMMU` validation split (multi-discipline multimodal), `concurrency=16`, default sampling. * **Benchmark Command:** ```shell Command theme={null} python3 benchmark/mmmu/bench_sglang.py \ --port 30000 \ --model XiaomiMiMo/MiMo-V2.5 \ --concurrency 16 ``` * **Test Results:** * MiMo-V2.5 (FP8) ``` Pending update ``` ### 5.2 Speed Benchmark — MiMo-V2.5-Pro **Test Environment:** * Hardware: NVIDIA B200 GPU (8×) * Model: `XiaomiMiMo/MiMo-V2.5-Pro` (FP8) * Tensor Parallelism: 8 (single-node, `--moe-runner-backend flashinfer_trtllm`, `--attention-backend fa4`, `--mem-fraction-static 0.8`, `--swa-full-tokens-ratio 0.1`) * Recipe: Blackwell verified baseline (EAGLE off for this benchmark — see note below) * sglang version: 0.5.11 > The numbers in §5.2 are the **no-EAGLE baseline** on `random 1024/1024`. On uniform-random token streams the MiMo-V2.5-Pro 3-layer MTP draft has very low accept-rate (\~0.13–0.27 vs \~0.75 on natural-text prompts, see §5.4) — there's no token-co-occurrence signal for the draft to model — so EAGLE here adds verify overhead without recovering enough draft tokens to be a net win on this workload. EAGLE MTP itself works on B200 + `--enable-multi-layer-eagle` (see §3 deployment command and §5.4 for an acceptance profile on natural text). #### 5.2.1 Latency-Sensitive Benchmark * **Model Deployment Command:** see the [command panel above](#3-model-deployment). * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model XiaomiMiMo/MiMo-V2.5-Pro \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 27.59 Total input tokens: 1997 Total input text tokens: 1997 Total generated tokens: 2798 Total generated tokens (retokenized): 2794 Request throughput (req/s): 0.36 Input token throughput (tok/s): 72.38 Output token throughput (tok/s): 101.41 Peak output token throughput (tok/s): 110.00 Peak concurrent requests: 3 Total token throughput (tok/s): 173.79 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2757.26 Median E2E Latency (ms): 3319.10 P90 E2E Latency (ms): 4157.47 P99 E2E Latency (ms): 4869.32 ---------------Time to First Token---------------- Mean TTFT (ms): 162.17 Median TTFT (ms): 68.11 P99 TTFT (ms): 929.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.19 Median TPOT (ms): 9.33 P99 TPOT (ms): 9.39 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.31 Median ITL (ms): 9.35 P95 ITL (ms): 9.44 P99 ITL (ms): 9.77 Max ITL (ms): 19.80 ================================================== ``` #### 5.2.2 Throughput-Sensitive Benchmark * **Model Deployment Command:** see the [command panel above](#3-model-deployment). * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model XiaomiMiMo/MiMo-V2.5-Pro \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 112.78 Total input tokens: 302118 Total input text tokens: 302118 Total generated tokens: 195775 Total generated tokens (retokenized): 191069 Request throughput (req/s): 8.87 Input token throughput (tok/s): 2678.83 Output token throughput (tok/s): 1735.90 Peak output token throughput (tok/s): 3040.00 Peak concurrent requests: 121 Total token throughput (tok/s): 4414.73 Concurrency: 87.80 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 9901.96 Median E2E Latency (ms): 6525.54 P90 E2E Latency (ms): 23567.98 P99 E2E Latency (ms): 42109.22 ---------------Time to First Token---------------- Mean TTFT (ms): 223.69 Median TTFT (ms): 139.45 P99 TTFT (ms): 1082.02 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 50.63 Median TPOT (ms): 51.66 P99 TPOT (ms): 91.41 ---------------Inter-Token Latency---------------- Mean ITL (ms): 49.79 Median ITL (ms): 33.69 P95 ITL (ms): 103.37 P99 ITL (ms): 151.34 Max ITL (ms): 1600.00 ================================================== ``` ### 5.3 Speed Benchmark — MiMo-V2.5 **Test Environment:** * Hardware: NVIDIA H200 GPU (8×) * Model: `XiaomiMiMo/MiMo-V2.5` (FP8) * Tensor Parallelism: 8 (DP-attention with `--attn-dp-size 2`) * Recipe: Balanced (DP-attn + EAGLE MTP) * sglang version: `0.0.0.dev1+g7d99af439` (`lmsysorg/sglang:dev-mimo-v2.5`) #### 5.3.1 Latency-Sensitive Benchmark * **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment). * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model XiaomiMiMo/MiMo-V2.5 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 14.72 Total input tokens: 1997 Total input text tokens: 1997 Total generated tokens: 2798 Total generated tokens (retokenized): 2697 Request throughput (req/s): 0.68 Input token throughput (tok/s): 135.67 Output token throughput (tok/s): 190.09 Peak output token throughput (tok/s): 245.00 Peak concurrent requests: 3 Total token throughput (tok/s): 325.77 Concurrency: 1.00 Accept length: 3.08 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1469.98 Median E2E Latency (ms): 1652.84 P90 E2E Latency (ms): 2210.80 P99 E2E Latency (ms): 2823.86 ---------------Time to First Token---------------- Mean TTFT (ms): 143.89 Median TTFT (ms): 99.25 P99 TTFT (ms): 481.01 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.87 Median TPOT (ms): 4.30 P99 TPOT (ms): 6.64 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.76 Median ITL (ms): 3.46 P95 ITL (ms): 13.52 P99 ITL (ms): 13.84 Max ITL (ms): 74.37 ================================================== ``` #### 5.3.2 Throughput-Sensitive Benchmark * **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment). * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model XiaomiMiMo/MiMo-V2.5 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 93.41 Total input tokens: 302118 Total input text tokens: 302118 Total generated tokens: 195775 Total generated tokens (retokenized): 188139 Request throughput (req/s): 10.71 Input token throughput (tok/s): 3234.48 Output token throughput (tok/s): 2095.97 Peak output token throughput (tok/s): 3019.00 Peak concurrent requests: 121 Total token throughput (tok/s): 5330.45 Concurrency: 91.04 Accept length: 2.95 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8503.45 Median E2E Latency (ms): 7491.96 P90 E2E Latency (ms): 13706.99 P99 E2E Latency (ms): 20474.33 ---------------Time to First Token---------------- Mean TTFT (ms): 4399.20 Median TTFT (ms): 4333.35 P99 TTFT (ms): 8004.81 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 58.23 Median TPOT (ms): 21.78 P99 TPOT (ms): 747.79 ---------------Inter-Token Latency---------------- Mean ITL (ms): 20.06 Median ITL (ms): 15.28 P95 ITL (ms): 48.36 P99 ITL (ms): 96.99 Max ITL (ms): 969.61 ================================================== ``` #### 5.3.3 Multimodal (Image) Benchmark * **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment). * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model XiaomiMiMo/MiMo-V2.5 \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 25.73 Total input tokens: 661 Total input text tokens: 631 Total input vision tokens: 30 Total generated tokens: 4220 Total generated tokens (retokenized): 0 Request throughput (req/s): 0.39 Input token throughput (tok/s): 25.69 Output token throughput (tok/s): 164.03 Peak output token throughput (tok/s): 1.00 Peak concurrent requests: 2 Total token throughput (tok/s): 189.73 Concurrency: 1.00 Accept length: 2.94 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2570.74 Median E2E Latency (ms): 2411.92 P90 E2E Latency (ms): 3711.62 P99 E2E Latency (ms): 4949.74 ---------------Time to First Token---------------- Mean TTFT (ms): 0.00 Median TTFT (ms): 0.00 P99 TTFT (ms): 0.00 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.31 Median TPOT (ms): 6.17 P99 TPOT (ms): 17.18 ---------------Inter-Token Latency---------------- Mean ITL (ms): 0.00 Median ITL (ms): 0.00 P95 ITL (ms): 0.00 P99 ITL (ms): 0.00 Max ITL (ms): 0.00 ================================================== ``` ### 5.4 Multi-Layer EAGLE Acceptance Profile — MiMo-V2.5-Pro Pro's 3-layer MTP behaves very differently on natural text vs uniform-random token streams. The §5.2 benchmarks use `random 1024/1024`, which collapses accept-rate; this section measures the same server on GSM8K so the acceptance number is comparable to real workloads. **Test Environment:** * Hardware: NVIDIA B200 GPU (8×) * Model: `XiaomiMiMo/MiMo-V2.5-Pro` (FP8) * Tensor Parallelism: 8 (single-node, `--moe-runner-backend flashinfer_trtllm`, `--attention-backend fa4`, `--mem-fraction-static 0.8`, `--swa-full-tokens-ratio 0.1`) * Recipe: 3-layer EAGLE — `--enable-multi-layer-eagle --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4` (top-1, max accept length 4) **Benchmark Command:** ```shell Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --model XiaomiMiMo/MiMo-V2.5-Pro \ --num-examples 200 \ --num-threads 4 ``` The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result. The `accept_rate` and `accept_length` rows below are not part of `run_eval`'s own output — they were aggregated from the server-side `Decode batch ... accept rate: X accept len: Y` log lines emitted during the GSM8K run (307 batches total). | Workload | accept\_rate | accept\_length (max = 4) | | - | - | - | | GSM8K (natural text) | **0.755** | **3.27** | | `random 1024/1024` (reference) | 0.13–0.27 | \~1.x | GSM8K Score: **0.97** (194 / 200), output throughput ≈ 635 tok/s end-to-end on this single-server run. The accept-rate gap is intrinsic to MTP-style speculative decoding: the draft model is trained on natural-language token distributions and has no useful signal on uniform-random byte sequences. Workloads with structure (chat, code, reasoning traces) should expect the GSM8K-class number; the random-prompt baseline in §5.2 is a worst case for draft acceptance. ### 5.5 Long-Context Prefill & MTP Decode — MiMo-V2.5-Pro (Reference) Reference numbers from the [day0 enablement PR](https://github.com/sgl-project/sglang/pull/23808), collected on a 2-node Hopper deployment with the **EP=16, DP=2, TP=16** recipe (`--moe-a2a-backend deepep`, `--attention-backend fa3`, `--enable-multi-layer-eagle`). The setup, parallelism, and benchmark methodology all differ from §5.2 (Blackwell TP=8 with `random 1024/1024`), so treat these as a separate operating point — long-context prefill scaling and the MTP decode speedup — rather than a comparison against §5.2. **Test Environment:** * Hardware: NVIDIA Hopper GPU (2 nodes × 8 GPUs, GPU SKU intentionally not disclosed) * Model: `XiaomiMiMo/MiMo-V2.5-Pro` (FP8) * Parallelism: `--tp 16 --attn-dp-size 2 --ep 16 --moe-dense-tp-size 1` * Recipe: Hopper EP16 (DeepEP + EAGLE multi-layer MTP) #### 5.5.1 Long-Context Prefill Throughput Test setting: `chunked_prefill_size=32K`, `random_output_len=1`, cache flushed before every run. For input lengths ≥ 512K the workload was split into two requests routed to distinct DP ranks and the per-node throughput was read from `bench_serving` output. * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model XiaomiMiMo/MiMo-V2.5-Pro \ --host 0.0.0.0 \ --port 30000 \ --dataset-name random \ --random-input-len \ --random-output-len 1 \ --random-range-ratio 1.0 \ --flush-cache \ --seed 12345 \ --num-prompts 10000 ``` * **Test Results** — single-node prefill throughput, cache-miss: | Input length | Output length | Single-node prefill throughput | | - | - | - | | 4K | 1 | 30.80K tok/s | | 8K | 1 | 30.65K tok/s | | 16K | 1 | 29.85K tok/s | | 32K | 1 | 28.60K tok/s | | 64K | 1 | 26.65K tok/s | | 128K | 1 | 23.00K tok/s | | 256K | 1 | 17.90K tok/s | | 512K | 1 | 11.30K tok/s | | 768K | 1 | 9.40K tok/s | | 1M | 1 | 7.30K tok/s | Prefill throughput stays within \~10% of peak from 4K up to 32K and degrades gracefully past 128K, confirming the hybrid SWA+GA attention works correctly at 1M context. #### 5.5.2 Decode Throughput — MTP Speedup Test setting: fixed **16K input / 1K output**, varying batch size per DP rank, with and without the 3-layer MTP module. `MTP accept length` is the average number of draft tokens accepted per step under EAGLE speculative decoding. **TPS** below is per-request output tokens/sec (i.e. single-user perceived speed); the rightmost column is aggregated single-node decode throughput (= TPS × batch size). * **Test Results** — single-node decode throughput: | BS per DP rank | MTP | MTP accept length | Per-request TPS | Single-node decode throughput | | - | - | - | - | - | | 64 | disabled | - | 29.3 | 1875 tok/s | | 64 | 3-layer | 3 | 60.5 | 3873 tok/s | | 64 | 3-layer | 4 | 79.7 | 5103 tok/s | | 96 | disabled | - | 26.7 | 2564 tok/s | | 96 | 3-layer | 3 | 50.4 | 4840 tok/s | | 96 | 3-layer | 4 | 64.8 | 6225 tok/s | **Summary — MTP on / off:** | BS per DP rank | Without MTP | 3-layer MTP, accept=3 | 3-layer MTP, accept=4 | | - | - | - | - | | 64 | 1875 tok/s | 3873 tok/s (2.07×) | 5103 tok/s (2.72×) | | 96 | 2564 tok/s | 4840 tok/s (1.89×) | 6225 tok/s (2.43×) | The 3-layer MTP module delivers \~2× decode throughput at accept length 3 and \~2.5–2.7× at accept length 4 — the same order of magnitude as the "2–3× decode speedup" guidance in §3.2. # MiMo-V2.6 Source: https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2.6 Deploy MiMo-V2.6-Flash and Pro with SGLang on NVIDIA H200 and B300 GPUs, using MXFP4 MoE weights, BF16 routing, DFlash decoding, and a 1M-token context window. ## Deployment The recipes below target **MiMo-V2.6-Flash on 4× H200 or B300** and **MiMo-V2.6-Pro on 8× H200 or B300**, each on a single node. Both use MXFP4 MoE weights, a BF16 MoE router, and DFlash speculative decoding. BF16 MoE routing and MXFP4 expert loading for MiMo V2.6 landed in [SGLang PR #40448](https://github.com/sgl-project/sglang/pull/40448) and are on `main`, so a recent nightly already carries them. Prepare an NVIDIA CUDA environment with FlashAttention 4, DeepGEMM, and DeepEP available. See the [installation guide](/docs/get-started/install) for platform prerequisites. Install from source. The commit below is the revision these recipes were captured at; build from `main` instead if you want later fixes. ```bash Command theme={null} git clone https://github.com/sgl-project/sglang.git sglang-mimo-v2.6 cd sglang-mimo-v2.6 git checkout 983e643854f15cf9ef4370a49dfd74b6af54c3e3 python3 -m pip install -e ./python ``` Then run the **Python** command from the panel below. Checkpoint paths refer to directories on this host. Pull the official nightly image, which already includes the MiMo-V2.6 support: ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` Select **Flash** or **Pro** to generate its command. Each variant has one base recipe; **Balanced** identifies that recipe and does not imply a measured throughput or latency optimum. The `/model/...` checkpoint and DFlash paths in the generated command are editable defaults, not download locations — set them to your own paths under **⚙ Env** before launching. Use the panel's cURL example to send a request to port **30000** after the server is ready. ## Playground Experiment with reasoning and tool-call parsers, or disable speculative decoding for comparison. These overrides inherit the selected Flash or Pro recipe; they require separate validation. Parallelism and attention backends remain at the PR settings. ## 1. Model introduction **MiMo-V2.6-Flash** is an open-source foundation model developed by Xiaomi. Its Mixture-of-Experts architecture has **309B total parameters and 15B activated per token**, with hybrid attention for computational efficiency. It supports a **1M-token context window** and **native multimodal capabilities**, and is designed for coding, visual understanding, general-purpose assistance, and research in agentic workflows, including complex tasks that require many steps. **MiMo-V2.6-Pro** is Xiaomi's flagship foundation model, with **1.02T total parameters and 42B activated per token**, designed for demanding workloads. It also supports a **1M-token context window** and **native multimodal capabilities**, with an emphasis on coding, visual understanding, general-purpose assistance, research, and long-horizon agentic tasks.
Variant Total parameters Context window Recipe (H200 / B300)
MiMo-V2.6-Flash 309B (15B active) 1M tokens 4 GPUs, TP=EP=4
MiMo-V2.6-Pro 1.02T (42B active) 1M tokens 8 GPUs, TP=EP=8
**License:** MIT, continuing the MiMo-V2.5 licensing as confirmed by the model team. **Architecture references:** The [MiMo-V2.6-Flash-RL](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL) and [MiMo-V2.6-Pro-RL](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL) model cards document this generation directly — the hybrid sliding-window/global attention backbone, the omnimodal encoders, and the published evaluation scores — and each repository also ships the MiMo-V2.6 technical report. Both checkpoints declare `store_dtype: mxfp4` alongside `moe_router_dtype: bfloat16` in `config.json`, which is the expert-only MXFP4 quantization and BF16 routing these recipes depend on, and both carry the paired DFlash drafter in a `dflash/` subdirectory. ## 2. Configuration tips ### Parallelism and precision * **Flash:** `--tp 4 --ep 4`; **Pro:** `--tp 8 --ep 8`. Both keep `--dp 1 --pp-size 1`. TP and EP use the same GPU ranks; do not multiply them to determine the GPU count. * **MoE compute and communication:** Keep `--moe-runner-backend deep_gemm` on B300 and `--moe-runner-backend marlin` on H200, with `--moe-a2a-backend deepep --deepep-mode auto`, together with `--moe-dense-tp-size 1` and `--enable-dp-lm-head`, as in the PR. * **Precision:** MXFP4 describes the MoE expert weights, not the entire model. BF16 router weights are selected by the checkpoint's `moe_router_dtype`; the support commit computes router output logits in FP32. The PR relies on checkpoint quantization metadata and does not add a `--quantization` override. * **Attention:** Keep `--attention-backend fa4` and `--mm-attention-backend fa4` on both H200 and B300. The recipe also enables `--mm-enable-dp-encoder` for multimodal encoding. ### Context length and memory Both commands set `--context-length 1048576`. This is the configured per-request context limit; it does not establish that 64 simultaneous 1M-token requests fit in memory. Flash uses `--mem-fraction-static 0.6 --swa-full-tokens-ratio 0.03`; Pro uses `--mem-fraction-static 0.7 --swa-full-tokens-ratio 0.08`. The memory fraction budgets weights and KV cache, while the SWA ratio controls sliding-window versus full-attention KV allocation. Preserve these model-specific values for the base recipe. The shared prefill settings are `--chunked-prefill-size 49152 --max-prefill-tokens 65536`. Both commands cap running requests and decode CUDA-graph batch size at **64**, disable prefill CUDA graphs, and use page size **1**. `--enable-cache-report` and `--log-level-http warning` retain the PR's reporting settings. ### DFlash and caching The base recipe uses `--speculative-algorithm DFLASH --speculative-num-draft-tokens 8`. Supply the DFlash checkpoint paired with the selected target model; the draft path is independently editable and does not have to be a subdirectory of the target checkpoint. At the pinned commit, DFlash on NVIDIA GPUs requires PP=1 and rejects DP-attention. Do not set `--attn-dp-size` above 1 or copy EAGLE-specific settings from the V2.5 cookbook. ## 3. Reasoning and tool calling Both launch commands enable `--reasoning-parser mimo --tool-call-parser mimo`. The MiMo reasoning parser uses the request's `chat_template_kwargs.enable_thinking` setting: set it to `true` for reasoning or `false` for a direct answer. The command panel's cURL example enables reasoning explicitly. For OpenAI-compatible clients, read reasoning from `choices[0].message.reasoning_content`, the final answer from `choices[0].message.content`, and structured tool requests from `choices[0].message.tool_calls`. Preserve both reasoning and answer fields when inspecting responses. Use the checkpoint's generation defaults unless your application needs explicit sampling settings. Runtime throughput, latency, accuracy, and example responses for V2.6 will be added when measurements from these checkpoints are available. # Overview Source: https://docs.sglang.io/cookbook/autoregressive/intro Practical guides for deploying and using large language models and vision language models with SGLang. # Autoregressive Model Benchmark Documentation Source: https://docs.sglang.io/cookbook/base/benchmarks/autoregressive_model_benchmark `sglang.bench_serving` is a command-line tool designed to benchmark the online serving throughput and latency of Large Language Models (LLMs) and Vision Language Models(VLMs). It supports various backends (`SGLang`, `vLLM`, etc.) and offers flexible configurations for request rates, dataset types, and profiling. ## 1. Quick Start ### Basic Usage (Random Data) Run a benchmark using randomly generated prompts with a local SGLang server. ```bash Command theme={null} python -m sglang.bench_serving --backend sglang --port 30000 --dataset-name random --num-prompts 100 ``` ### Real-World Data (ShareGPT) Run a benchmark using the ShareGPT dataset with a specific request rate. ```shell Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --dataset-name sharegpt \ --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json \ --num-prompts 1000 \ --request-rate 10 ``` ## 2. Parameter Reference ### 2.1 Backend & Server Configuration These parameters define the target server and the inference engine being used.
Parameter Description
`--backend` **Required.** Specifies the backend engine. Options: `sglang`, `sglang-native`, `sglang-oai`, `sglang-oai-chat`, `vllm`, `vllm-chat`, `lmdeploy`, `lmdeploy-chat`, `trt`, `gserver`, `truss`.
`--base-url` The API base URL (if not using specific host/port flags).
`--host` Server hostname. Default: `0.0.0.0`.
`--port` Server port. If not set, it defaults to the specific backend's standard port.
`--model` Model name or path. If unset, it queries `/v1/models` for configuration.
`--served-model-name` The model name used in the API request body. Defaults to the value of `--model`.
`--tokenizer` Path or name of the tokenizer. Defaults to the model configuration.
### 2.2 Dataset Configuration Controls the source of the prompts used for benchmarking.
Parameter Description
`--dataset-name` The type of dataset. Options: `sharegpt`, `custom`, `random`, `random-ids`, `generated-shared-prefix`, `mmmu`, `image`, `mooncake`.
`--dataset-path` File path to the dataset (e.g., local JSON file for ShareGPT).
`--num-prompts` Total number of prompts to process. Default: `1000`.
`--seed` Random seed for reproducibility.
`--tokenize-prompt` Uses integer IDs instead of strings for inputs. Useful for precise length control.
### 2.3 Input/Output Length Control Parameters to control the shape of requests (context length and generation length). #### For Random/Image Datasets: * `--random-input-len`: Number of input tokens per request. * `--random-output-len`: Number of output tokens per request. * `--random-range-ratio`: Range ratio for sampling input/output lengths. #### For ShareGPT Dataset: * `--sharegpt-output-len`: Overrides the output length defined in the dataset for each request. * `--sharegpt-context-len`: Max context length. Requests exceeding this are dropped. #### General Request Modifiers: * `--extra-request-body`: Appends a JSON object to the request payload (e.g., \{"key": "value"}). Useful for passing sampling parameters. * `--prompt-suffix`: A string suffix appended to all user prompts. * `--disable-ignore-eos`: If set, the model will stop generation upon hitting the EOS token (benchmarks usually ignore EOS to force max generation length). * `--apply-chat-template`: Applies the model's chat template to the input. ### 2.4 Traffic & Concurrency Controls how fast requests are sent to the server.
Parameter Description
`--request-rate` Requests per second (RPS). If `inf` (default), all requests are sent immediately (burst). Otherwise, arrival times follow a Poisson process.
`--max-concurrency` The maximum number of active requests allowed at once. Even if `request-rate` is high, the client will hold back requests if this limit is reached.
`--warmup-requests` Number of requests to run before the actual measurement begins to warm up the server.
`--flush-cache` Flushes the server cache before starting the benchmark.
### 2.5 Output & Logging
Parameter Description
`--output-file` Path to save the results in JSONL format.
`--output-details` Includes detailed metrics in the output.
`--print-requests` Prints requests to stdout as they are sent (useful for debugging).
`--disable-tqdm` Hides the progress bar.
`--disable-stream` Disables streaming mode (waits for full response).
`--return-logprob` Requests logprobs from the server.
`--tag` An arbitrary string tag added to the output file for identification.
### 2.6 Advanced #### 2.6.1 Image / Multi-modal Only applicable when --dataset-name is set to image. * `--image-count`: Number of images per request. * `--image-resolution`: Resolution (e.g., 1080p, 4k, or custom 1080x1920). * `--image-format`: jpeg or png. * `--image-content`: random (noise) or blank. #### 2.6.2 LoRA Benchmarking Used to simulate multi-LoRA serving scenarios. * `--lora-name`: A list of LoRA adapter names (e.g., `--lora-name` adapter1 adapter2). * `--lora-request-distribution`: How requests are assigned to adapters: * `uniform`: Equal probability. * `distinct`: New adapter for every request. * `skewed`: Follows a Zipf distribution (simulating hot/cold adapters). * `--lora-zipf-alpha`: The alpha parameter for the Zipf distribution (if `skewed` is used). #### 2.6.3 Profiling Tools for deep performance analysis. * `--profile`: Enables Torch Profiler (Requires `SGLANG_TORCH_PROFILER_DIR` env var on server). * `--plot-throughput`: Generates throughput/concurrency plots (requires `termplotlib` and `gnuplot`). * `--profile-activities`: Activities to profile (CPU, GPU, CUDA\_PROFILER). * `--profile-num-steps`: Number of steps to profile. * `--profile-by-stage` / `--profile-stages`: Profile specific processing stages. #### 2.6.4 PD Disaggregation For benchmarking Prefill-Decode (PD) separated architectures. * `--pd-separated`: Enable PD disaggregation benchmarking. * `--profile-prefill-url`: URL(s) of prefill workers for profiling. * `--profile-decode-url`: URL(s) of decode workers for profiling. Note: In PD mode, `prefill` and `decode` must be profiled separately. ### 2.7 Specialized Datasets #### 2.7.1 Generated Shared Prefix (GSP): Designed to test system prompt caching/prefix sharing performance. * `--gsp-num-groups`: Number of unique system prompts. * `--gsp-prompts-per-group`: How many user questions share the same system prompt. * `--gsp-system-prompt-len`: Length of the shared prefix. * `--gsp-fast-prepare`: Skips some statistics calculation for faster startup. #### 2.7.2 Mooncake Designed for trace replay. * `--mooncake-slowdown-factor`: Slows down the trace replay (e.g., 2.0 = 2x slower). * `--mooncake-num-rounds`: Number of conversation rounds (supports multi-turn). * `--use-trace-timestamps`: Schedules requests based on timestamps found in the trace file. ## 3. Metrics After running the benchmark, the tool generally reports: * `E2E` (End-to-End Latency): The total time from sending the request to receiving the final token. * `TTFT` (Time To First Token): The time between sending the request and seeing the first word appear. This represents the Prefill time (processing the image and text prompt). * `TPOT` (Time per Output Token): The average time it takes to generate one token (excluding the first one). This is calculated per request. * `ITL` (Inter-Token Latency): The time gap between two distinct streaming packets. While TPOT is an average, ITL measures the "jitter" or smoothness of the stream. # Diffusion Models Benchmark Documentation Source: https://docs.sglang.io/cookbook/base/benchmarks/diffusion_model_benchmark `sglang.multimodal_gen.benchmarks.bench_serving` is a command-line tool designed to benchmark the online serving throughput and latency of diffusion models. It selects the image or video API from the requested task and offers flexible configurations for request rates, dataset types, and profiling. ## 1. Quick Start ### 1.1 Benchmarking in Low Concurrency Run a benchmark on a local server (port 30000) generating 1 videos/images from the `vbench` dataset. ```bash Command theme={null} # For text to video: such as Wan2.2-T2V-A14B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 # For image to video: such as Wan2.2-I2V-A14B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-video --num-prompts 1 --max-concurrency 1 # For image-text to video: such as Wan2.2-TI2V-5B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-video --num-prompts 1 --max-concurrency 1 # For text to image: such as Qwen-Image python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 # For image-text to image: such as Qwen-Image-Edit python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-image --num-prompts 1 --max-concurrency 1 ``` ### 1.2 Benchmarking in High Concurrency Run a benchmark on a local server (port 30000) generating 20 videos/images from the `vbench` dataset. ```bash Command theme={null} # For text to video: such as Wan2.2-T2V-A14B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-video --num-prompts 20 --max-concurrency 20 # For image to video: such as Wan2.2-I2V-A14B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-video --num-prompts 20 --max-concurrency 20 # For image-text to video: such as Wan2.2-TI2V-5B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-video --num-prompts 20 --max-concurrency 20 # For text to image: such as Qwen-Image python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 # For image-text to image: such as Qwen-Image-Edit python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-image --num-prompts 20 --max-concurrency 20 ``` ## 2. Parameter Reference ### 2.1 Connection Settings
Argument Default Description
`--base-url` `None` Base URL of the server (e.g., `http://localhost:30000`). If specified, this overrides `--host` and `--port`.
`--host` `None` The server host (e.g., `127.0.0.1`).
`--port` `None` The server port.
`--model` `None` Model name or path.
### 2.2 Workload & Task Configuration
Argument Choices Description
`--task` `text-to-video`, `image-to-video`, `text-to-image`, `image-to-image`, `video-to-video` Defines the generation task when it cannot be inferred from the model metadata.
`--dataset` `vbench`, `random` The source of prompts/inputs.
`--dataset-path` `None` (Optional) Path to a local dataset file if not using built-in presets.
`--num-prompts` `None` The total number of prompts/requests to execute during the benchmark.
### 2.3 Generation Parameters
Argument Description
`--width` The target width for the generated image or video.
`--height` The target height for the generated image or video.
`--num-frames` Number of frames to generate (Specific to Video backends).
`--fps` Frames Per Second configuration (Specific to Video backends).
### 2.4 Concurrency & Load Control
Argument Description
`--request-rate` The number of requests initiated per second. If set to `inf`, all requests are sent immediately (burst). If set to a number, request arrival times follow a Poisson process.
`--max-concurrency` The maximum number of requests allowed to execute simultaneously. This simulates a semaphore or upstream limit. Even if `request-rate` is high, the actual processing rate is capped by this value.
### 2.5 Logging & Output
Argument Description
`--output-file` Path to save the benchmark metrics (JSON format).
`--disable-tqdm` If set, disables the progress bar in the console.
## 3. Metrics * `Request Throughput` (req/s), Output Throughput (tok/s) * `Latency Mean` (ms): Time to Per Step * `Peak Memory Max` (ms): Max Memory Usage during running # Server Arguments Source: https://docs.sglang.io/cookbook/base/reference/server_arguments This guide explains the parallelism configuration fields used in SGLang model configurations and how they map to SGLang server command-line arguments. ## Quick Reference
Config Field SGLang CLI Argument Description
`tp` `--tp-size`, `--tensor-parallel-size` Tensor Parallelism - splits model across GPUs
`dp` `--dp-size`, `--data-parallel-size` Data Parallelism - runs multiple model replicas
`ep` `--ep-size`, `--expert-parallel-size`, `--ep` Expert Parallelism - distributes MoE experts
`attn_dp_size` `--attn-dp-size`, `--attention-data-parallel-size` Attention Data Parallelism - data-parallel attention groups inside the TP group, TP for FFN (hybrid)
`enable_dp_attention` `--enable-dp-attention` Deprecated. Use --attn-dp-size \ instead of --dp-size \ --enable-dp-attention.
# Anima Source: https://docs.sglang.io/cookbook/diffusion/CircleStone/Anima Deploy Anima Base v1.0 with SGLang Diffusion for anime and illustration generation, using native components and single- or multi-GPU execution. ## 1. Quick start On Linux with NVIDIA CUDA, use a nightly Docker image containing this integration, or install from source, following the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). Run the commands below inside that environment. Use the official `circlestone-labs/Anima-Base-v1.0-Diffusers` checkpoint. SGLang reads its `modular_model_index.json` directly; no checkpoint conversion or newer Diffusers runtime is required. Start with **Resident + Compiled** for repeated generation on RTX 5090, DGX Spark, or H200. Select **Eager** for shorter startup or frequent shape changes. Spark's 128 GB is unified memory shared with the CPU, not dedicated VRAM. See the [measurements and tradeoffs](#5-measured-tuning) below. ## 2. Model capabilities [Anima](https://huggingface.co/circlestone-labs/Anima) is CircleStone Labs and Comfy Org's text-to-image model for anime and illustration. It combines a Cosmos Predict2 transformer with Qwen3 text encoding, a learned T5-token conditioner, and a Qwen-Image VAE. Prompts can combine tags with natural-language descriptions. This integration targets **Base v1.0 text-to-image**. It does not claim support for the separate Aesthetic or Turbo checkpoints, single-file ComfyUI weights, or image editing. The checkpoint uses the CircleStone Labs Non-Commercial License; review the [official license](https://huggingface.co/circlestone-labs/Anima/blob/main/LICENSE.md) before deployment. ## 3. Sampling Defaults are 1024 x 1024, 30 steps, guidance scale 4, and an empty negative prompt. Width and height should be divisible by 16. `max_sequence_length` defaults to 512 and accepts 1 through 4096. The text conditioner pads short sequences to 512 tokens; these padding positions remain part of the transformer's cross-attention, matching the official implementation. For offline generation: ```bash Generate theme={null} sglang generate \ --model-path circlestone-labs/Anima-Base-v1.0-Diffusers \ --prompt "masterpiece, best quality, safe, watercolor landscape, a quiet seaside village at sunset" \ --seed 42 \ --save-output ``` Use the same seed, generator device, dimensions, scheduler settings, and precision when comparing runtimes. Latents and scheduler updates remain FP32; the transformer, text components, and VAE default to BF16. Different attention kernels can produce small floating-point differences that accumulate during denoising. ## 4. Runtime features The pipeline reuses SGLang's native Qwen3 encoder, Qwen-Image VAE, component loaders, denoising loop, and residency management. Anima's transformer and text conditioner are native modules, not wrappers around Diffusers models. For multi-GPU execution, choose TP to shard transformer weights or Ulysses/Ring to shard image tokens. The transformer has 16 heads, so `tp_size * ulysses_degree` must divide 16. CFG parallelism additionally splits conditional and unconditional denoising and requires guidance greater than 1. The total GPU count must match the selected parallel topology. CPU and layerwise offload trade memory for transfers. The additional `text_conditioner` component accepts the same residency controls as other native components. Cache-DiT and quantized attention are approximate optimizations; assess image quality for your prompts before enabling them. See [performance optimization](/docs/sglang-diffusion/performance-optimization) for the shared controls. H200 functional checks cover TP, Ulysses, Ring, CFG parallelism, encoder folding, parallel tiled VAE decode, layerwise offload, FlashAttention, Torch SDPA, SageAttention, Cache-DiT, and breakable CUDA graphs. These checks are not a quality guarantee for approximate optimizations. Combined TP and SP, multi-node execution, non-NVIDIA devices, and third-party LoRAs remain unverified. Breakable CUDA graphs reuse the conditioner's actual sequence length, without additional text-bucket padding. Requests with uncaptured shapes fall back to eager execution. Standard short prompts use the same 512-token conditioning shape. ## 5. Measured tuning For repeated requests, compilation was faster than eager execution on all three tested platforms. The picker keeps eager available because initial compilation and recompilation for new shapes can take minutes and use additional CPU memory. No CPU offload, quantized attention, or Cache-DiT is enabled in the recommended recipes. The following are warm, sequential HTTP request medians: 1024 x 1024, 30 steps, CFG 4, one image, CPU generator, and PNG/base64 output. Startup is excluded. | GPU | Recommended recipe | Original eager | Recommended | Peak allocated | | - | - | - | - | - | | RTX 5090, 32 GB | cuDNN SDPA + compile, tiled VAE | 6.99 s | 6.10 s | 5.7 GiB | | DGX Spark, 128 GB unified | Torch SDPA + compile, tiled VAE | 26.03 s | 17.41 s | 5.6 GiB | | H200, 141 GB | FlashAttention + compile, untiled VAE | 3.22 s | 2.22 s | 10.2 GiB | Measurements used five timed requests after warmup, PyTorch 2.13.0+cu130, and the native Anima implementation at `214c9a48bb1` (Spark used `7f3f903f039`, with identical runtime code). Results are workload-specific, not throughput-at-saturation measurements. The original eager baseline uses Torch SDPA on 5090/Spark and FlashAttention on H200, with VAE tiling enabled. Peak allocated memory excludes the CUDA context, reserved pool, and compilation CPU memory; it is not total device usage. On **two NVLink-connected H200s**, Auto selects CFG parallelism instead of TP: the compiled, untiled recipe measured **1.28 s**. Untiled decoding trades about 4-5 GiB more allocated memory for lower latency and is verified at 1024 x 1024, one output. The tiled recipes additionally passed repeated 512 x 512 and 1536 x 1536 requests, plus two outputs at 1024 x 1024. Use **Tiled** for that verified scope. At 512 x 512, two-GPU CFG eager was faster than compiled; more GPUs or compilation are not universally better. The recommended recipes change floating-point kernels and, on H200, VAE tiling. They are **not pixel-identical to eager**: across three fixed prompt/seed pairs, the single-GPU recipes above measured PSNR 21.95-35.44 dB and SSIM 0.851-0.969 against their same-device eager outputs. These measure output differences, not perceptual quality guarantees. Use eager with the same backend and tiling settings when reproducing an eager reference. Breakable CUDA graphs were near parity at 1024 x 1024 while consuming about 1.2 GiB more allocated GPU memory for a single captured shape. They are not the default. Cache-DiT can accelerate further, but changes the denoising computation; validate it separately against your quality requirements. # Cosmos3 Source: https://docs.sglang.io/cookbook/diffusion/Cosmos/Cosmos3 Serve NVIDIA Cosmos3 image, video, sound, and action generation with SGLang Diffusion. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. Choose a [generation or policy request](#4-openai-compatible-requests) matching the checkpoint; the Request tab queries model metadata. Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). Docker is recommended for Linux GPU deployments; run the commands below inside the container or your installed Python environment. Cosmos3 guardrails are enabled by default when the package is available: ```bash Command theme={null} pip install "cosmos-guardrail==0.3.1" ``` `cosmos-guardrail` downloads gated NVIDIA guardrail weights, so pass a Hugging Face token if your environment needs one. If the package is not installed, SGLang skips Cosmos3 guardrails and logs a warning. To disable Cosmos3 guardrails for local experiments, set `SGLANG_DISABLE_COSMOS3_GUARDRAILS=1` before starting the server. There may be problems loading the Cosmos-1.0-Guardrail weights on Ascend NPU. If the *\_pickle.UnpicklingError* error occurs during startup, you should change `weight_only=True` to `weights_only=False` parameter in *cosmos\_guardrail/cosmos\_utils.py*: ``` #!/usr/bin/env bash COSMOS_GUARDRAIL_DIR="$(dirname "$(python -c 'import cosmos_guardrail; print(cosmos_guardrail.__file__)')")" sed -i 's/weights_only=True/weights_only=False/g' "$COSMOS_GUARDRAIL_DIR/cosmos_utils.py" ``` ## 2. Model capabilities [NVIDIA Cosmos3](https://huggingface.co/collections/nvidia/cosmos3) is an omnimodal world-model family spanning text/image/video generation, optional synchronized sound, and robot action prediction. Its main advantage is breadth: the same native SGLang pipeline can serve media-generation checkpoints and the DROID policy checkpoint without routing through an LLM sampler. Choose Nano for the broadest modality coverage and lower deployment cost, Super for the larger 64B image/video model, and a specialized checkpoint when only T2I or I2V is needed. Sound and action are checkpoint-specific heads, so they are not available from every Cosmos3 repository. | Model | Status | Notes | | - | - | - | | `nvidia/Cosmos3-Nano` | Supported | T2I, T2V, I2V, V2V, joint sound, and action | | `nvidia/Cosmos3-Super` | Supported | T2I, T2V, I2V, and V2V; use multi-GPU for the 64B checkpoint | | `nvidia/Cosmos3-Super-Text2Image` | Supported | T2I-specialized checkpoint | | `nvidia/Cosmos3-Super-Image2Video` | Supported | I2V-specialized checkpoint | | `nvidia/Cosmos3-Nano-Policy-DROID` | Supported | DROID policy action generation | | `nvidia/Cosmos3-Edge` | Supported | 4B dense model for T2I, T2V, I2V, V2V, and action generation | | `nvidia/Cosmos3-Edge-Policy-DROID` | Supported | 4B DROID policy action generation | | `nvidia/Cosmos3-Super-Text2Image-4Step` | Supported | 64B T2I checkpoint distilled to a fixed 4-step schedule | | `nvidia/Cosmos3-Super-Image2Video-4Step` | Supported | 64B I2V checkpoint distilled to a fixed 4-step schedule | Sound and action generation require the corresponding checkpoint heads. The pipeline reads the transformer and scheduler configs at startup, so Edge and distilled checkpoints do not require architecture-specific server flags. Non-distilled checkpoints use the flow-native `FlowUniPCMultistepScheduler`; distilled checkpoints use the fixed sigma schedule stored in the checkpoint. The default `flow_shift` is `3.0` for T2I, `10.0` for non-Edge video and all action modes, and `3.0` for Edge video modes. Distilled checkpoints bake the schedule into their sigmas and do not use a request-level `flow_shift`. ## 3. Serve Cosmos3 Serve `Cosmos3-Nano` directly from the Hugging Face model ID: ```bash Command theme={null} sglang serve \ --model-path nvidia/Cosmos3-Nano \ --num-gpus 1 ``` With `--performance-mode auto`, Cosmos3 Nano keeps its DiT and VAE resident when every selected GPU has at least 90 GiB available at startup. Other Cosmos3 checkpoints use a 120 GiB threshold. Below the applicable threshold, auto mode retains the conservative DiT component-offload policy. Cosmos3 runs one DiT per pipeline, so component offload above the threshold only pays to copy the weights out to host memory and back on every request. Serve `Cosmos3-Super` across multiple GPUs as shown below so each rank holds a shard of the weights. For `Cosmos3-Super`, split the model across multiple GPUs: ```bash Command theme={null} sglang serve \ --model-path nvidia/Cosmos3-Super \ --num-gpus 4 ``` The server also accepts the specialized `nvidia/Cosmos3-Super-Text2Image` and `nvidia/Cosmos3-Super-Image2Video` checkpoint IDs. ### Edge checkpoints `Cosmos3-Edge` is a 4B dense model and can be served on one GPU: ```bash Command theme={null} sglang serve \ --model-path nvidia/Cosmos3-Edge \ --num-gpus 1 ``` Edge is trained for 256p and 480p generation. Its default video configuration is `832x480` with `guidance_scale=5.0`; its default image configuration is `640x640` with `guidance_scale=7.0`. Supported sizes are `832x480`, `480x832`, `640x480`, `480x640`, `480x480`, `640x640`, `448x256`, `256x448`, and `256x256`. Serve the Edge DROID policy checkpoint with the same single-GPU configuration, replacing the model path with `nvidia/Cosmos3-Edge-Policy-DROID`. ### Distilled checkpoints The distilled Super checkpoints are 64B models. Use multiple GPUs unless the complete model and request workload fit on one GPU: ```bash Command theme={null} sglang serve \ --model-path nvidia/Cosmos3-Super-Text2Image-4Step \ --num-gpus 4 ``` For distilled I2V, replace the model path with `nvidia/Cosmos3-Super-Image2Video-4Step`. SGLang detects both checkpoints from `scheduler/scheduler_config.json`, uses the checkpoint's fixed four-step sigma schedule, and forces `guidance_scale=1.0`. Do not tune `num_inference_steps` or `flow_shift` for these checkpoints. ## 4. OpenAI-Compatible Requests ### Text to image Cosmos3 text-to-image uses `/v1/images/generations`. The default Cosmos3 image response is `b64_json`, matching vLLM-Omni's examples. ```bash Command theme={null} curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "prompt": "A warehouse robot folds a blue cloth on a clean workbench.", "size": "1280x720", "n": 1, "num_inference_steps": 35, "guidance_scale": 6.0, "flow_shift": 3.0, "seed": 0, "extra_body": { "use_resolution_template": false, "guardrails": true } }' ``` With a server running `nvidia/Cosmos3-Super-Text2Image-4Step`, omit the scheduler controls and use `guidance_scale=1.0`: ```bash Command theme={null} curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "prompt": "A warehouse robot folds a blue cloth on a clean workbench.", "size": "640x640", "n": 1, "guidance_scale": 1.0, "seed": 0, "extra_body": { "use_resolution_template": false, "guardrails": true } }' ``` ### Text to video with sound Use `/v1/videos` to create an asynchronous job, then poll the job and download the completed MP4. Set `generate_sound=true` to generate and mux a stereo 48 kHz audio track; omit it for a silent video. ```bash Command theme={null} job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \ --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \ --form-string "negative_prompt=blurry, distorted, low quality" \ --form-string "size=1280x720" \ --form-string "num_frames=81" \ --form-string "fps=24" \ --form-string "num_inference_steps=35" \ --form-string "guidance_scale=4.0" \ --form-string "flow_shift=10.0" \ --form-string "generate_sound=true" \ --form-string "seed=42" \ --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \ | python -c 'import json, sys; print(json.load(sys.stdin)["id"])') while true; do status=$(curl -sS "http://127.0.0.1:30010/v1/videos/${job_id}" \ | python -c 'import json, sys; print(json.load(sys.stdin)["status"])') [ "$status" = "completed" ] && break [ "$status" = "failed" ] && exit 1 sleep 1 done curl -sS -L "http://127.0.0.1:30010/v1/videos/${job_id}/content" \ -o cosmos3_t2v.mp4 ``` ### Image to video This mirrors the official `nvidia/Cosmos3-Nano` Hugging Face image-to-video example: ```python Python theme={null} import json import time from pathlib import Path import requests from huggingface_hub import snapshot_download base_url = "http://127.0.0.1:30010" model_dir = Path(snapshot_download("nvidia/Cosmos3-Nano")) asset_dir = model_dir / "assets" prompt = json.dumps(json.loads((asset_dir / "example_i2v_prompt.json").read_text())) negative_prompt = json.dumps( json.loads((asset_dir / "negative_prompt.json").read_text()) ) data = { "prompt": prompt, "negative_prompt": negative_prompt, "size": "1280x720", "num_frames": "189", "fps": "24", "num_inference_steps": "35", "guidance_scale": "6.0", "max_sequence_length": "4096", "flow_shift": "10.0", "seed": "1111", "extra_params": json.dumps( { "use_resolution_template": False, "use_duration_template": False, "guardrails": True, } ), } with (asset_dir / "example_i2v_input.jpg").open("rb") as image: response = requests.post( f"{base_url}/v1/videos", data=data, files={"input_reference": ("example_i2v_input.jpg", image, "image/jpeg")}, timeout=60, ) response.raise_for_status() video_id = response.json()["id"] while True: job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json() if job["status"] == "completed": break if job["status"] == "failed": raise RuntimeError(job.get("error") or "Video generation failed") time.sleep(1) response = requests.get(f"{base_url}/v1/videos/{video_id}/content", timeout=300) response.raise_for_status() Path("cosmos3_i2v.mp4").write_bytes(response.content) ``` For the distilled I2V checkpoint, use the same API with a server running `nvidia/Cosmos3-Super-Image2Video-4Step`. The recommended request is 480p and does not specify scheduler controls: ```bash Command theme={null} job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \ --form-string "prompt=A warehouse robot carefully places a blue box on a shelf." \ --form "input_reference=@first_frame.png;type=image/png" \ --form-string "size=832x480" \ --form-string "num_frames=189" \ --form-string "fps=24" \ --form-string "guidance_scale=1.0" \ --form-string "seed=42" \ --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \ | python -c 'import json, sys; print(json.load(sys.stdin)["id"])') ``` Poll and download this job with the same status and content endpoints used by the T2V example. ### Video to video Upload a source video with `video_reference`. Cosmos3 keeps latent frames `[0, 1]` by default and generates the remaining frames. Use `condition_frame_indexes` to select different latent frames, and `condition_video_keep` to take conditioning frames from the start or end of the source. ```bash Command theme={null} job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \ --form-string "prompt=A robotic arm pours liquid into a glass on a white tabletop." \ --form "video_reference=@robot_pouring.mp4;type=video/mp4" \ --form-string "size=1280x704" \ --form-string "num_frames=45" \ --form-string "fps=24" \ --form-string "num_inference_steps=35" \ --form-string "guidance_scale=6.0" \ --form-string 'condition_frame_indexes=[0,1]' \ --form-string "condition_video_keep=first" \ | python -c 'import json, sys; print(json.load(sys.stdin)["id"])') ``` Poll and download this job with the same status and content endpoints used by the T2V example. ### Action generation For DROID policy generation, start a single-GPU server with either the Nano or Edge policy checkpoint. Cosmos3 action generation does not currently support CFG or sequence parallelism. ```bash Command theme={null} sglang serve \ --model-path nvidia/Cosmos3-Nano-Policy-DROID \ --num-gpus 1 ``` Use `nvidia/Cosmos3-Edge-Policy-DROID` in the same command to serve the smaller 4B policy checkpoint. `policy` and `inverse_dynamics` return actions, so their canonical API is the synchronous `/v1/actions/generations` endpoint. The following request predicts a 16-step action chunk from one observation image. `action_horizon=16` maps to the model's `num_frames=17` convention. ```python Python theme={null} import base64 from pathlib import Path import requests image_b64 = base64.b64encode(Path("observation.png").read_bytes()).decode() response = requests.post( "http://127.0.0.1:30010/v1/actions/generations", json={ "input": { "task": "Put the pot to the left of the purple item.", "observation": { "image": {"b64_json": image_b64}, }, }, "parameters": { "action_mode": "policy", "action_horizon": 16, "domain_name": "droid_lerobot", "height": 480, "width": 832, "fps": 5, "num_inference_steps": 30, "guidance_scale": 1.0, "seed": 42, }, }, timeout=300, ) response.raise_for_status() action = response.json()["data"][0]["action"] print(action["shape"], action["values"]) ``` Use `GET /v1/actions/metadata` to inspect the action modes, default horizon, padded action dimension, and accepted observation modalities. Msgpack requests and the `/v1/actions/realtime` websocket use the same action envelope. To batch policy observations inside one request, opt in with a bounded batch size: ```bash Command theme={null} sglang serve \ --model-path nvidia/Cosmos3-Nano-Policy-DROID \ --num-gpus 1 \ --batching-max-size 4 ``` Send one image per observation as a list or `[B, H, W, C]` uint8 array in `input.input_reference`, and either one prompt per image or one scalar prompt to broadcast across the batch. Batched prompts must currently tokenize to the same length because Cosmos3 GEN cross-attention does not mask padded text K/V. All items in one request share the domain, resolution, action horizon, and denoise settings. The standard action envelope returns one `data[i]` item per input, each with action shape `[H, D]`. For a compact msgpack response containing one `[B, H, D]` array, set `runtime.response_format="raw"` and read the top-level `actions` field. For JSON, `input_reference` can be a list of base64 image payloads. For msgpack, it can be a packed uint8 numpy array directly: ```json JSON theme={null} { "input": { "prompt": ["pick up the block", "close the drawer"], "input_reference": [ {"b64_json": ""}, {"b64_json": ""} ] }, "parameters": { "action_mode": "policy", "domain_name": "droid_lerobot" } } ``` The batch size cannot exceed `--batching-max-size`; this keeps one request from bypassing the server's configured memory limit. Batching applies to `action_mode="policy"` only. A request seed controls the random stream for the whole batch, so a batched result is deterministic for that request but is not expected to be bit-exact with separately seeded B=1 requests. `inverse_dynamics` also uses `/v1/actions/generations`; set `action_mode="inverse_dynamics"` and pass an observation video URL or server-local path as `input.observation.video`. Select the embodiment head with `domain_name` or `domain_id`; set `raw_action_dim` explicitly when it cannot be inferred from the domain name. `forward_dynamics` is intentionally different: it consumes an action array and predicts video, so it remains on `/v1/videos`. Action-producing modes submitted to `/v1/videos` return HTTP 400 with the canonical action endpoint in the error message. ## 5. Cosmos3 Parameters Cosmos3 supports the standard SGLang video and image fields such as `size`, `num_frames`, `fps`, `num_inference_steps`, `guidance_scale`, `negative_prompt`, and `seed`. For distilled checkpoints, SGLang replaces `num_inference_steps` with the checkpoint's fixed four-step schedule and forces `guidance_scale=1.0`; negative-prompt CFG and request-level `flow_shift` do not apply. Top-level Cosmos3 request fields: * `max_sequence_length`: maximum text token length used by the Cosmos3 tokenizer. * `flow_shift`: per-request scheduler shift for non-distilled checkpoints. If omitted, SGLang uses `--flow-shift`, then the mode default (`3.0` for T2I, `10.0` for non-Edge video and all action modes, or `3.0` for Edge video). * `guidance_interval`: optional `[start, end]` noise interval for CFG. Non-distilled T2I defaults to `[400, 1000]`; video modes guide at every step. Cosmos3 omnimodal fields are accepted as extra JSON fields or multipart form fields: * `generate_sound`: generate a sound track whose duration follows `num_frames / fps`. * `sound_duration`: explicit sound duration in seconds; takes precedence over the derived duration. * `condition_frame_indexes`: V2V latent-frame indexes to keep from the source video; defaults to `[0, 1]`. * `condition_video_keep`: use the `first` or `last` source frames for V2V conditioning. * `action_mode`: `policy`, `forward_dynamics`, or `inverse_dynamics`. * `domain_name` / `domain_id`: select the action embodiment head. * `raw_action_dim`: number of active action dimensions; inferred for known domain names. * `action`: action array with shape `[T, D]`, required by `forward_dynamics`. * `action_fps`: action-token frame rate for temporal mRoPE; defaults to the video FPS. * `action_view_point`: viewpoint used in the structured action caption. * `action_normalization`: dataset normalization mode, such as `quantile`, `meanstd`, or `minmax`. Pass model-specific controls through `extra_body` with the OpenAI Python SDK. Raw JSON may keep them at the top level; multipart video requests should put them in the `extra_params` JSON object. The legacy image `extra_args` container remains accepted for compatibility, but new clients should use `extra_body`: * `use_duration_template`: whether to append SGLang's generated duration suffix to video prompts. * `use_resolution_template`: accepted for vLLM-Omni request compatibility. * `use_system_prompt`: whether to add the Cosmos3 system prompt to the chat template. * `guardrails` or `use_guardrails`: per-request guardrail toggle when the server started with guardrails enabled. ## 6. Run in ComfyUI # ERNIE-Image Source: https://docs.sglang.io/cookbook/diffusion/Ernie-Image/Ernie-Image Deploy ERNIE-Image and ERNIE-Image-Turbo with SGLang Diffusion. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [ERNIE-Image](https://huggingface.co/baidu/ERNIE-Image) is Baidu's text-to-image family with separate standard and Turbo checkpoints. The standard model is the quality-oriented path; Turbo is the lower-latency choice, and both load through SGLang's native `ErnieImagePipeline`. This integration currently targets text-only image generation rather than image editing or reference conditioning. Choose it for a straightforward single-GPU T2I deployment; use a dedicated editing model when preserving source-image structure is part of the task. | Model | Hugging Face model ID | Notes | | - | - | - | | ERNIE-Image | `baidu/ERNIE-Image` | Regular text-to-image checkpoint | | ERNIE-Image-Turbo | `baidu/ERNIE-Image-Turbo` | Turbo text-to-image checkpoint | ## 3. Serve the model The commands below target a single supported NVIDIA CUDA or AMD ROCm GPU. Start with `--performance-mode auto`; use `speed` only when the full pipeline fits comfortably on the selected GPU(s), and use `memory` when you need lower peak GPU memory. Serve ERNIE-Image: ```bash Command theme={null} sglang serve \ --model-path baidu/ERNIE-Image \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` Serve ERNIE-Image-Turbo: ```bash Command theme={null} sglang serve \ --model-path baidu/ERNIE-Image-Turbo \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` ## 4. Generate an image Use the OpenAI-compatible image generation API after the server starts: ```python Python theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://127.0.0.1:30010/v1") response = client.images.generate( model="baidu/ERNIE-Image-Turbo", prompt="A cinematic photo of a quiet lakeside cabin at sunrise", n=1, response_format="b64_json", ) image_bytes = base64.b64decode(response.data[0].b64_json) with open("ernie_image.png", "wb") as f: f.write(image_bytes) ``` ## 5. Configuration tips * ERNIE-Image is a text-to-image pipeline; do not pass `--image-path`. * `--performance-mode auto` keeps conservative defaults while preserving explicit user flags. * If the checkpoint includes a PE component, SGLang loads it automatically with the native Ministral3 runtime. Use `--layerwise-offload-components pe` when the local PE decoder needs to trade latency for lower GPU memory usage. * Treat FSDP, SP/Ulysses/Ring, and TP as explicit benchmark knobs. Measure the target resolution, step count, and GPU type before making them production defaults. ## 6. Run in ComfyUI # FLUX Source: https://docs.sglang.io/cookbook/diffusion/FLUX/FLUX Deploy FLUX diffusion models with SGLang - 12B/32B rectified flow transformers for high-quality text-to-image generation. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [FLUX](https://blackforestlabs.ai/) is Black Forest Labs' rectified-flow image model family. [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) is the smaller 12B text-to-image checkpoint; [FLUX.2-dev](https://huggingface.co/black-forest-labs/FLUX.2-dev) is a 32B model that adds instruction-based editing plus single- and multi-reference composition. FLUX is a strong default when prompt adherence, polished image quality, or reference consistency matters. The tradeoff is deployment weight: FLUX.2 needs substantially more memory than FLUX.1, and the dev checkpoints use the FLUX non-commercial license, so review the model license before production use. | Checkpoint | Best fit | Main limitation | | - | - | - | | `black-forest-labs/FLUX.1-dev` | High-quality text-to-image with the lighter FLUX deployment | No native multi-reference editing path | | `black-forest-labs/FLUX.2-dev` | Text-to-image, editing, and reference-guided composition in one model | 32B model with a larger memory footprint | ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration FLUX models are optimized for high-quality image generation. The recommended launch configurations vary by hardware and model version. Use the [command builder](#1-quick-start) above. ### 3.2 Configuration Tips See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. * `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. * `--num-gpus`: Number of GPUs to use * `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) * `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs) * `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP * `--ring-degree`: The degree of ring attention-style SP in USP ### 3.3 Breakable CUDA graph for FLUX.1-dev For repeated `quality=lossless` requests, FLUX.1-dev supports breakable CUDA graph (BCG) execution with the native backend. BCG captures DiT segments during startup and replays them for the warmed resolutions. It can reduce recurring host launch overhead without enabling `torch.compile`. The following configuration was validated on two NVIDIA H200 GPUs with BF16 weights, PyTorch 2.13, and CUDA 13.0. Set `HF_TOKEN` to a Hugging Face token with access to the checkpoint before downloading it. ```bash theme={null} CUDA_VISIBLE_DEVICES=0,1 sglang generate \ --model-path black-forest-labs/FLUX.1-dev \ --backend sglang \ --num-gpus 2 --tp-size 2 \ --component-residency dit=resident \ --enable-torch-compile false \ --enable-breakable-cuda-graph \ --warmup-resolutions 1024x1024 \ --quality exact \ --width 1024 --height 1024 \ --num-inference-steps 50 --guidance-scale 3.5 --seed 42 \ --prompt "A futuristic cyberpunk city at night, neon lights reflecting on wet streets" ``` Confirm `[Diffusion BCG] captured` in the log, then compare warmed request latency with eager execution on the same GPUs. Capture time and graph memory are additional startup costs. Add other served resolutions to `--warmup-resolutions`; a request with an uncaptured signature falls back to eager. FLUX.1-dev uses a fixed 512-token T5 conditioning sequence, so changing `--bcg-text-buckets` does not create additional prompt-length graphs. Its request-gated DiT fusions at `quality=high` and `lossless` cannot be combined with BCG: the runtime rejects those requests because the captured graph uses the lossless branches. FLUX.2 and quantized transformer overrides require separate validation. ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api). ### 4.1 Generate an Image ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:3000/v1") response = client.images.generate( model="black-forest-labs/FLUX.1-dev", prompt="A cat holding a sign that says hello world", size="1024x1024", n=1, response_format="b64_json", ) # Save the generated image image_bytes = base64.b64decode(response.data[0].b64_json) with open("output.png", "wb") as f: f.write(image_bytes) ``` ### 4.2 Advanced Usage #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit). **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path black-forest-labs/FLUX.1-dev ``` **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path black-forest-labs/FLUX.1-dev ``` #### 4.2.2 CPU Offload * `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. * `--vae-cpu-offload`: Use CPU offload for VAE. * `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". #### 4.2.3 Known LoRA examples Use `--lora-path` at startup or the [LoRA management API](/docs/sglang-diffusion/api/openai_api#lora-management) to load an adapter. Known FLUX examples include: * [`dvyio/flux-lora-simple-illustration`](https://huggingface.co/dvyio/flux-lora-simple-illustration) * [`XLabs-AI/flux-furry-lora`](https://huggingface.co/XLabs-AI/flux-furry-lora) * [`XLabs-AI/flux-RealismLora`](https://huggingface.co/XLabs-AI/flux-RealismLora) ## 5. Benchmark ### 5.1 Speedup Benchmark #### 5.1.1 Generate a image Test Environment: * Hardware: NVIDIA B200 GPU (1x) * Model: black-forest-labs/FLUX.1-dev * sglang diffusion version: 0.5.6.post2 **Server Command**: ```shell Command theme={null} sglang serve --model-path black-forest-labs/FLUX.1-dev --port 30000 ``` **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Model: black-forest-labs/FLUX.1-dev Dataset: vbench Task: text-to-image -------------------------------------------------- Benchmark duration (s): 50.97 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.02 Latency Mean (s): 50.9681 Latency Median (s): 50.9681 Latency P99 (s): 50.9681 -------------------------------------------------- Peak Memory Max (MB): 27905.19 Peak Memory Mean (MB): 27905.19 Peak Memory Median (MB): 27905.19 ============================================================ ``` **Server Command**: ```shell Command theme={null} #One A3 Series card has 2 npu chips sglang serve --tp-size 2 --sp-degree 1 --model-path black-forest-labs/FLUX.1-dev --num-gpus 2 ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: black-forest-labs/FLUX.1-dev Dataset: vbench -------------------------------------------------- Benchmark duration (s): 16.30 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 Completed outputs: 1 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.06 Output throughput (outputs/s): 0.06 Latency Mean (s): 16.30 Latency Median (s): 16.30 Latency P90 (s): 16.30 Latency P95 (s): 16.30 Latency P99 (s): 16.30 -------------------------------------------------- Peak Memory Max (MB): 19972.00 Peak Memory Mean (MB): 19972.00 Peak Memory Median (MB): 19972.00 ------------------------------------------------------------ ``` #### 5.1.2 Generate images with high concurrency **Server Command** : ```shell Command theme={null} sglang serve --model-path black-forest-labs/FLUX.1-dev --port 30000 ``` **Benchmark Command** : ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 ``` **Result** : ```text Output theme={null} ================= Serving Benchmark Result ================= Model: black-forest-labs/FLUX.1-dev Dataset: vbench Task: text-to-image -------------------------------------------------- Benchmark duration (s): 111.79 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 -------------------------------------------------- Request throughput (req/s): 0.18 Latency Mean (s): 67.0646 Latency Median (s): 66.9691 Latency P99 (s): 110.8949 -------------------------------------------------- Peak Memory Max (MB): 27917.19 Peak Memory Mean (MB): 27916.59 Peak Memory Median (MB): 27917.19 ============================================================ ``` **Server Command** : ```shell Command theme={null} #One A3 Series card has 2 npu chips sglang serve --tp-size 2 --sp-degree 1 --model-path black-forest-labs/FLUX.1-dev --num-gpus 2 ``` **Benchmark Command** : ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 ``` **Result** : ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: black-forest-labs/FLUX.1-dev Dataset: vbench -------------------------------------------------- Benchmark duration (s): 300.85 Request rate: inf Max request concurrency: 20 Successful requests: 18/20 Completed outputs: 18 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.06 Output throughput (outputs/s): 0.06 Latency Mean (s): 155.16 Latency Median (s): 155.11 Latency P90 (s): 266.30 Latency P95 (s): 280.15 Latency P99 (s): 291.23 -------------------------------------------------- Peak Memory Max (MB): 19972.00 Peak Memory Mean (MB): 19972.00 Peak Memory Median (MB): 19972.00 ------------------------------------------------------------ ``` ## 6. Run in ComfyUI # Ideogram 4 Source: https://docs.sglang.io/cookbook/diffusion/Ideogram/Ideogram4 Deploy Ideogram 4 with SGLang Diffusion for high-aesthetic text-to-image generation. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. Set `HF_TOKEN` with access to the gated Ideogram repositories. Install `bitsandbytes>=0.46.1` and `accelerate>=1.1.0` for NF4 and the shared NF4 text encoder. * NVIDIA CUDA GPU. * SGLang installed with diffusion dependencies. * `bitsandbytes>=0.46.1` and `accelerate>=1.1.0` for the NF4 checkpoint and the fal variants' shared NF4 text encoder. * `HF_TOKEN` with access to the Ideogram 4 gated repositories. ## 2. Model capabilities [Ideogram 4](https://huggingface.co/ideogram-ai/ideogram-4-nf4) is a text-to-image family focused on polished composition and typography-aware generation. SGLang supports the official NF4 and FP8 checkpoints, Comfy-Org's NVFP4 transformer, and fal's distilled Fast and Instant variants. Choose NF4 when memory is the primary constraint, FP8/NVFP4 for supported accelerator deployments, Fast for a 20-step balance, and Instant for the 8-step latency target. The public repositories are gated, and the available releases are already quantized or distilled rather than a single full-precision baseline. | Variant | Hugging Face model ID | Notes | | - | - | - | | NF4 | `ideogram-ai/ideogram-4-nf4` | Official bitsandbytes NF4 checkpoint. Use this path first for low-memory deployment. | | FP8 | `ideogram-ai/ideogram-4-fp8` | Official FP8 checkpoint. | | NVFP4 | `Comfy-Org/Ideogram-4` | Comfy-Org NVFP4 transformer weights. SGLang loads non-transformer components from `ideogram-ai/ideogram-4-fp8`. | | Fast | `fal/ideogram-v4-fast` | 20-step, CFG-distilled, FP4-targeted floating checkpoint. Defaults to `V4_FAST_20`. | | Instant | `fal/ideogram-v4-instant` | 8-step, CFG- and timestep-distilled BF16 checkpoint. Defaults to `V4_INSTANT_8`. | The fal repositories contain only the transformer component. When either model ID is passed directly to `--model-path`, SGLang loads the text encoder, tokenizer, VAE, and scheduler from the `ideogram-ai/ideogram-4-nf4-diffusers` revision referenced by fal's model cards, then runs the distilled transformer without the unconditional branch. ## 3. Serve the model NF4: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path ideogram-ai/ideogram-4-nf4 \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` FP8: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path ideogram-ai/ideogram-4-fp8 \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` Comfy-Org NVFP4: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path Comfy-Org/Ideogram-4 \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` Use B200 or another Blackwell GPU for NVFP4. fal Fast: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path fal/ideogram-v4-fast \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` fal Instant: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path fal/ideogram-v4-instant \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` ### Tensor and sequence parallelism Both fal variants support native DiT tensor parallelism and Ulysses sequence parallelism. The following two-GPU layouts were validated at 1024×1024 on B200 GPUs for both `fal/ideogram-v4-fast` and `fal/ideogram-v4-instant`. TP2 shards the distilled DiT weights. The shared bitsandbytes NF4 text encoder is replicated because 4-bit row-parallel quantization states cannot be safely sharded: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path fal/ideogram-v4-instant \ --num-gpus 2 \ --tp-size 2 \ --port 30010 ``` Ulysses/SP2 shards the image-token sequence while keeping the DiT weights replicated: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path fal/ideogram-v4-instant \ --num-gpus 2 \ --ulysses-degree 2 \ --port 30010 ``` Replace the model ID with `fal/ideogram-v4-fast` to use the same layouts for Fast. TP2 and Ulysses/SP2 are the validated configurations; Ring SP is not yet validated for this pipeline. Ideogram 4 has 18 attention heads, so any other TP degree must divide 18. ### Layerwise offload If the distilled DiT does not fit in GPU memory, enable transformer layerwise offload. This keeps the shared NF4 text encoder on the GPU while streaming the DiT blocks from pinned CPU memory. It reduces peak VRAM at the cost of additional latency: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path fal/ideogram-v4-instant \ --num-gpus 1 \ --dit-layerwise-offload \ --layerwise-offload-components transformer \ --port 30010 ``` The same option applies to Fast. Layerwise offload was validated with a real Instant generation; TP2 and Ulysses/SP2 were validated separately without offload. The released Fast weights are stored in a floating-point pre-pack form and can be loaded directly, but fal trained them with QAD for an NVFP4 execution path. SGLang does not currently quantize dense NVIDIA DiTs to NVFP4 at load time, so direct Fast inference bypasses the intended quantization path and may be visibly degraded. Treat direct floating-point Fast inference as compatibility/testing support, not the production-quality path. The released Instant checkpoint is BF16 and is the recommended local checkpoint today. ## 4. Generate an image ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:30010/v1") response = client.images.generate( model="ideogram-ai/ideogram-4-nf4", prompt="A cinematic poster of a quiet bookstore at dusk with elegant hand-lettered signage", size="1024x1024", n=1, response_format="b64_json", extra_body={"preset": "V4_QUALITY_48", "seed": 0}, ) image_bytes = base64.b64decode(response.data[0].b64_json) with open("ideogram4.png", "wb") as f: f.write(image_bytes) ``` fal's hosted endpoint expands natural-language prompts, but these local checkpoints expect Ideogram 4's structured JSON caption format. Pass a complete caption as the API `prompt`, for example: ```python Example theme={null} import json prompt = json.dumps( { "high_level_description": ( "A bold typographic poster centered on the exact words INSTANT BY FAL, " "printed in black and electric orange on warm white paper." ), "compositional_deconstruction": { "background": "Warm white textured paper with generous negative space.", "elements": [ { "type": "text", "text": "INSTANT BY FAL", "desc": "Large uppercase geometric sans-serif lettering, precisely centered.", } ], }, }, ensure_ascii=False, separators=(",", ":"), ) response = client.images.generate( model="fal/ideogram-v4-instant", prompt=prompt, size="1024x1024", n=1, response_format="b64_json", extra_body={"seed": 42}, ) ``` Base Ideogram 4 presets are `V4_DEFAULT_20`, `V4_QUALITY_48`, and `V4_TURBO_12`. The fal variants automatically select `V4_FAST_20` and `V4_INSTANT_8`, respectively. A preset controls both `num_inference_steps` and guidance, so do not set those fields directly. You can also produce captions with a separate SRT LLM using [prompt enhancement](/docs/sglang-diffusion/models_with_pe). Configure an Ideogram-specific JSON caption template and schema; the generic prose template is not appropriate for these checkpoints. This does not reproduce fal's checkpoint-specific prompt expander or establish equivalent image quality. ## 5. Run in ComfyUI # JoyAI-Echo Source: https://docs.sglang.io/cookbook/diffusion/JoyEcho/JoyEcho Run JoyAI-Echo multi-shot audio–video generation with SGLang Diffusion. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. Prepare the [pinned Echo 1.0 checkpoint](#3-1-prepare-the-echo-1-0-checkpoint) and set `JOY_ECHO_MODEL_PATH` before starting the server. ## 2. Model capabilities [JoyAI-Echo](https://huggingface.co/jdopensource/JoyAI-Echo) is an 8-step long-form audio-video model built on LTX-2. Its paired memory bank carries decoded visual context and audio latents across prompt changes, making it strongest for multi-shot, minute-scale sequences that need continuity in both picture and soundtrack. Choose JoyEcho over a standard LTX pipeline when shots must share audiovisual memory. Its distilled 832×480 path prioritizes long-form continuity and throughput rather than the higher-resolution two-stage quality modes offered by LTX-2.3. SGLang materializes the Echo 1.0 monolithic release through the built-in [JoyAI-Echo overlay](https://huggingface.co/Niehen6174/JoyAI-Echo-overlay). Prepare the pinned checkpoint below before running the examples. | Aspect | Standard LTX-2.3 | JoyEcho | | - | - | - | | Pipeline | `LTX2Pipeline` / `LTX2TwoStageHQPipeline` | `JoyEchoPipeline` (default for this model) | | Denoising | Multi-step flow matching + CFG | LTX-2 DMD distilled path (8 steps, `guidance_scale=1.0`) | | Multi-shot | Not supported | Paired audio–video memory bank across shots | | Sequence parallelism | LTX-2 SP (video/audio sharded) | Ulysses SP (`ulysses_degree=2`): single-shot and multi-shot + memory bank | | Post-processing | Optional two-stage HQ upscaling | Per-shot mp4 output | Review the model license on the [JoyAI-Echo Hugging Face page](https://huggingface.co/jdopensource/JoyAI-Echo) before production or commercial use. SGLang support does not grant additional model usage rights. ## 3. Model Deployment ### 3.1 Prepare the Echo 1.0 checkpoint The native overlay requires `JoyAI-Echo-release.safetensors`. The upstream repository's Echo 1.5 revision does not contain that file. Download the [Echo 1.0 revision](https://huggingface.co/jdopensource/JoyAI-Echo/tree/4187f9a53c6eff3a76c51e79bd27f70d10f7591b) into the Hugging Face cache: ```bash theme={null} JOY_ECHO_MODEL_PATH=$(python - <<'PY' from huggingface_hub import snapshot_download print(snapshot_download( repo_id="jdopensource/JoyAI-Echo", revision="4187f9a53c6eff3a76c51e79bd27f70d10f7591b", allow_patterns=["JoyAI-Echo-release.safetensors", "*.json", "*.md", "LICENSE"], )) PY ) ``` Use the returned cache path as `--model-path` and keep `--model-id jdopensource/JoyAI-Echo` when using local weights. The model ID lets BCG select JoyEcho's support policy. Initial startup also downloads and materializes the overlay and its text encoder dependencies. ### 3.2 Serve the model JoyEcho uses the default `JoyEchoPipeline` registered for `jdopensource/JoyAI-Echo`. A single high-VRAM GPU (for example H100 or H200) is enough for the common 832x480 / 121-frame / 8-step setting. ```bash theme={null} sglang serve \ --model-path "$JOY_ECHO_MODEL_PATH" \ --model-id jdopensource/JoyAI-Echo ``` Optional environment variable for long runs: ```bash theme={null} export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True ``` For multi-GPU serving, tensor parallelism (TP) and **Ulysses sequence parallelism (SP)** are supported. JoyEcho SP uses an **asymmetric layout**: video target latents are time-sharded across ranks, while audio (including memory tokens) is **replicated** on every rank so cross-attention stays temporally aligned. Multi-shot runs with `enable_memory_bank=true` are supported on SP. ```bash theme={null} sglang serve \ --model-path "$JOY_ECHO_MODEL_PATH" \ --model-id jdopensource/JoyAI-Echo \ --num-gpus 2 \ --ulysses-degree 2 ``` JoyEcho SP currently targets **Ulysses-only** parallelism (`ulysses_degree=2`, `ring_degree=1`). Ring SP is not validated for this pipeline. For `sglang generate`, add `--num-gpus 2 --ulysses-degree 2` to the commands in section 4. ## 4. Model Invocation ### 4.1 Default sampling | Setting | Default | | - | - | | Resolution | 832x480 | | Frames | 121 | | FPS | 25 | | Steps | 8 | | Guidance scale | 1.0 | | Seed | 12345 | ### 4.2 Single-shot text-to-video ```bash theme={null} sglang generate \ --model-path "$JOY_ECHO_MODEL_PATH" \ --model-id jdopensource/JoyAI-Echo \ --prompt "A curious raccoon walks through a sunlit forest path" \ --height 480 --width 832 --num-frames 121 --fps 25 \ --num-inference-steps 8 --seed 42 \ --save-output ``` Disable the memory bank for standalone clips with a config file: ```bash theme={null} cat > /tmp/joy_echo_single.json <<'EOF' { "prompt": "A curious raccoon walks through a sunlit forest path", "enable_memory_bank": false, "seed": 42, "height": 480, "width": 832, "num_frames": 121, "fps": 25, "num_inference_steps": 8 } EOF sglang generate --config /tmp/joy_echo_single.json \ --model-path "$JOY_ECHO_MODEL_PATH" --model-id jdopensource/JoyAI-Echo \ --save-output ``` ### 4.3 Multi-shot generation JoyEcho does **not** generate all shots in one forward pass. Each shot is one generation request. Continuity is carried by an in-process **memory bank** on the pipeline instance. Typical workflow: 1. **Shot 0** — memory bank is empty; the model generates a standalone A/V clip. 2. **After decode** — decoded video frames and packed audio latents are committed to the memory bank (up to 7 slots by default). 3. **Shot 1+** — prior-shot frames are re-encoded and prepended as a memory prefix before denoising. 4. **Per-shot seeding** — official semantics use `prompt_seed = base_seed + shot_index`. Pass multiple prompts as a list in a config file: ```bash theme={null} cat > /tmp/joy_echo_4shot.json <<'EOF' { "prompt": [ "Shot 0: A raccoon wakes up in a cozy attic.", "Shot 1: The raccoon climbs down and opens the back door.", "Shot 2: It walks through a rainy alley under neon signs.", "Shot 3: The raccoon finds a warm bakery window and stops." ], "enable_memory_bank": true, "reset_memory_bank": true, "seed": 42, "height": 480, "width": 832, "num_frames": 121, "fps": 25, "num_inference_steps": 8 } EOF sglang generate --config /tmp/joy_echo_4shot.json \ --model-path "$JOY_ECHO_MODEL_PATH" --model-id jdopensource/JoyAI-Echo \ --save-output ``` You can also pass prompts from a text file (one prompt per line) with `--prompt-path`: ```bash theme={null} sglang generate \ --model-path "$JOY_ECHO_MODEL_PATH" \ --model-id jdopensource/JoyAI-Echo \ --prompt-path /tmp/joy_echo_shots.txt \ --seed 42 \ --height 480 --width 832 --num-frames 121 --fps 25 \ --num-inference-steps 8 \ --save-output ``` ### 4.4 Memory bank controls | Parameter | Default | Meaning | | - | - | - | | `enable_memory_bank` | `true` | Read/write the paired A/V memory bank between shots. | | `reset_memory_bank` | `true` | Clear the bank and shot counter at the start of a new session (`request_id` change or first shot). | Set `enable_memory_bank=false` when you want independent shots without cross-shot continuity. ### 4.5 Measured two-H200 single-shot configuration For short independent clips, keep the text and audio/video components on GPU with `--component-residency=all=resident`. Full-stage profiles showed that this removes repeated host-to-device weight copies between component uses. The following configuration was measured on two H200s with Ulysses degree 2, TP1, PyTorch 2.11.0+cu130, 640x384, 33 frames, 8 steps and seed 42. It disables compilation and the memory bank: ```bash theme={null} cat > joy_echo_h200.json <<'EOF' {"enable_memory_bank": false} EOF CUDA_VISIBLE_DEVICES=0,1 sglang generate \ --model-path "$JOY_ECHO_MODEL_PATH" --model-id jdopensource/JoyAI-Echo \ --config joy_echo_h200.json --prompt "A curious raccoon" \ --width 640 --height 384 --num-frames 33 --num-inference-steps 8 --seed 42 \ --num-gpus 2 --ulysses-degree 2 \ --performance-mode manual --enable-torch-compile=false --quality exact \ --component-residency=all=resident --warmup-mode request \ --save-output --perf-dump-path joy_echo_h200.json.perf ``` For BCG, add `--enable-breakable-cuda-graph --warmup-resolutions 640x384 --warmup-num-frames 33`. Check for successful `[Diffusion BCG] captured` logs and absence of request signature misses. Keep the same model ID, resolution, frame count and quality as warmup. Two fresh-process saved requests per configuration, after request warmup: | Lossless mode | Auto residency E2E | All resident E2E | Reduction | Peak reserved per rank, auto → resident | | - | -: | -: | -: | -: | | Eager | 2.563 s | 2.366 s | 7.65% | 46.43 → 69.32 GiB | | BCG | 1.229 s | 1.058 s | 13.88% | 48.77–48.88 → 69.32 GiB | Loading, warmup and profiling are excluded from these timings. The paired eager profiles remove 71 pinned host-to-device copies (13.07 GB, 278.86 ms on the profiled rank), with the same 77,045 kernel launches. All eight lossless outputs have pixel-identical video frames; audio differences are comparable to baseline repeat variability. These results cover this compact single-shot workload, rather than the default 121-frame or multi-shot memory-bank workload. Use `quality=lossless` for this recipe. High-mode output did not pass the separate quality comparison, and high + BCG is rejected by the runtime. If a larger request exceeds available memory, return to auto residency or keep only selected components resident. ## 5. Practical Tips * Use `--num-inference-steps 8` and `--guidance-scale 1.0` to match the official JoyEcho DMD distilled path. * Multi-shot prompts can be passed as a `prompt` list, via `prompt_path`, or as sequential API calls on the same server instance. * The memory bank caps at **7 slots**; from shot 8 onward the oldest slots roll off. * For **2-GPU latency**, try **Ulysses SP** (`--num-gpus 2 --ulysses-degree 2`) on both single-shot and multi-shot runs. Use **TP** when you need a different sharding strategy or more than two GPUs. * Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for long multi-shot SP sessions. * JoyEcho outputs per-shot mp4 files with synchronized audio. There is no built-in two-stage HQ upscaling path like LTX-2.3 HQ. ## 6. Run in ComfyUI # Kandinsky6 Source: https://docs.sglang.io/cookbook/diffusion/Kandinsky/Kandinsky6 Deploy Kandinsky 6.0 for joint text- and image-conditioned video and audio generation. ## 1. Quick start Use Linux with NVIDIA CUDA. This model is not yet in a released package; use a checkout containing this implementation, not an unmodified release image. Your Hugging Face account must have access to the checkpoint; authenticate with `hf auth login` or supply `HF_TOKEN` in the server environment. Install from the repo root: ```bash theme={null} uv pip install -e "python[diffusion]" --prerelease=allow ``` Start with Pro-distill and resident weights on B200, GB300, or RTX PRO 6000 (96 GB). The picker separates verified lightweight HTTP requests from full-size offline generation and unverified custom topologies. See the [installation guide](/docs/sglang-diffusion/installation) for platform setup. The request returns a video job ID, not the video bytes. Poll `GET /v1/videos/{id}` until `status` is `completed`, then download `GET /v1/videos/{id}/content` for the MP4 containing both video and audio. Treat `failed` as an error; delete the job with `DELETE /v1/videos/{id}` when finished. See [Download a completed video](#5-download-a-completed-video) for a complete polling example. ## 2. Model capabilities Kandinsky 6.0 generates video and audio jointly from a text prompt, optionally conditioned on a reference image. One DiT predicts both modalities at each denoising step. A Reason1 encoder derived from Qwen2.5-VL supplies token features, and CLIP supplies pooled text features. Both request modes use the same server; image conditioning does not require switching checkpoints. Choose Pro-distill for the 10-step path, or Pro for 50-step classifier-free guidance. The recipes cover 768 x 512, 121-frame clips at 24 fps. They use native SGLang components, without calling a third-party model pipeline. Validation compares against the initial SGLang implementation; it does not establish parity with the unpublished Kandinsky6 Diffusers implementation. | Checkpoint | Scheduler | Default steps | Default guidance | | - | - | -: | -: | | `kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers` | PiFlow | 10 | 1.0 | | `kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers` | Flow-match Euler | 50 | 5.0 | Pro-distill only accepts guidance 1.0 and does not encode a negative prompt. The scheduler is loaded from the checkpoint. For a local directory with an unrelated name, specify `--model-id Kandinsky-6.0-Pro-distill-5s-Diffusers` to select its sampling defaults, or explicitly pass 10 steps and guidance 1.0. ## 3. Offline generation The default request produces a muxed video and audio file: ```bash theme={null} sglang generate \ --model-path kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers \ --performance-mode speed --attention-backend fa --warmup-mode off \ --prompt "A woman chops vegetables in a sunlit kitchen, with soft jazz playing." \ --seed 42 --save-output ``` Add `--image-path /path/to/reference.png` for image conditioning. On RTX PRO 6000, use `--attention-backend torch_sdpa` in the CLI or `attention_backend="torch_sdpa"` in Python. For two B200 GPUs, add `--num-gpus 2 --tp-size 1 --ulysses-degree 2 --ring-degree 1`. To use Pro, change the model path; its defaults select 50 steps and guidance 5.0. ### Python API ```python theme={null} from sglang.multimodal_gen import DiffGenerator generator = DiffGenerator.from_pretrained( model_path="kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers", num_gpus=1, performance_mode="speed", attention_backend="fa", warmup_mode="off", ) try: result = generator.generate(sampling_params_kwargs={ "prompt": "A woman chops vegetables in a sunlit kitchen, with soft jazz playing.", "seed": 42, "save_output": True, "output_path": "kandinsky6_samples", "return_file_paths_only": False, "return_frames": True, }) print(result.output_file_path) finally: generator.shutdown() ``` `result.frames` contains RGB frames and `result.audio` contains the decoded waveform. Add `image_path` to the sampling parameters for a reference image. Omit the two return-data flags when only the output file is needed. ## 4. Parallelism and memory * **Ulysses**: full-size Pro-distill generation on two B200 or four GB300 GPUs matches single-GPU output on the same hardware, including every decoded RGB frame and the audio waveform. It shards the video stream; audio and text streams remain replicated. Video K/V is gathered for audio-to-video attention. * **CFG parallel**: Pro can distribute its conditional and unconditional branches with `--num-gpus 2 --ulysses-degree 1 --enable-cfg-parallel`. This is a different topology from Ulysses. It does not accelerate Pro-distill, which has no guidance branch. * **TP and Ring**: implemented, but full-model outputs differ from the single-GPU baseline. Do not treat them as verified bitwise-equivalent modes. Ring2, Ring4, and TP2 x Ring2 complete full-size Pro-distill generation on GB300 and repeat exactly within each topology. Ring2 serving also covers image conditioning, Cache-DiT, and dynamic/merged LoRA restoration. Use `--num-gpus 2 --ulysses-degree 1 --ring-degree 2` to select Ring2; validate both video and audio quality before deployment. * **TP x SP**: four-GB300 TP2 x Ulysses2 full generation is tested. It reduces weight memory but was slower than Ulysses4 and produced different video and audio. Prefer Ulysses when memory permits; validate quality before deploying TP. * **FSDP + Ulysses**: add `--use-fsdp-inference true` to shard DiT weights without sharding linear reductions. Four-GB300 Ulysses4 generation matches resident Ulysses4 in every decoded RGB frame and the audio waveform. This is a lower-memory alternative with additional weight-gather latency, not a faster default. Request-level Cache-DiT switching is also tested with FSDP. * **Encoder parallelism**: `--encoder-parallel fold` shards Reason1 and CLIP across the DiT replica, including a Ulysses-only replica. Four-GPU generation is tested, but folding changes reductions and the generated video and audio. Keep `auto` for the baseline recipes and validate quality before forcing folding. See [Encoder parallelism](/docs/sglang-diffusion/encoder_parallel). * **Offload**: keep components resident when memory allows. Use `--cpu-offload-components transformer text_encoder vae` for whole-component offload, or `--layerwise-offload-components all` for layerwise offload. Layerwise offload includes both DiT text towers and the audio VAE's encoder, decoder, and vocoder blocks. Use component-specific residency settings to keep selected components resident. Both require host RAM and add transfers; lower device memory does not imply lower total memory or lower latency. The picker's **Server** tab exposes attention, weight placement and offload components. With DiT layerwise offload, you can also tune prefetch depth and resident layers using `--dit-offload-prefetch-size` and `--dit-layerwise-resident-layers`. Increasing either uses more VRAM; these non-default settings are unverified tuning choices, not faster recommendations. Component offload moves whole components; layerwise offload streams supported blocks. They are alternatives to FSDP in this picker, not additive presets. ### Deployment limits Use monolithic serving. Encoder/denoiser/decoder disaggregation is not supported. Breakable CUDA graphs are also not enabled for Kandinsky6: the server disables `--enable-breakable-cuda-graph`. Its variable-length text path needs additional handling to preserve the unpadded computation; same-shape graph experiments do not establish support for arbitrary requests. Keep eager execution for the recipes on this page. ### Choosing a topology Pro-distill, batch 1, 768 x 512, 121 frames, 10 steps, BF16/FA, no offload or compilation, on GB300 with 277.5 GiB per GPU: | Topology | Hot E2E (s) | Output-rank peak reserved (GiB) | Matches single-GPU RGB/audio | | - | -: | -: | - | | Single GPU | 33.95 | 80.36 | Reference | | Ulysses4 | 11.33 | 80.91 | Exact | | Ulysses4 + FSDP | 12.13 | 42.54 | Exact | | TP2 x Ulysses2 | 13.99 | 50.97 | No | | TP4 | 17.72 | 35.91 | No | Times are medians of three hot requests after one initial request, with startup warmup disabled and default conditioning caching enabled. They include MP4 saving but exclude model loading. Memory is PyTorch runtime peak reserved on the output rank, not loading peak or the maximum across all GPUs. These results do not establish output quality for TP; TP also changes the text encoder's numerical path. For this workload, FSDP reduces measured memory by 47% versus resident Ulysses4 at a 7% latency cost. Add it when weight memory is the constraint; keep ordinary Ulysses when latency is the priority. ### RTX PRO 6000 and SageAttention3 Pro-distill also runs with resident weights on one 96 GB RTX PRO 6000 Blackwell Server Edition. Use `--attention-backend torch_sdpa` on this SM120 device: the current platform resolves `fa` to SDPA, so requesting `fa` does not establish a FlashAttention benchmark. With [SageAttention3](https://github.com/thu-ml/SageAttention/tree/d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5/sageattention3_blackwell) installed, `--component-attention-backends transformer=sage_attn_3` selects it only for the DiT, keeping the encoders on SDPA. The tested upstream revision requires SM120/121 and rejects B200 SM100. With PyTorch 2.14, its build required `NVCC_APPEND_FLAGS=-std=c++20`; no kernel source or hardware guard was modified. Same full-size workload and timing method as above, on one RTX PRO 6000 with driver 595.84, PyTorch 2.14.1+cu130, Transformers 5.17.0, and Diffusers 0.37.0: | DiT backend | Hot E2E (s) | Runtime peak reserved (GiB) | | - | -: | -: | | SDPA | 133.91 | 80.34 | | SageAttention3 | 105.33 | 82.05 | Sage3 reduced latency by 21.3% but changed the outputs substantially: all-frame PSNR 11.97 dB, mean SSIM 0.375, and raw-audio waveform SNR -2.21 dB versus SDPA. These are paired differences, not perceptual quality scores. Each backend was repeatable across four requests, but Sage3 is **not lossless** and is not the default recipe. Validate both video and audio before choosing it. ### VAE decode The video VAE defaults to tiled decoding, including multi-GPU execution. Its decode group is separate from the DiT's TP and CFG collectives. Explicit `--vae-config.parallel-decode-mode spatial_shard` uses whole-clip decoding with frame-causal attention computed without a quadratic whole-video mask. Whole-clip decoding still needs substantial activation memory. A 768 x 512, 121-frame run on two B200 GPUs completed with the transformer, both text encoders, and audio VAE offloaded, but reached 170.4 GiB output-rank peak reserved. This is not a lower-memory replacement for tiled decode. Tiled and whole-clip decoding use different normalization and attention contexts and are not interchangeable numerical baselines; keep tiled decoding for the validated default output. The audio VAE contains both the mel decoder and vocoder. Current checkpoints include a complete vocoder copy inside `audio_vae`, so this pipeline does not also load the redundant standalone `vocoder` component. CUDA matrix RoPE uses a fused kernel that preserves the eager FP32 rounding boundaries. CPU and autograd paths retain the eager implementation. No sampling parameter or sparse-attention approximation is introduced. The checkpoint's 128-wide Q/K normalization uses native PyTorch FP32 fusion during CUDA inference without changing the stored parameter dtype or offload policy. Other head widths retain the original normalization path. ### Cache-DiT Cache-DiT is opt-in per request through `enable_cache_dit` and `cache_dit_params` in the Python sampling parameters or the JSON `/v1/videos` body. Set `enable_cache_dit: false` to restore uncached execution without restarting. See [Cache-DiT configuration](/docs/sglang-diffusion/cache_dit) for the available knobs. The **Request** tab offers an opt-in Cache-DiT switch for both text and image conditioning. It uses the runtime cache defaults, not a quality-calibrated preset. A short request may finish before cache warmup ends and gain no speedup. The adapter caches both evolving video and audio streams together. Its dynamic decision uses the video residual, not an independent audio-quality metric. Skipping blocks is approximate and can change both outputs substantially; validate video and audio on your own prompts before enabling it. It is not part of the lossless defaults, and should not be combined with breakable CUDA graphs. ### LoRA adapters Use adapters trained for the same Kandinsky6 checkpoint architecture. The native loader accepts PEFT A/B weights with Diffusers or SGLang DiT layer names, including the `transformer.` prefix. Both video and audio branches belong to `transformer`; an adapter can change both outputs. Add `--lora-path /path/to/adapter.safetensors` to load an adapter at startup. To switch an existing server to dynamic LoRA without restarting: ```bash theme={null} curl --fail-with-body http://localhost:30000/v1/set_lora \ -H "Content-Type: application/json" \ -d '{ "lora_nickname": "kandinsky6", "lora_path": "/path/to/adapter.safetensors", "target": "transformer", "strength": 1.0, "merge_mode": "dynamic" }' ``` The adapter path must be readable by the server. `dynamic` applies the low-rank update during inference; `merge` updates the base weights before inference. These modes may differ numerically. To disable either mode and restore the base model: ```bash theme={null} curl --fail-with-body http://localhost:30000/v1/unmerge_lora_weights \ -H "Content-Type: application/json" -d '{"target": "transformer"}' ``` Adapter selection is server-wide, not isolated per request. Coordinate changes with other clients. Merging offloaded weights retains CPU snapshots of the modified layers for exact restoration; budget for this host memory. Validation uses synthetic adapters to check loading, strength changes, repeatability, and exact restoration of video and audio; it does not establish the quality of a community adapter. See the [model inventory](/docs/sglang-diffusion/compatibility_matrix) and the [super-resolution recipe](/cookbook/diffusion/Kandinsky/Kandinsky6-SR) for the separate video SR pipeline. ## 5. Download a completed video Use the `id` returned by the picker's Request command. The same endpoints serve joint generation and SR. Set the base URL to match your server port: ```python Download theme={null} import json import shutil import time import urllib.request base_url = "http://localhost:30000" video_id = "REPLACE_WITH_RETURNED_ID" deadline = time.monotonic() + 1800 while time.monotonic() < deadline: with urllib.request.urlopen(f"{base_url}/v1/videos/{video_id}", timeout=30) as response: job = json.load(response) if job["status"] == "failed": raise RuntimeError(job) if job["status"] == "completed": with urllib.request.urlopen( f"{base_url}/v1/videos/{video_id}/content", timeout=120 ) as response, open("output.mp4", "wb") as output: shutil.copyfileobj(response, output) break time.sleep(2) else: raise TimeoutError("Video generation did not finish within 30 minutes") ``` After downloading, remove the server-side job when no longer needed: ```bash Cleanup theme={null} curl --fail-with-body -X DELETE http://localhost:30000/v1/videos/REPLACE_WITH_RETURNED_ID ``` ## 6. ComfyUI There is no Kandinsky6 integrated DiT executor in the [ComfyUI plugin](/docs/sglang-diffusion/comfyui). The generic server video node has not been validated for this model's joint audio/video output. Use the HTTP or offline recipes above until that integration is verified. # Kandinsky6 SR Source: https://docs.sglang.io/cookbook/diffusion/Kandinsky/Kandinsky6-SR Deploy Kandinsky 6 video super-resolution with x2, x4 or x2.25 upscaling, source audio and explicit decode speed, memory and numerical tradeoffs. ## 1. Quick start Use Linux with NVIDIA CUDA and a checkout containing this implementation; it is not yet in a released package or unmodified release image. Authenticate an account with checkpoint access using `hf auth login`, or supply `HF_TOKEN` to the server. From the repo root: ```bash Install theme={null} uv pip install -e "python[diffusion]" --prerelease=allow ``` The picker uses the distilled two-step checkpoint. Four-GB300 whole-tile decode is the output-preserving choice measured on multi-tile clips; one GPU suffices for a functional check when it has enough memory. The picker also includes explicitly **extrapolated** Hopper and RTX 5090 starting points, not measured speed or memory guarantees. HTTP verification is scoped separately from offline measurements. See the [installation guide](/docs/sglang-diffusion/installation) for platform setup. The request uploads a local video and returns a job `id`. Poll `GET /v1/videos/{id}`, then download `GET /v1/videos/{id}/content` when completed. Use the shared [polling and download example](/cookbook/diffusion/Kandinsky/Kandinsky6#5-download-a-completed-video). For JSON requests, `video_path` refers to a file on the **server**, not the client. ## 2. Model capabilities Kandinsky 6 VSR upscales a source video without a prompt. It encodes the source with a causal KVAE, denoises overlapping tiles and blends their decoded outputs. Unlike the [generation model](/cookbook/diffusion/Kandinsky/Kandinsky6), it retains the source soundtrack rather than generating audio. Audio is converted to mono 44.1 kHz and trimmed to the processed video duration. Use `kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers` for x2, x4 or x2.25 upscaling of up to 121 frames. Its checkpoint-selected PiFlow scheduler uses two DiT calls per tile; changing `num_inference_steps` does not increase quality or the number of evaluations. The scheduler and transformer must come from the same Diffusers checkpoint; sampler settings are read from `scheduler/`. The native path uses dense attention, not the checkpoint's requested NABLA sparse attention, so this is not a claim of numerical parity with the sparse reference. The tested non-distilled `Kandinsky-6.0-VSR-5s-Diffusers` revision `d6ddfabc62b3920b6da6487e4f444d37c591d5f4` declares a flow-Euler scheduler but contains a 640-wide output head instead of the required 64. It is not supported as-is. Do not change the scheduler to bypass this error; use the distilled model. ## 3. Offline usage No prompt is required. A single-GPU example: ```bash Generate theme={null} sglang generate \ --model-path kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers \ --video-path /path/to/input.mp4 \ --sr-resolution-scale 2 \ --num-inference-steps 2 \ --seed 42 --save-output --output-path ./sr_outputs ``` ```python Generate theme={null} from sglang.multimodal_gen import DiffGenerator generator = DiffGenerator.from_pretrained( model_path="kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers", num_gpus=1, ) try: result = generator.generate(sampling_params_kwargs={ "video_path": "/path/to/input.mp4", "sr_resolution_scale": 2, "num_inference_steps": 2, "seed": 42, "output_path": "sr_outputs", "save_output": True, }) print(result.output_file_path) finally: generator.shutdown() ``` ### Request parameters * `sr_resolution_scale`: `2`, `4` or `2.25` (default). The x2.25 path first enlarges pixels x1.125, then processes x2 tiles. * `sr_tiles_batch_size`: default `1`. More tiles per DiT call need more activation memory. Each chunk is seeded by its first tile index, so changing this value changes the noise and output. It is **not a lossless batching optimization**. * `sr_tile_min_overlap`: default `0.2`. Changing overlap changes tile geometry and blending, not just performance. * `sr_target_resolution`: optional `hd`, `fullhd`, `2k` or `WxH`; only downscales the generated result. `sr_target_resize_mode` is `fit` (default) or `exact`. * `seed`: default `42`. Compare identical seeds, tile batching, source frames and attention policy when evaluating a decode optimization. The input must fit at least one tile. A tested landscape source is 384 x 256. The default filename is `_srx_.mp4`; override it with `--output-file-name`. ## 4. Decode speed, memory and numerics These are **VAE decode choices**, independent of DiT TP/Ulysses/Ring:
DecodeHot E2EPeak reservedOutput versus serial
Serial45.26 s47.55 GiBReference
Whole-tile parallel21.73 s47.55 GiBRGB and audio exact in this test
Spatial shard18.01 s24.37 GiBRGB differs; audio exact
Setup: four GB300 GPUs (277.5 GiB each, NV18 links), host RAM 955.7 GiB with an 860 GiB cgroup limit; BF16/FA, Ulysses4, resident weights, no compile or startup warmup. PyTorch 2.14.1+cu130, Transformers 5.17.0, Diffusers 0.37.0. Distilled checkpoint revision `3e3e10bcd23659eeeb23cac740258ee4fc14aa7d`; seed 42, tile batch 1, two steps per tile. A generated kitchen clip, 768 x 512 x 121 frames at 24 fps, was upscaled x2 to 1536 x 1024. Times are medians of three hot requests after one initial request, including MP4 saving and excluding model loading. Memory is PyTorch request peak reserved on the output rank, not loading, NVML or the maximum across ranks. Host memory is not claimed unchanged. **Whole-tile parallel** assigns complete existing tiles to ranks and preserves their temporal caches and blend order. It matched serial RGB and raw audio exactly on the tested workloads **within the same topology**. Select `--vae-config.use-parallel-tiling true`. A single tile falls back to serial. Communication can outweigh savings for small clips or slow interconnects. **Spatial shard is not bitwise lossless.** It splits each tile's height, using halo exchanges and the original temporal cache. Changed convolution shapes change rounding. In the experiment above, all-frame RGB PSNR was **57.51 dB**, minimum SSIM over the first/middle/last frames **0.99921**, mean absolute difference **0.1132/255**, and maximum difference **4/255**. Audio was exact. These measurements are not a guarantee for other videos or perceptual-quality certification. Select it only after validating the output for your workload: ```bash Spatial decode flags theme={null} --vae-config.use-parallel-decode true --vae-config.parallel-decode-mode spatial_shard ``` Both modes are opt-in runtime features. The picker selects whole-tile parallel for its multi-tile recipe; it does not change runtime defaults. If both flags are enabled, spatial sharding takes precedence. One GPU or a tile too short to split retains replicated decoding. Neither mode changes the tile geometry. ### RTX PRO 6000 over PCIe | GPUs / placement | Decode | Hot E2E | Sampled GPU peak | | - | - | - | - | | 1 / resident | Serial | 134.09 s | 48.91 GiB | | 2 / resident, Ulysses2 | Serial | 122.11 s | 48.81 GiB | | 2 / resident, Ulysses2 | Whole-tile | 83.04 s | 48.81 GiB | | 2 / resident, Ulysses2 | Spatial | 73.62 s | 37.45 GiB | | 1 / all-layerwise offload | Serial | 133.60 s | 37.43 GiB | Setup: two RTX PRO 6000 Blackwell Server Edition GPUs (96GB nominal), PCIe/SYS across CPU sockets, **no NVLink**; 499.2 GiB host RAM, no cgroup cap. Driver 595.84, the same PyTorch/Transformers/Diffusers versions and checkpoint as above, but **SDPA**, not FA. A separate generated 768 x 512 x 121-frame clip at 24 fps was upscaled x2 with seed 42, tile batch 1 and two steps per tile. Each row uses one initial request and three hot requests, no compile/startup warmup. E2E includes MP4 saving and raw-frame return, excludes model loading. GPU memory is the maximum **per-GPU NVML sample across ranks**, sampled every 0.5 seconds during requests, not the sum or PyTorch reserved memory; brief peaks may be missed. These are within-machine comparisons, not GB300-versus-RTX speed ratios. Ulysses2 alone reduced latency by only 8.9%. Whole-tile decode reduced it by 38.1% versus one GPU, with **exact RGB/audio** in this test. Spatial reached 45.1% lower latency, but RGB differed: PSNR **57.54 dB**, minimum sampled SSIM **0.99919**, maximum difference **4/255**; audio remained exact. All five configurations reproduced their own outputs across repeated requests. Layerwise offload reduced memory with essentially unchanged latency here; this does not make offload free on other workloads or slower host links. A separate **memory-budget probe**, still on RTX PRO 6000, completed the full clip twice with all-layerwise offload and a **30 GiB PyTorch allocator cap**. Sampled whole-GPU usage peaked at **31.03 GiB**, and RGB/audio matched the resident baseline exactly. The cap excludes CUDA allocations outside PyTorch; it is a test setting, not a deployment recommendation. This supports trying the 32GB recipe, but is **not a 5090 speed, memory-fit or quality measurement**. The complete HTTP regression also passed on one-GPU resident serial, Ulysses2 resident whole-tile and one-GPU layerwise serial with the allocator cap: six requests each covering JSON, uploads, x2/x4/x2.25, tile batch 2, audio/silent inputs and repeated-request equality. Arbitrary videos still need their own validation. ### Small workloads do not necessarily scale On GB300, a 384 x 256 x 33-frame source upscaled x2 (one tile, serial decode, otherwise the same BF16/FA/seed/step policy) took **1.77 s on one GPU, 1.72 s on Ulysses2 and 1.74 s on Ulysses4**. These are three-hot-request medians including saving. The roughly 2% difference is not evidence of a useful speedup; prefer one GPU for this workload. More tiles, not just more GPUs, create work that whole-tile decoding can distribute. ## 5. Memory and parallelism ### Hardware and GPU counts | Hardware | Starting point | Evidence boundary | | - | - | - | | GB300 | Resident, FA; Ulysses4 + whole-tile for multi-tile clips | Measured offline; smaller 1/2/4-GPU workloads also tested | | RTX PRO 6000 96GB | Resident, SDPA; compare 1 GPU with Ulysses2 | PCIe measurements, not a substitute for 5090 memory validation | | H100 80GB / H200 141GB | Resident, FA; start with 1 GPU, then Ulysses2/4 | Extrapolated from native CUDA support; no Hopper SR measurement | | B200 | Resident, FA; start with 1 GPU, then Ulysses2/4 | Extrapolated SR recipe; generation-model tests do not validate SR | | RTX 5090 32GB | SDPA, layerwise offload, tile batch 1; short 384 x 256 input first | Extrapolated from the RTX memory-budget probe; no 5090 execution claim | Only compare GPU counts on the **same input, scale, tile batching and backend**. For small single-tile clips, one GPU avoids communication overhead. For larger multi-tile clips, try Ulysses2/4 with whole-tile decode; extra GPUs do not divide replicated KVAE/latent-upscaler weights. Spatial decode can reduce activation memory, but changes rounding. Offload needs sufficient host RAM as well as VRAM. Do not extrapolate NVLink scaling to a PCIe system, especially across CPU sockets. Eight-GPU FA configurations are selectable as **Ulysses4 x Ring2**, not Ulysses8: 28 heads are not divisible by 8. This is an unmeasured topology, not a speedup recommendation. The current RTX SDPA path has no Ring support, so the picker rejects that combination. Start with fewer GPUs before trying an eight-GPU run. ### Placement and topology DiT Ulysses, TP, Ring and TP x SP are supported. For example, use `--num-gpus 4 --tp-size 1 --ulysses-degree 4` for Ulysses4, or `--num-gpus 4 --tp-size 2 --ulysses-degree 2` for TP2 x Ulysses2. TP times Ulysses must divide the checkpoint's 28 attention heads. **TP and Ring can change output numerics independently of decode mode**; whole-tile decode being exact does not make a TP/Ring topology change lossless. The decode group includes TP/SP ranks within each DP replica, not different requests. KVAE encoding, latent upscaling and tile scheduling remain replicated. For short two-step clips, extra GPUs can add communication without a speedup. Keep components resident when memory permits. For less GPU weight memory, use `--layerwise-offload-components all`, or select `vae latent_upscaler` instead. Temporal caches and non-block weights remain on GPU. Offload is compatible with both decode modes, but consumes host RAM and adds transfers for tiles and temporal segments. The assembled output also occupies host memory until saved. The **Server** tab also offers whole-component offload with `--cpu-offload-components`, selectable offload components, and DiT prefetch and resident-layer counts when streaming the DiT. Increasing those counts consumes additional VRAM. Partial offload recipes, non-default layer counts and FSDP (`--use-fsdp-inference true`) are **unverified options**, not measured improvements. FSDP requires multiple GPUs; SDPA cannot be combined with Ring. The default backend is FA on datacenter Blackwell/Hopper and SDPA on RTX. Hardware starting points do not automatically adapt to every uploaded video's memory demand. Request scale and tile batching do not require restarting the server. ## 6. Limits * **Input:** at most 121 frames. Sources near 24 fps retain their frames; faster videos are subsampled, slower videos retain their native rate. This can change the source timing; SR is not a bit-preserving video/audio transcode. * **Attention:** this native path is dense. Do not compare it with a NABLA sparse reference and attribute every difference to parallel decoding. The latent upscaler uses SDPA motion attention when `natten` is unavailable. * **Deployment:** monolithic only. Disaggregated roles, `torch.compile`, LoRA, direct latent inputs and KVAE-bridge chaining are not supported. * **Loading:** component config and weight mismatches fail explicitly. For a renamed local checkpoint, pass `--model-id Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers` to retain SR routing. See the [compatibility matrix](/docs/sglang-diffusion/compatibility_matrix) for other model families. The opt-in server regression is `python/sglang/multimodal_gen/test/server/test_server_kandinsky6_sr.py`; it covers uploads, JSON, all three scales, source audio and repeated requests, not official pipeline parity. # Krea-2 Source: https://docs.sglang.io/cookbook/diffusion/Krea/Krea-2 Deploy Krea-2 with SGLang - fast, high-quality text-to-image generation. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [Krea-2](https://huggingface.co/krea/Krea-2-Turbo) is Krea's photorealistic text-to-image family, built as a single-stream MMDiT with a Qwen3-VL text encoder and Qwen-Image VAE. Both public variants use the same native SGLang pipeline and differ mainly in their sampling target. Choose Turbo for interactive generation: it is distilled to 8 steps with `guidance_scale=1.0`. Choose Raw when maximum fidelity matters more than latency: it uses roughly 52 steps with classifier-free guidance. Neither checkpoint is an image-editing model; use the Qwen-Image-Edit or FLUX.2 path when an input image must be preserved or transformed. | Variant | Model ID | Sampling profile | | - | - | - | | Turbo | `krea/Krea-2-Turbo` | 8 steps, no CFG; fastest path | | Raw | `krea/Krea-2-Raw` | About 52 steps with CFG; higher-fidelity path | ## 3. Model Deployment This section covers deploying Krea-2-Turbo for fast, high-quality image generation. ### 3.1 Basic Configuration Krea-2-Turbo generates high-quality images in only 8 inference steps. Launch the server with: ```bash Command theme={null} sglang serve \ --model-path krea/Krea-2-Turbo \ --num-gpus 1 \ --port 30000 ``` The step count and guidance scale are **request-time** settings (see [API Usage](#4-api-usage)); Krea-2-Turbo defaults to 8 steps with `guidance_scale = 1.0`. ### 3.2 Configuration Tips See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. * `--num-gpus`: Number of GPUs to use. * Multi-GPU (tensor and/or sequence parallelism): see [Section 3.3](#3-3-multi-gpu-tensor-and-sequence-parallelism). ### 3.3 Multi-GPU: tensor and sequence parallelism Krea-2 supports two multi-GPU axes that can be combined; `--num-gpus` must equal `tp_size × ulysses_degree`. * **Tensor parallelism (`--tp-size N`)** shards the DiT weights across GPUs, lowering per-GPU VRAM. Krea-2's attention heads (48 query / 12 KV) and text heads (20) are divisible by a tp size of 1, 2, or 4. * **Sequence parallelism / Ulysses (`--ulysses-degree N`)** shards the image-token sequence across GPUs while keeping the text prefix replicated. It does **not** shard weights (per-GPU VRAM is unchanged), but its output is **bitwise-identical** to single-GPU. It currently requires a single prompt per request (ragged/padded multi-prompt batches under SP are not supported — use `--tp-size` for those). ```bash Command theme={null} # Tensor parallel (2 GPUs) — lowest per-GPU VRAM (DiT weights sharded) sglang serve --model-path krea/Krea-2-Turbo --num-gpus 2 --tp-size 2 --port 30000 # Sequence parallel / Ulysses (2 GPUs) — output bitwise-identical to single-GPU sglang serve --model-path krea/Krea-2-Turbo --num-gpus 2 --ulysses-degree 2 --port 30000 # Hybrid TP × SP (4 GPUs) — composes both axes sglang serve --model-path krea/Krea-2-Turbo --num-gpus 4 --tp-size 2 --ulysses-degree 2 --port 30000 ``` Measured on 2× H200 (Krea-2-Turbo, 8 steps, 1024×1024): `--tp-size 2` and `--ulysses-degree 2` each give \~1.7× denoise speedup over single-GPU; the hybrid TP=2 × SP=2 reaches \~2.8× on 4 GPUs. **Choosing:** on memory-constrained GPUs prefer `--tp-size` (it shards the \~24 GB DiT, e.g. \~38 GB → \~27 GB per GPU on 2 GPUs); on large-VRAM GPUs sequence parallelism is marginally faster and numerically exact, and the two compose for the highest throughput. ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api). ### 4.1 Generate an Image Generate an image with the OpenAI-compatible images API: ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1") response = client.images.generate( model="krea/Krea-2-Turbo", prompt="a red fox sitting in fresh snow, golden hour, photorealistic", n=1, response_format="b64_json", ) # Save the generated image image_bytes = base64.b64decode(response.data[0].b64_json) with open("output.png", "wb") as f: f.write(image_bytes) ``` You can also generate a single image from the command line: ```bash Command theme={null} sglang generate --model-path krea/Krea-2-Turbo \ --prompt "a red fox sitting in fresh snow, golden hour, photorealistic" \ --num-inference-steps 8 --height 1024 --width 1024 --save-output ``` ### 4.2 Advanced Usage #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to speed up inference with minimal quality loss. Enable it by setting `SGLANG_CACHE_DIT_ENABLED=true`. For more details, see the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit). Cache-DiT works for **both** Krea-2 variants with no extra configuration: SGLang tracks each request's classifier-free-guidance mode, so Krea-2-Turbo (no CFG, `guidance_scale = 1.0`) and Krea-2-Raw (CFG, `guidance_scale ≈ 4.5`) both cache correctly and automatically. **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve \ --model-path krea/Krea-2-Turbo \ --num-gpus 1 \ --port 30000 ``` Measured per-image denoise speedup with the default cache settings (NVIDIA H200, 1024x1024, seed 0): | Variant | Inference steps | Denoise (no cache → cache) | Speedup | | :- | :- | :- | :- | | Krea-2-Turbo (no CFG) | 8 | 1.27s → 0.92s | \~1.4x | | Krea-2-Raw (CFG 4.5) | 50 | 18.0s → 6.3s | \~2.9x | Caching has the most headroom on Raw's longer schedule; the 8-step distilled Turbo has only a few cacheable steps after warmup. **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion (best suited to the longer Raw schedule; not recommended for the 8-step Turbo):
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example (Krea-2-Raw, default cache settings shown explicitly): ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=1 \ SGLANG_CACHE_DIT_BN=0 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.24 \ SGLANG_CACHE_DIT_MC=3 \ sglang serve --model-path krea/Krea-2-Raw ``` #### 4.2.2 Memory and Component Residency Krea-2's DiT is \~24 GB in bf16 (the bulk of the model). On memory-constrained GPUs you can keep less of it resident: * `--component-residency dit=layerwise-offload`: stream the DiT's transformer blocks layer-by-layer with async host-to-device prefetch overlap, so only a small working set stays on the GPU. This is the primary way to fit Krea-2 on a single consumer / 32 GB-class card, at a modest latency cost. Tune the memory/latency trade-off with `--dit-offload-prefetch-size` (`0.0` prefetches one layer for the lowest memory; larger values prefetch more layers -- faster but more memory). * `--component-residency dit=component-offload`: keep the complete DiT on CPU between denoising uses. This and layerwise offload are distinct modes; do not combine them for the same component. * `--component-residency text_encoder=component-offload`: offload the Qwen3-VL text encoder while it is idle during denoising. * `--component-residency vae=component-offload`: offload the VAE between uses. * `--pin-cpu-memory`: pin host memory for offload. Add only as a temporary workaround if you hit `CUDA error: invalid argument`. The legacy `--dit-layerwise-offload`, `--dit-cpu-offload`, `--text-encoder-cpu-offload`, and `--vae-cpu-offload` forms remain accepted. If both legacy DiT offload flags are enabled, layerwise offload is the effective DiT mode. On large-VRAM GPUs (e.g. H200), keep everything resident (offloads off) for the fastest latency. ## 5. Benchmark Test Environment: * Hardware: NVIDIA H200 GPU (1x) * Model: krea/Krea-2-Turbo (8 inference steps) * sglang diffusion version: 0.5.13 **Server Command** (used for both benchmarks below): ```shell Command theme={null} sglang serve --model-path krea/Krea-2-Turbo --port 30000 ``` ### 5.1 Generate an image **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --model krea/Krea-2-Turbo --dataset vbench --task text-to-image \ --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: krea/Krea-2-Turbo Dataset: vbench -------------------------------------------------- Benchmark duration (s): 1.56 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.64 Latency Mean (s): 1.5600 Latency Median (s): 1.5600 Latency P99 (s): 1.5600 -------------------------------------------------- Peak Memory Max (MB): 37466.00 Peak Memory Mean (MB): 37466.00 Peak Memory Median (MB): 37466.00 ============================================================ ``` ### 5.2 Generate images with high concurrency **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --model krea/Krea-2-Turbo --dataset vbench --task text-to-image \ --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: krea/Krea-2-Turbo Dataset: vbench -------------------------------------------------- Benchmark duration (s): 31.47 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 -------------------------------------------------- Request throughput (req/s): 0.64 Latency Mean (s): 16.5000 Latency Median (s): 16.5200 Latency P99 (s): 31.1300 -------------------------------------------------- Peak Memory Max (MB): 37468.00 Peak Memory Mean (MB): 37466.40 Peak Memory Median (MB): 37466.00 ============================================================ ``` ## 6. Run in ComfyUI # LTX2 & LTX2.3 Source: https://docs.sglang.io/cookbook/diffusion/LTX/LTX2 & LTX2.3 Run LTX-2 and LTX-2.3 video generation pipelines with SGLang Diffusion. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [LTX-2](https://huggingface.co/Lightricks/LTX-2) and [LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3) are Lightricks video models for text-to-video and image-conditioned generation. Their defining deployment choice is the pipeline: one-stage minimizes latency, standard two-stage refines quality, and LTX-2.3 adds a 1920×1088 HQ path. Choose LTX when one model family must span fast generation, higher-quality refinement, and LoRA adaptation. Two-stage quality costs substantially more memory and startup work because both DiTs and the spatial upsampler participate; use the one-stage path when latency or capacity matters more than final detail. Use `Lightricks/LTX-2` or `Lightricks/LTX-2.3` as `--model-path`. SGLang loads the two-stage upsampler and distilled components from the selected snapshot. **License notice:** LTX-2 and LTX-2.3 are released under the LTX-2 Community License Agreement, not Apache 2.0. The license includes commercial-use restrictions for some entities. Review the [official Lightricks license](https://huggingface.co/Lightricks/LTX-2.3/blob/main/LICENSE) before production or commercial use; SGLang support does not grant additional model usage rights. ## 3. Model Deployment This section provides deployment configurations optimized for different LTX pipelines and hardware targets. ### 3.1 Basic Configuration The LTX series supports one-stage and two-stage pipelines. LTX-2.3 also supports the HQ two-stage pipeline. The recommended launch configuration depends on whether the target GPU can keep both two-stage DiTs resident. Use the [command builder](#1-quick-start) above. ### 3.2 Configuration Tips Choose the pipeline class based on the quality and latency target: | Use case | Pipeline class | Notes | | - | - | - | | One-stage generation | `LTX2Pipeline` | Fastest LTX native path. Supports T2V and TI2V. | | Two-stage generation | `LTX2TwoStagePipeline` | Uses a base stage and a refinement stage. Supported by LTX-2 and LTX-2.3. | | Two-stage High Quality (HQ) generation | `LTX2TwoStageHQPipeline` | LTX-2.3 HQ path; defaults to 1920x1088 unless you override `--width` and `--height`. | Feature compatibility: | Pipeline class | T2V | TI2V (`--image-path`) | LoRA (`--lora-path`) | Notes | | - | - | - | - | - | | `LTX2Pipeline` | Yes | Yes | Yes | One-stage path. Cannot be combined with HQ because HQ is a separate two-stage pipeline class. | | `LTX2TwoStagePipeline` | Yes | Yes | Yes | Standard two-stage path for LTX-2 and LTX-2.3. | | `LTX2TwoStageHQPipeline` | Yes | Yes | Yes | High Quality two-stage path for LTX-2.3. Use this instead of `LTX2Pipeline`; it is not a one-stage mode flag. | For two-stage pipelines, `--ltx2-two-stage-device-mode` controls transformer residency: | Mode | When to use it | | - | - | | `resident` | Best latency on high-VRAM GPUs because both DiTs can stay resident. | | `original` | Closest to the original two-stage switching semantics. | Other deployment flags: * `--lora-path`: Preload a community LoRA adapter. * `--lora-weight-name`: Select the exact safetensors file when the LoRA repository contains multiple weight files. For native LTX-2.3 two-stage serving without a user LoRA, `resident` is the fastest high-VRAM path. LTX-2 still applies the distilled LoRA during the stage switch, so `--ltx2-two-stage-device-mode` is mainly an LTX-2.3 optimization. When you pass `--lora-path`, SGLang still applies the user LoRA during the two-stage switch, so use `resident` on H200-class GPUs for enough VRAM, but do not expect the same premerged-stage2 benefit as the no-user-LoRA path. ### 3.3 Fast multi-GPU presets For latency-oriented LTX serving, prefer CFG parallel over sequence parallelism. CFG parallel splits guidance branches across GPUs, while SP/Ulysses is mainly a memory/long-sequence tool for LTX. | Target | Recommended server flags | Notes | | - | - | - | | LTX-2.3, 1 high-VRAM GPU | `--ltx2-two-stage-device-mode resident` | Fastest two-stage setup when both DiTs fit. | | LTX-2.3, 1 standard GPU | `--ltx2-two-stage-device-mode original` | Lower VRAM than `resident`; use this when H100-class memory is tight. | | LTX-2, 2 GPUs | `--num-gpus 2 --enable-cfg-parallel` | Fastest verified 2-GPU setup; keep `--dit-layerwise-offload` disabled unless memory is tight. | | LTX-2.3, 2 GPUs | `--num-gpus 2 --enable-cfg-parallel --ltx2-two-stage-device-mode resident` | Fastest common 2-GPU setup. | | LTX-2.3, 4 GPUs | `--num-gpus 4 --tp-size 2 --enable-cfg-parallel --ltx2-two-stage-device-mode resident` | Fastest common 4-GPU layout: TP2 inside each CFG branch. | | Official comparison | `--ltx2-two-stage-device-mode original` | Use this only when matching the original LTX-2.3 stage-switch semantics matters. | Use `--enable-cfg-parallel` for degree-2 CFG parallel. Use `--cfg-parallel-size` only when you explicitly need a different CFG branch count. If `resident` exceeds available VRAM, keep the same parallelism preset and switch only the device mode to `original`. On high-VRAM GPUs, add `--text-encoder-cpu-offload false` if text encoding latency matters and you have enough memory. #### 3.3.1 Two GPUs ```bash theme={null} sglang serve \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --num-gpus 2 \ --enable-cfg-parallel \ --ltx2-two-stage-device-mode resident ``` #### 3.3.2 Four GPUs ```bash theme={null} sglang serve \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --num-gpus 4 \ --tp-size 2 \ --enable-cfg-parallel \ --ltx2-two-stage-device-mode resident ``` ## 4. Model Invocation ### 4.1 Basic Usage The examples below spell out the current SGLang sampling defaults for reproducibility: | Model path | Default output | Default frames | Default steps | | - | - | - | - | | `Lightricks/LTX-2` | 768x512 | 121 | 40 | | `Lightricks/LTX-2.3` | 768x512 | 121 | 30 | | `Lightricks/LTX-2.3` with `LTX2TwoStageHQPipeline` | 1920x1088 | 121 | 15 | #### 4.1.1 LTX-2 one-stage text-to-video ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2 \ --pipeline-class-name LTX2Pipeline \ --prompt "A quiet coastal town at sunrise, fishing boats moving slowly through golden mist, cinematic camera movement" \ --save-output ``` #### 4.1.2 LTX-2.3 one-stage text-to-video ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2Pipeline \ --prompt "A quiet coastal town at sunrise, fishing boats moving slowly through golden mist, cinematic camera movement" \ --save-output ``` #### 4.1.3 LTX-2 two-stage text-to-video ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2 \ --pipeline-class-name LTX2TwoStagePipeline \ --prompt "A handheld shot follows a red tram crossing a rainy city square at night, reflections on the pavement, cinematic lighting" \ --save-output ``` #### 4.1.4 LTX-2.3 two-stage text-to-video ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --prompt "A handheld shot follows a red tram crossing a rainy city square at night, reflections on the pavement, cinematic lighting" \ --save-output ``` #### 4.1.5 LTX-2.3 HQ text-to-video ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStageHQPipeline \ --prompt "A wide cinematic shot of alpine clouds rolling over a mountain ridge, soft morning light, slow aerial camera movement" \ --save-output ``` #### 4.1.6 Image-to-video with one reference image Pass one image to `--image-path` for image-conditioned generation: ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --image-path ./inputs/start.png \ --prompt "The camera slowly pushes forward as the subject turns toward warm window light, subtle natural motion, cinematic" \ --save-output ``` #### 4.1.7 First-to-last-frame transition with two reference images Pass two images to `--image-path` for transition-style TI2V. The first image is used as the starting condition and the second image is used as the ending condition. ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --image-path ./inputs/start.png ./inputs/end.png \ --prompt "A smooth cinematic transition from the first scene into the final scene, dynamic camera motion, motion blur, zhuanchang" \ --save-output ``` ### 4.2 Advanced Usage #### 4.2.1 Use community LoRAs Use `--lora-path` to load a LoRA adapter. If the Hugging Face repo contains multiple safetensors files, use `--lora-weight-name` to select the exact file. `--lora-scale` maps to the standard LoRA merge scale and defaults to `1.0`. The following example uses [`valiantcat/LTX-2.3-Transition-LORA`](https://huggingface.co/valiantcat/LTX-2.3-Transition-LORA): ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --lora-path valiantcat/LTX-2.3-Transition-LORA \ --lora-weight-name ltx2.3-transition.safetensors \ --prompt "A low-angle tracking shot moves through a foggy forest road. The camera rises above the treetops and transitions into a clear view of a snowy mountain peak under bright sunlight, zhuanchang" \ --save-output ``` You can combine the Transition LoRA with two reference images: ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --image-path ./inputs/start.png ./inputs/end.png \ --lora-path valiantcat/LTX-2.3-Transition-LORA \ --lora-weight-name ltx2.3-transition.safetensors \ --prompt "A fast cinematic transition from the first image to the second image, whip-pan motion, atmospheric lighting, zhuanchang" \ --save-output ``` Some community LoRAs only include weights for transformer blocks. In that case, SGLang logs a concise coverage summary and leaves unmatched LoRA-capable layers on the base model weights. This is expected when the adapter format intentionally omits those layers. ## 5. Practical Tips * Use `--pipeline-class-name LTX2TwoStagePipeline` as the default LTX two-stage quality path. * Use `--pipeline-class-name LTX2TwoStageHQPipeline` when you want the HQ path and have enough VRAM for larger outputs. * Use `--ltx2-two-stage-device-mode resident` on high-VRAM GPUs if latency matters more than memory usage. * Use `--ltx2-two-stage-device-mode original` when comparing against official two-stage behavior. * Keep `--width` and `--height` aligned with the target model resolution; for LTX models, these are output video dimensions. ## 6. Run in ComfyUI # LTX2.5 Source: https://docs.sglang.io/cookbook/diffusion/LTX/LTX2.5 Run LTX-2.5 video + audio generation with SGLang Diffusion. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). Docker is recommended for Linux GPU deployments; run the commands below inside the container or your installed Python environment. NATTEN is an optional extra, worth installing only if you plan to use the [diffusion decoder](#4-6-diffusion-decoder) — see that section for why. To use diffusion decoding, load **Diffusion decoder weights** in Server, then select **Diffusion** in Request. Duration and decoder selection are request-time choices; loading the optional decoder requires a server restart. ## 2. Model capabilities [LTX-2.5](https://huggingface.co/Lightricks/LTX-2.5) is an open world model from Lightricks, built for local execution and fine-tuning. Its established use is generating synchronized, high-fidelity video and audio from text, image and video inputs. It is a 22B DiT paired with a Gemma-4-12B text encoder, separate video and audio VAEs, and a vocoder that outputs 48 kHz stereo. Video and audio are denoised jointly in one pass rather than dubbed afterwards, so they stay in sync. Use **`Lightricks/LTX-2.5-Diffusers`** as `--model-path`. **License notice:** LTX-2.5 is released under the LTX-2.x Community License Agreement, not Apache 2.0. The license includes commercial-use restrictions for some entities. Review the [official Lightricks license](https://github.com/Lightricks/LTX-2/blob/main/LICENSE.md) before production or commercial use; SGLang support does not grant additional model usage rights. ### 2.1 New in LTX-2.5 Two capabilities have no equivalent in LTX-2 / LTX-2.3: A duration head predicts how long the shot the caption implies should run, and picks the frame count for you. Pass `--auto-duration` instead of `--num-frames`. A diffusion model replaces the convolutional VAE decoder for the latent-to-pixel step. Enable with `--use-diffusion-decoder`. Both are optional and off by default. ### 2.2 Components | Path | Component | Used by | | - | - | - | | `transformer/` | Distilled DiT (the default) | always | | `transformer_full/` | Full / SFT DiT | `--model-variant dev` | | `vae/` | Convolutional video VAE | encode always; decode by default | | `diffusion_decoder/` | Diffusion video decoder, decoder-only | `--use-diffusion-decoder` | | `latent_upsampler/` | Spatial x2 latent upsampler | `LTX2TwoStagePipeline` | | `duration_head/` | Predicts clip length from the caption | `--auto-duration` | | `audio_vae/`, `vocoder/`, `connectors/`, `text_encoder/`, `tokenizer/`, `scheduler/` | Shared | always | Encoding always uses `vae/`, and both decoders consume the same latents, so the decoder choice does not change anything upstream of it. ## 3. Model Deployment ### 3.1 Basic Configuration ```bash theme={null} sglang serve \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline ``` On a single high-VRAM GPU no extra flags are needed. Use the [command builder](#1-quick-start) above. ### 3.2 Configuration Tips Choose the pipeline class based on the quality and latency target: | Use case | Pipeline class | Notes | | - | - | - | | One-stage generation | `LTX2Pipeline` | Fastest path. Supports T2V and TI2V, auto-duration and the diffusion decoder. | | Two-stage generation | `LTX2TwoStagePipeline` | Half-resolution base stage, x2 latent upsample, then a short refinement. Pass the **final** resolution. | There is no HQ pipeline class for LTX-2.5, and no `--distilled-lora-path` for either weights variant: LTX-2.5 distils the weights themselves rather than merging a LoRA per stage, so `--ltx2-two-stage-device-mode` (which governs that swap) does not apply either. Every feature on this page — text-to-video, image conditioning, auto-duration, the diffusion decoder, and either weights variant — works with both pipeline classes. Selecting weights: * `--model-variant dev` serves the full / SFT DiT from `transformer_full/`; the default is the distilled one. See [section 4.5](#4-5-the-dev-transformer). ### 3.3 Multi-GPU presets | Target | Recommended server flags | Notes | | - | - | - | | 1 high-VRAM GPU | *(no extra flags)* | 960×544 fits comfortably on an H200. | | 1 tight-VRAM GPU | `--quantization fp8` | Halves the DiT and cuts peak memory \~18 GB at unchanged speed. See [section 3.4](#3-4-fp8-quantization). | | 1 very tight GPU | `--dit-layerwise-offload` | Cuts peak memory by roughly 10 GB, at about 4x the wall clock. | | 1 32 GB GPU (e.g. RTX 5090) | `--layerwise-offload-components dit text_encoder` | Streams the DiT and the 22 GB Gemma-4 encoder layer by layer and keeps the rest resident. 960×544 / 121 frames / 8 steps: 15.2 s at 17 GB peak, vs 17.0 s at 31 GB with `--cpu-offload-components text_encoder ...`. Pins \~55 GB of host memory per GPU. | | 2 GPUs, long sequences | `--num-gpus 2 --ulysses-degree 2` | Sequence parallel; the memory/long-sequence tool. | | 2 32 GB GPUs | `--num-gpus 2 --ulysses-degree 2 --layerwise-offload-components dit text_encoder --vae-tiling false --vae-sp false --dit-layerwise-resident-layers 12` | Untiled VAE decode splits the frame height across the ranks (2.2 s to 1.1 s); 12 resident DiT layers cut the per-step host stream. RTX 5090 960×544 / 121 frames: 16.9 s at 26 GB peak per GPU, vs 21.8 s. Pins \~55 GB of host memory per GPU. | | 2 GPUs, large DiT | `--num-gpus 2 --tp-size 2` | Tensor parallel across attention heads. | | 2 GPUs, dev weights | `--num-gpus 2 --enable-cfg-parallel` | Splits the guided and unguided branches across GPUs. Measured 1.77x on denoising (15.1s to 8.5s, 960×544 / 57 frames / 30 steps). | | 2 GPUs, diffusion decoder | `--num-gpus 2 --ulysses-degree 2` | The decoder's tiles are split across the ranks by default. See [section 4.6.1](#4-6-1-memory-and-multi-gpu). | **CFG parallelism does not apply on the default (distilled) path.** That DiT runs unguided, so there is no negative branch to split across GPUs and `--enable-cfg-parallel` buys nothing — the CFG-parallel presets on the LTX-2 / LTX-2.3 page do not carry over. It *is* worth using with `--model-variant dev`, which runs with guidance. ### 3.4 fp8 quantization `--quantization fp8` quantizes the DiT's linear layers as it loads them, so it needs no pre-quantized checkpoint: ```bash theme={null} sglang serve \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --quantization fp8 ``` At 960×544 / 49 frames the transformer loads in 18.11 GB against 35.37 GB for bf16, and the run peaks at 53.5 GB against 71.1 GB. Denoising time is unchanged: the distilled 8-step path at this size is bound by memory traffic rather than matmul throughput, so fp8 buys headroom rather than speed. Expect a different sample for a given seed. Quantization nudges the denoising trajectory and diffusion amplifies that, so the result differs from bf16 without being worse. ## 4. Model Invocation ### 4.1 Text-to-video with audio ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --prompt "A cinematic shot of a red fox walking through a snowy forest at dawn, the camera tracking alongside, snow crunching underfoot." \ --save-output ``` Defaults: 960×544, 121 frames, 24 fps. Video and audio are generated jointly and muxed into one MP4. The default DiT is distilled and runs off a fixed 8-sigma schedule rather than a step count, so `--num-inference-steps` and `--guidance-scale` have no effect here. Use [`--model-variant dev`](#4-5-the-dev-transformer) when you want control over either. ### 4.2 Image-to-video ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --image-path ./inputs/start.png \ --prompt "The camera pushes forward as the subject turns toward the light." \ --save-output ``` The conditioning image is re-compressed to match the compression the model was trained against — CRF 18 for LTX-2.5, where LTX-2 / 2.3 use 33. SGLang picks the right one from the checkpoint, so nothing needs to be passed. ### 4.3 Auto-duration LTX-2.5 ships a duration head — a small module that reads the encoded caption and regresses the natural length of the shot it describes. Use it when the prompt implies a duration ("a quick glance" vs "a slow pan across the valley") and you would rather not guess a frame count: ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --prompt "A red fox walking through a snowy forest at dawn." \ --auto-duration \ --save-output ``` The prediction is clamped to `--auto-duration-min-seconds` / `--auto-duration-max-seconds` (default 1–20 s) and snapped to the VAE's temporal grid, so the result is always a valid frame count. It overrides `--num-frames`. For an online server, pass the same LTX-2.5-only controls through `extra_body`: ```python Python theme={null} from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:30010/v1") video = client.videos.create( model="Lightricks/LTX-2.5-Diffusers", prompt="A red fox walking through a snowy forest at dawn.", extra_body={ "auto_duration": True, "auto_duration_min_seconds": 2.0, "auto_duration_max_seconds": 8.0, }, ) ``` ### 4.4 Two-stage (higher quality) Stage 1 runs at half the requested resolution, the latents are upsampled 2x, and a short sigma tail refines at full resolution. Pass the **final** size: ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2TwoStagePipeline \ --prompt "A cinematic shot of a red fox walking through a snowy forest at dawn." \ --height 1088 --width 1920 \ --save-output ``` Resolution must be divisible by 64. Unlike LTX-2.3, no `--distilled-lora-path` is needed: the LTX-2.5 transformer is already distilled. ### 4.5 The dev transformer LTX-2.5 ships two DiTs. `model_index.json` points at the distilled one; the full / SFT weights live in `transformer_full/` and are deliberately left out of the index. Select them with `--model-variant dev`: ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --model-variant dev \ --prompt "A cinematic shot of a red fox walking through a snowy forest at dawn." \ --num-inference-steps 30 --guidance-scale 3.0 \ --save-output ``` The dev variant is not distilled, so SGLang automatically drops the pinned distilled sigma schedule and re-enables the dynamic shifting that `scheduler/` turns off for the distilled DiT. Unlike the distilled path it *is* driven by a step count and *does* want CFG, so pass `--num-inference-steps` and `--guidance-scale` yourself. Note that `from_pretrained` only fetches what `model_index.json` lists, so a partial snapshot download will not include `transformer_full/` (another 38 GB). ### 4.6 Diffusion decoder LTX-2.5 adds a diffusion-based video decoder as an alternative to the convolutional VAE decoder. Rather than deconvolving the latent it denoises pixels conditioned on a context volume built from it, which recovers detail a convolutional decoder tends to smooth away: ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --prompt "A red fox walking through a snowy forest at dawn." \ --use-diffusion-decoder \ --save-output ``` It is a diffusion model in its own right and decodes more slowly than the VAE decoder, so it is off by default — matching upstream, where `LTX2Pipeline` also decodes with the VAE. The offline `generate` command loads the optional decoder automatically when `--use-diffusion-decoder` is present. For an online server, opt into loading the decoder at startup, then select it per request with `use_diffusion_decoder: true`: ```bash theme={null} sglang serve \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --load-diffusion-decoder ``` ```python Python theme={null} video = client.videos.create( model="Lightricks/LTX-2.5-Diffusers", prompt="A red fox walking through a snowy forest at dawn.", extra_body={"use_diffusion_decoder": True}, ) ``` This keeps the default server footprint unchanged while still allowing VAE and diffusion-decoder requests to share one server. When GPU memory is constrained, `--cpu-offload-components diffusion_decoder` keeps the optional decoder on CPU between uses. **Install NATTEN for this decoder.** Its stages run 3D neighborhood attention, and SGLang uses NATTEN's fused `na3d` kernel for it when the package is present. NATTEN is *not* a dependency of `sglang[diffusion]`: without it the decoder falls back to a compiled FlexAttention block mask. The two agree to bf16 rounding, but the fallback is roughly **5x slower** on the decoder's largest attention grid, and has to build the mask on top of that. NATTEN ships prebuilt wheels pinned to a specific torch and CUDA build, so install the one matching your environment rather than a bare version — check your combination at [natten.org](https://natten.org). For torch 2.11 / CUDA 13.0, for example: ```bash theme={null} uv pip install natten==0.21.6+torch2110cu130 -f https://whl.natten.org/ ``` Nothing else changes if you skip it: the decoder still produces the same video, just slower. #### 4.6.1 Memory and multi-GPU Two flags govern how the decode is executed. Both default to on, so the numbers below are what you already get — they matter when you want to turn one off. `--diffusion-decoder-tiling` runs the decoder's two expensive stages over overlapping tiles instead of the whole volume. It is a **memory** control, not a speed one: it costs wall clock and buys headroom. `--diffusion-decoder-parallel-tiling` splits those tiles across the decode-parallel ranks -- the TP, SP, PP and CFG ranks of one replica, since the decoder is replicated over all of them. Without it every one of those ranks decodes every tile and keeps its own identical copy. It only applies on the tiled path, so it does nothing when tiling is off, and nothing at a single rank. Decoding stage on 2xH200 at 960×544, Ulysses degree 2. "Peak" is the whole process, not the decode alone: | Frames | Tiling | Parallel tiling | Decode | Peak | | - | - | - | - | - | | 121 | off | *(n/a)* | **3.38s** | 103.2 GB | | 121 | on | off | 5.68s | 78.2 GB | | 121 | on | on | 4.08s | 79.3 GB | | 49 | off | *(n/a)* | **1.93s** | 84.0 GB | | 49 | on | off | 2.79s | 78.2 GB | | 49 | on | on | 2.12s | 78.2 GB | Reading that: * **Untiled is the fastest option** whenever it fits. Tiling exists for the \~25 GB it saves at 121 frames, which is the difference between fitting on an 80 GB card and not. * **Parallel tiling recovers most of tiling's cost** — 1.39x at 121 frames, 1.32x at 49 — but does not beat an untiled decode. It also adds about 1 GB for the gather buffers. * The gain grows with tile count, so it is larger at higher resolution: at 1920×1088 / 49 frames the decode goes from 12.32s to 7.86s, **1.57x**. Output is bitwise identical however many ranks the tiles are split over: every rank draws the whole grid's noise in the same order, and only the decode is shared out. Turning *tiling* on or off does change the result slightly near tile borders, so pick one and stay with it if you need reproducible frames: ```bash theme={null} # Fastest, if the untiled decode fits in VRAM sglang serve \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --load-diffusion-decoder \ --diffusion-decoder-tiling false # Memory-bound: keep tiling, and split the tiles over both GPUs sglang serve \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --load-diffusion-decoder \ --num-gpus 2 --ulysses-degree 2 ``` # LingBot Video MoE Source: https://docs.sglang.io/cookbook/diffusion/LingBot-Video/LingBot-Video-MoE Serve the native LingBot Video MoE 30B-A3B text-to-video model with SGLang Diffusion. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [LingBot Video MoE 30B-A3B](https://huggingface.co/robbyant/lingbot-video-moe-30b-a3b) is a text-to-video mixture-of-experts model served through a native SGLang Diffusion pipeline. Its main advantage is the MoE economics: 30B total parameters with 3B active per token, so a plain `sglang serve` invocation covers the released 480×480, 81-frame profile without parallelism flags. Prompting is the distinctive part of this checkpoint: it expects a structured JSON caption rather than an unexpanded natural-language prompt, passed as the request's `prompt` string. A compact 17-frame, 12-step smoke-test profile is available for quick validation next to the released 40-step profile. The public checkpoint served by the pipeline: | Model ID | Task | Default output | | - | - | - | | `robbyant/lingbot-video-moe-30b-a3b` | Text to video | 480x480, 81 frames at 16 FPS | The checkpoint expects a structured JSON caption rather than an unexpanded natural-language prompt. The JSON is passed as the request's `prompt` string; it is not an `extra_params` object. ## 3. Serve LingBot Video MoE Start the server with the Hugging Face model ID: ```bash Command theme={null} sglang serve \ --model-path robbyant/lingbot-video-moe-30b-a3b \ --port 30010 ``` ## 4. Generate a video The following request uses the compact 17-frame, 12-step smoke-test profile. Use the model defaults of 81 frames and 40 steps for the released generation profile. ```python Python theme={null} import json import time from pathlib import Path import requests base_url = "http://127.0.0.1:30010" prompt = json.dumps( { "comprehensive_description": { "scene_content_description": ( "A small silver robot arm on a white table slowly reaches " "toward a red cube. The background is a softly lit laboratory wall." ), "camera_movement_description": ( "The camera is static at eye level in a medium shot." ), }, "camera_info": { "color": "Neutral", "frame_size": "Medium", "shot_type_angle": "Eye level", "lens_size": "Medium", "composition": "Center", "lighting": "Soft light", "lighting_type": "Artificial light", }, "world_knowledge": [], "prominent_elements": [ { "name": "robot arm", "description": "A small silver robot arm with a two-finger gripper.", "actions": [ { "timestamp": "[0.0s - 1.0s]", "action": "reaches toward the red cube", } ], "location": "center of the frame", "relative_size": "dominant", "shape_and_color": "articulated silver metal arm", "texture": "brushed metal", "appearance_details": "two-finger gripper and visible joints", "relationship": "reaching toward the red cube on the table", "orientation": "upright, base on the table", "pose": "reaching", } ], }, separators=(",", ":"), ) response = requests.post( f"{base_url}/v1/videos", json={ "model": "robbyant/lingbot-video-moe-30b-a3b", "prompt": prompt, "size": "640x384", "num_frames": 17, "fps": 16, "num_inference_steps": 12, "guidance_scale": 6.0, "flow_shift": 3.0, "seed": 0, }, timeout=60, ) response.raise_for_status() video_id = response.json()["id"] while True: job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json() if job["status"] == "completed": break if job["status"] == "failed": raise RuntimeError(job.get("error") or "Video generation failed") time.sleep(1) video = requests.get( f"{base_url}/v1/videos/{video_id}/content", timeout=300, ) video.raise_for_status() Path("lingbot_video_moe.mp4").write_bytes(video.content) ``` ## 5. Request constraints * `num_frames` must be `1` or `4n+1`; examples include 17 and 81. * Width and height must both be multiples of 16. * The native defaults are `guidance_scale=6.0`, `flow_shift=3.0`, `num_inference_steps=40`, and `fps=16`. * Keep the prompt as serialized JSON. Raw free text is outside the checkpoint's expected caption format. # LingBot World Source: https://docs.sglang.io/cookbook/diffusion/LingBot-World/LingBot-World Serve LingBot World realtime camera-controlled video world models with SGLang-diffusion. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. Use the [Realtime WebUI](#3-realtime-webui) to start a camera-controlled session. The Request tab checks model metadata; video generation uses WebSocket, not an HTTP POST. ## 2. Model capabilities [LingBot World](https://huggingface.co/robbyant/lingbot-world-fast-diffusers) is a realtime camera-controlled video world model. It keeps a live causal session, applies prompt and camera events between chunks, reuses DiT/VAE state, and streams decoded frames instead of finishing a bounded clip before returning. Choose it for interactive exploration and continuous control, not one-shot cinematic generation. The causal window makes long sessions practical but gives up the global bidirectional context available to offline Wan or LTX pipelines; session state and cache policy are therefore part of the serving contract. ## 3. Deployment ```bash Command theme={null} sglang serve \ --model-path robbyant/lingbot-world-fast-diffusers \ --pipeline-class-name LingBotWorldCausalDMDPipeline \ --num-gpus 4 \ --ulysses-degree 4 \ --dit-cpu-offload false \ --text-encoder-cpu-offload false ``` ### Optional KV-Cache Compression Long-running sessions can enable lossy int4 PRQ compression for completed causal KV-cache chunks by installing `quant-videogen` as described in the quantization guide and adding `--kv-cache-quant int4` to the server command. The current and most recent completed chunks remain in BF16. See [Causal KV-Cache Quantization](/docs/sglang-diffusion/quantization#causal-kv-cache-quantization) for the algorithm, tuning options, measured memory-latency tradeoff, and support limits. ## 4. Realtime WebUI The lightweight local WebUI is useful for validating latency, frame transport, and camera control behavior. ```bash Command theme={null} python -m http.server 18080 -d python/sglang/multimodal_gen/apps/realtime_webui ``` Open `http://127.0.0.1:18080` and use: ```text Example theme={null} ws://127.0.0.1:30000/v1/realtime_video/generate ``` ## 5. HTTP and WebSocket API LingBot World uses the realtime video WebSocket endpoint. The server keeps one live session, generates one chunk at a time, and accepts runtime control events while generation is running. ### Endpoints | API | Method | Purpose | Notes | | - | - | - | - | | `/v1/models` | `GET` | Query the served model id before opening a session. | The WebUI uses this to fill the model field when the server exposes model metadata. | | `/v1/realtime_video/generate` | `WebSocket` | Create one realtime LingBot session and stream generated video chunks. | The first client message must be an `init` message encoded with MessagePack. | ### `init` message Send this MessagePack map immediately after the WebSocket opens. | Parameter | Type | Required | Meaning | | - | - | - | - | | `type` | string | Yes | Must be `"init"`. | | `model` | string | No | Model id. Leave empty to use the served model. | | `prompt` | string | Yes | Text prompt for the initial scene and motion style. | | `first_frame` | bytes or string | Yes | Initial reference image. Send bytes from the WebUI/client, or a server-readable image path/string. | | `size` | string | Yes | Generation size as `WIDTHxHEIGHT`, for example `832x480`. | | `fps` | number | Yes | Target playback FPS for the generated stream. | | `num_frames` | integer | Yes | Frames per generated chunk. LingBot uses chunked causal generation, so this controls per-chunk latency and queue size. | | `seed` | integer | No | Random seed for deterministic sampling. | | `num_inference_steps` | integer | No | Denoising steps per chunk. LingBot defaults to `4` when omitted. | | `guidance_scale` | number | No | Classifier-free guidance scale. Realtime LingBot commonly uses `1`. | | `negative_prompt` | string | No | Negative prompt passed to the diffusion pipeline. | | `quality` | `"exact"`, `"lossless"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `lossless` keeps FP32 decode and enables only any eligible request-gated kernel fusions. `high` includes those fusions and uses the validated BF16 decode path for lower per-chunk latency. | | `max_chunks` | integer | No | Stop after this many chunks. Omit for a continuous session. | | `realtime_causal_sink_size` | integer | No | Number of sink frames/tokens retained in the causal attention window. | | `realtime_causal_kv_cache_num_frames` | integer | No | Number of recent frames retained in the causal KV cache window. | | `realtime_output_format` | `"webp"`, `"jpeg"`, `"raw"` | No | Preview/output transport. `webp` and `jpeg` send encoded preview frames; `raw` sends raw RGB; omit for lossless delta-gzip RGB. | | `output_compression` | integer | No | Preview quality for `webp` or `jpeg`, from `1` to `100`. | | `enable_upscaling` | boolean | No | Enable server-side super resolution after frame decode. | | `upscaling_scale` | integer | No | Super-resolution scale. Current default is `4` when upscaling is enabled. | | `upscaling_model_path` | string | No | Optional Real-ESRGAN model path. | | `enable_frame_interpolation` | boolean | No | Enable frame interpolation. Keep this disabled when measuring true generated FPS. | | `frame_interpolation_exp` | integer | No | Interpolation multiplier exponent. `1` means 2x frames. | | `frame_interpolation_scale` | number | No | RIFE internal scale for interpolation. | | `frame_interpolation_model_path` | string | No | Optional RIFE model path. | | `condition_inputs.camera_actions` | `list[list[string]]` | No | Initial scripted camera actions, one action list per frame. | ### Runtime `event` messages After `init`, send MessagePack event maps to update the live session. | Parameter | Type | Required | Meaning | | - | - | - | - | | `type` | string | Yes | Must be `"event"`. | | `kind` | `"prompt"` or `"camera_actions"` | Yes | Runtime event kind. | | `payload` | string or object/list | Yes | For `prompt`, a non-empty string. For `camera_actions`, either scripted `list[list[string]]` or state-mode payload. | | `event_id` | integer | No | Client sequence id. The server echoes it in chunk/frame metadata after the event is sampled. | `camera_actions` supports two payload modes: | Mode | Payload shape | Meaning | | - | - | - | | Script | `list[list[string]]` | A fixed sequence of per-frame actions consumed by upcoming chunks. | | State | `{ "mode": "state", "transitions": [{"actions": [...], "client_ts_ms": ...}] }` | Live control state transitions from keyboard or UI controls. | Supported LingBot action tokens include `w`, `a`, `s`, `d` for camera movement and `i`, `j`, `k`, `l` for look controls. ### Server messages | Message | Payload | Meaning | | - | - | - | | `frame_batch` | MessagePack map with `payload` bytes | One batch of frames. The map includes `chunk_index`, `num_frames`, `content_type`, `encoding`, `width`, `height`, and frame-batch metadata. | | `chunk_stats` | MessagePack map | Per-chunk timing and transport metrics, including `scheduler_forward_ms`, `raw_payload_build_ms`, `chunk_total_ms`, `num_frames`, and `ws_payload_bytes`. | | `error` | MessagePack map | Server-side validation or generation error. | ### Minimal client sketch ```python Python theme={null} import msgspec.msgpack import websocket ws = websocket.create_connection("ws://127.0.0.1:30000/v1/realtime_video/generate") ws.send_binary(msgspec.msgpack.encode({ "type": "init", "prompt": "A quiet rainy London alley, stable camera motion.", "first_frame": open("reference.jpg", "rb").read(), "size": "832x480", "fps": 25, "num_frames": 9, "num_inference_steps": 4, "guidance_scale": 1, "realtime_output_format": "webp", "output_compression": 95, })) ws.send_binary(msgspec.msgpack.encode({ "type": "event", "kind": "camera_actions", "event_id": 1, "payload": {"mode": "state", "transitions": [{"actions": ["w"], "client_ts_ms": 0}]}, })) ``` ## 6. Consistency LingBot World uses raw-frame websocket GT plus per-chunk latency guards for consistency checks. ## 7. Notes * Use the realtime endpoint for interactive sessions: `/v1/realtime_video/generate`. * Prefer WebP preview transport for interactive testing; use raw-frame transport for consistency checks. * Long-running sessions should be validated with raw-frame consistency before changing causal cache, condition sampling, or VAE decode behavior. ## 8. Run in ComfyUI # LingBot World 2.0 Source: https://docs.sglang.io/cookbook/diffusion/LingBot-World/LingBot-World-2.0 Serve LingBot World 2.0 realtime camera-controlled video world models with SGLang-diffusion. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. Use the [Realtime WebUI](#3-realtime-webui) to start a camera-controlled session. The Request tab checks model metadata; video generation uses WebSocket, not an HTTP POST. ## 2. Model capabilities [LingBot World 2.0](https://huggingface.co/robbyant/lingbot-world-v2-14b-causal-fast-diffusers) is the 14B realtime camera-controlled world model in the LingBot family. It generates continuously from a live causal session, accepts control updates between chunks, and reuses DiT and VAE state for incremental frame delivery. Choose 2.0 when the larger model and interactive KV-window path justify an 8-GPU deployment. Like the earlier LingBot release, it prioritizes responsive control and bounded causal state over the full-clip global context of offline video diffusion. ## 3. Deployment ```bash Command theme={null} export SGLANG_LINGBOT_LAZY_VAE_ENCODE_BLACK_FRAMES=60 export SGLANG_LINGBOT_ENABLE_INTERACTIVE_KV_WINDOW=true sglang serve \ --model-path robbyant/lingbot-world-v2-14b-causal-fast-diffusers \ --pipeline-class-name LingBotWorldCausalDMDPipeline \ --num-gpus 8 \ --ulysses-degree 8 \ --dit-cpu-offload false \ --text-encoder-cpu-offload false \ --vae-config.use-parallel-decode true \ --vae-config.parallel-decode-mode spatial \ --enable-torch-compile false ``` ### Optional KV-Cache Compression Long-running sessions can enable lossy int4 PRQ compression for completed causal KV-cache chunks by installing `quant-videogen` as described in the quantization guide and adding `--kv-cache-quant int4` to the server command. The current and most recent completed chunks remain in BF16. See [Causal KV-Cache Quantization](/docs/sglang-diffusion/quantization#causal-kv-cache-quantization) for the algorithm, tuning options, measured memory-latency tradeoff, and support limits. ## 4. Realtime WebUI The lightweight local WebUI is useful for validating latency, frame transport, and camera control behavior. ```bash Command theme={null} python -m http.server 18080 -d python/sglang/multimodal_gen/apps/realtime_webui ``` Open `http://127.0.0.1:18080` and use: ```text Example theme={null} ws://127.0.0.1:30000/v1/realtime_video/generate ``` ## 5. HTTP and WebSocket API LingBot World 2.0 uses the realtime video WebSocket endpoint. The server keeps one live session, generates one chunk at a time, and accepts runtime control events while generation is running. ### Endpoints | API | Method | Purpose | Notes | | - | - | - | - | | `/v1/models` | `GET` | Query the served model id before opening a session. | The WebUI uses this to fill the model field when the server exposes model metadata. | | `/v1/realtime_video/generate` | `WebSocket` | Create one realtime LingBot session and stream generated video chunks. | The first client message must be an `init` message encoded with MessagePack. | ### `init` message Send this MessagePack map immediately after the WebSocket opens. | Parameter | Type | Required | Meaning | | - | - | - | - | | `type` | string | Yes | Must be `"init"`. | | `model` | string | No | Model id. Leave empty to use the served model. | | `prompt` | string | Yes | Text prompt for the initial scene and motion style. | | `first_frame` | bytes or string | Yes | Initial reference image. Send bytes from the WebUI/client, or a server-readable image path/string. | | `size` | string | Yes | Generation size as `WIDTHxHEIGHT`, for example `832x480`. | | `fps` | number | Yes | Target playback FPS for the generated stream. | | `num_frames` | integer | Yes | Frames per generated chunk. LingBot uses chunked causal generation, so this controls per-chunk latency and queue size. | | `seed` | integer | No | Random seed for deterministic sampling. | | `num_inference_steps` | integer | No | Denoising steps per chunk. LingBot defaults to `4` when omitted. | | `guidance_scale` | number | No | Classifier-free guidance scale. Realtime LingBot commonly uses `1`. | | `negative_prompt` | string | No | Negative prompt passed to the diffusion pipeline. | | `quality` | `"exact"`, `"lossless"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `lossless` keeps FP32 decode and enables only any eligible request-gated kernel fusions. `high` includes those fusions and uses the validated BF16 decode path for lower per-chunk latency. | | `max_chunks` | integer | No | Stop after this many chunks. Omit for a continuous session. | | `realtime_causal_sink_size` | integer | No | Number of sink frames/tokens retained in the causal attention window. | | `realtime_causal_kv_cache_num_frames` | integer | No | Number of recent frames retained in the causal KV cache window. | | `realtime_output_format` | `"webp"`, `"jpeg"`, `"raw"` | No | Preview/output transport. `webp` and `jpeg` send encoded preview frames; `raw` sends raw RGB; omit for lossless delta-gzip RGB. | | `output_compression` | integer | No | Preview quality for `webp` or `jpeg`, from `1` to `100`. | | `enable_upscaling` | boolean | No | Enable server-side super resolution after frame decode. | | `upscaling_scale` | integer | No | Super-resolution scale. Current default is `4` when upscaling is enabled. | | `upscaling_model_path` | string | No | Optional Real-ESRGAN model path. | | `enable_frame_interpolation` | boolean | No | Enable frame interpolation. Keep this disabled when measuring true generated FPS. | | `frame_interpolation_exp` | integer | No | Interpolation multiplier exponent. `1` means 2x frames. | | `frame_interpolation_scale` | number | No | RIFE internal scale for interpolation. | | `frame_interpolation_model_path` | string | No | Optional RIFE model path. | | `condition_inputs.camera_actions` | `list[list[string]]` | No | Initial scripted camera actions, one action list per frame. | ### Runtime `event` messages After `init`, send MessagePack event maps to update the live session. | Parameter | Type | Required | Meaning | | - | - | - | - | | `type` | string | Yes | Must be `"event"`. | | `kind` | `"prompt"`, `"camera_actions"`, or `"composite_input"` | Yes | Runtime event kind. | | `payload` | string, object, or list | Yes | For `prompt`, a non-empty string. For `camera_actions`, either scripted `list[list[string]]` or state-mode payload. For `composite_input`, a map containing `input_types` plus each named input. | | `event_id` | integer | No | Client sequence id. The server echoes it in chunk/frame metadata after the event is sampled. | `camera_actions` supports two payload modes: | Mode | Payload shape | Meaning | | - | - | - | | Script | `list[list[string]]` | A fixed sequence of per-frame actions consumed by upcoming chunks. | | State | `{ "mode": "state", "transitions": [{"actions": [...], "client_ts_ms": ...}] }` | Live control state transitions from keyboard or UI controls. | Supported LingBot action tokens include `w`, `a`, `s`, `d` for camera movement and `i`, `j`, `k`, `l` for look controls. Use `composite_input` when multiple runtime inputs should be sampled together, such as updating the prompt and camera controls in one event. ### Server messages | Message | Payload | Meaning | | - | - | - | | `frame_batch` | MessagePack map with `payload` bytes | One batch of frames. The map includes `chunk_index`, `num_frames`, `content_type`, `encoding`, `width`, `height`, and frame-batch metadata. | | `chunk_stats` | MessagePack map | Per-chunk timing and transport metrics, including `scheduler_forward_ms`, `raw_payload_build_ms`, `chunk_total_ms`, `num_frames`, and `ws_payload_bytes`. | | `error` | MessagePack map | Server-side validation or generation error. | ### Minimal client sketch ```python Python theme={null} import msgspec.msgpack import websocket ws = websocket.create_connection("ws://127.0.0.1:30000/v1/realtime_video/generate") ws.send_binary(msgspec.msgpack.encode({ "type": "init", "prompt": "A quiet rainy London alley, stable camera motion.", "first_frame": open("reference.jpg", "rb").read(), "size": "832x480", "fps": 25, "num_frames": 9, "num_inference_steps": 4, "guidance_scale": 1, "realtime_output_format": "webp", "output_compression": 95, })) ws.send_binary(msgspec.msgpack.encode({ "type": "event", "kind": "camera_actions", "event_id": 1, "payload": {"mode": "state", "transitions": [{"actions": ["w"], "client_ts_ms": 0}]}, })) ws.send_binary(msgspec.msgpack.encode({ "type": "event", "kind": "prompt", "event_id": 2, "payload": "A quiet snowy Tokyo alley, stable camera motion.", })) ws.send_binary(msgspec.msgpack.encode({ "type": "event", "kind": "composite_input", "event_id": 3, "payload": { "input_types": ["prompt", "camera_actions"], "prompt": "A quiet neon Shanghai alley, stable forward camera motion.", "camera_actions": [["w"], ["w"], []], }, })) ``` ## 6. Consistency LingBot World 2.0 uses raw-frame websocket GT plus per-chunk latency guards for consistency checks. ## 7. Notes * Use the realtime endpoint for interactive sessions: `/v1/realtime_video/generate`. * Prefer WebP preview transport for interactive testing; use raw-frame transport for consistency checks. * Long-running sessions should be validated with raw-frame consistency before changing causal cache, condition sampling, or VAE decode behavior. ## 8. Run in ComfyUI # LongCat-Image Source: https://docs.sglang.io/cookbook/diffusion/LongCat/LongCat-Image Deploy LongCat-Image with SGLang Diffusion and its native in-process Qwen2.5-VL prompt rewriter. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [LongCat-Image](https://huggingface.co/meituan-longcat/LongCat-Image) is a text-to-image model from Meituan. SGLang runs its Qwen2.5-VL prompt rewriter in process with the native SGLang runtime before text encoding and denoising. The native pipeline keeps prompt rewriting and diffusion behind one OpenAI-compatible image endpoint. Rewriting is enabled by default for stronger prompt expansion, but each request can disable it when lower latency matters more than the rewritten prompt. ## 3. Serve the model ```bash Command theme={null} sglang serve \ --model-path meituan-longcat/LongCat-Image \ --performance-mode auto \ --port 30010 ``` Prompt rewriting is enabled by default for LongCat-Image. It adds an autoregressive Qwen2.5-VL pass before diffusion; set `enable_prompt_rewrite=false` on a request when lower latency is more important than rewritten prompt quality. ## 4. Generate an image ```python Python theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://127.0.0.1:30010/v1") response = client.images.generate( model="meituan-longcat/LongCat-Image", prompt="A quiet bookshop on a rainy evening, warm light in the windows", n=1, response_format="b64_json", ) image_bytes = base64.b64decode(response.data[0].b64_json) with open("longcat_image.png", "wb") as f: f.write(image_bytes) ``` To skip prompt rewriting with the OpenAI client, pass the model-specific request field through `extra_body`: ```python Python theme={null} response = client.images.generate( model="meituan-longcat/LongCat-Image", prompt="A quiet bookshop on a rainy evening", extra_body={"enable_prompt_rewrite": False}, ) ``` ## 5. Memory placement Use the unified component-residency selector when the complete pipeline does not fit on the accelerator. For example, keep the repeatedly used DiT resident while moving auxiliary components to CPU between stages: ```bash Command theme={null} sglang serve \ --model-path meituan-longcat/LongCat-Image \ --component-residency dit=resident text_encoder=component-offload vae=component-offload \ --pin-cpu-memory \ --port 30010 ``` See [Component Residency](/docs/sglang-diffusion/api/cli#component-residency) for mode semantics and compatibility with the existing CPU-offload flags. # LongLive 2.0 Source: https://docs.sglang.io/cookbook/diffusion/LongLive/LongLive-2.0 Serve LongLive 2.0 distilled text-to-video and image-to-video models with SGLang-diffusion. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [LongLive 2.0](https://nvlabs.github.io/LongLive/LongLive2/) is NVIDIA's 4-step text/image-to-video model distilled from Wan2.2-TI2V-5B. Its main strength is extending few-step causal generation across prompt changes, so a single request can produce multi-shot sequences without paying a full diffusion schedule for every shot. Choose it for low-step long or multi-shot generation rather than maximum one-shot fidelity. The SGLang path uses a Diffusers conversion of the official weights, and scene continuity still depends on prompt-block and sink settings; validate transitions on the target storyboard. The model weights use the NVIDIA Open Model License. See the [paper](https://arxiv.org/abs/2605.18739) and [GitHub repository](https://github.com/NVlabs/LongLive) for training details. ## 3. Deployment ```bash Command theme={null} sglang serve --model-path Rabinovich/LongLive-2.0-5B-Diffusers ``` In `auto` mode, GPUs with at least 60 GiB available keep the DiT, text encoder, and VAE resident. This uses about 44 GiB on H200 for the 832x480 preset and avoids moving encoder and decoder layers from host memory on every request. Smaller GPUs retain the layerwise-offload defaults. If the GPU runs out of memory, move the text encoder, VAE, and DiT to CPU between stages: ```bash Command theme={null} sglang serve \ --model-path Rabinovich/LongLive-2.0-5B-Diffusers \ --dit-cpu-offload \ --text-encoder-cpu-offload \ --vae-cpu-offload ``` `Rabinovich/LongLive-2.0-5B-Diffusers` is the Diffusers-format conversion of the official `Efficient-Large-Model/LongLive-2.0-5B` weights. ## 4. Generation ### 4.1 Single prompt Generate one clip without starting a server: ```bash Command theme={null} sglang generate \ --model-path Rabinovich/LongLive-2.0-5B-Diffusers \ --prompt "A quiet street at dusk" \ --num-frames 61 \ --save-output \ --output-path outputs ``` 61 frames is 16 latent frames, which is two causal blocks of 8. ### 4.2 Multi-shot long video Multi-shot prompts are sampling parameters, so pass them through the Python API: ```python Python theme={null} from sglang import DiffGenerator gen = DiffGenerator.from_pretrained("Rabinovich/LongLive-2.0-5B-Diffusers") result = gen.generate(sampling_params_kwargs={ "shot_prompts": [ "A husky walks down a sunlit hallway.", "The husky turns and looks at the camera.", "Two dogs play together on a carpet.", ], "chunks_per_shot": 4, "num_frames": 381, # 3 shots x 4 chunks x 8 = 96 latent frames -> 381 frames "scene_cut_prefix": "The scene transitions. ", "multi_shot_sink": True, "multi_shot_rope_offset": 8.0, "save_output": True, "output_path": "outputs", }) ``` Each shot runs for `chunks_per_shot` causal blocks before the next prompt is used. The multi-shot defaults mirror the original LongLive prompt-block settings. ### 4.3 Key parameters These are SGLang request parameters. Original LongLive configs use latent-frame `num_output_frames`; SGLang exposes output-video `num_frames`. * `num_frames`: 61 in the examples. This maps to 16 latent frames, while the original release config defaults to 128 latent frames. * `num_inference_steps`: 4, matching original `sampling_steps`. * `guidance_scale`: 1.0, matching the original inference config. * `height` / `width`: 704 / 1280 by default, matching original latent H/W 44 / 80 with 16x spatial compression. * `shot_prompts`, `chunks_per_shot`, `scene_cut_prefix`, `multi_shot_sink`, and `multi_shot_rope_offset`: SGLang request fields for the original prompt-block and multi-shot behavior. ### 4.4 Image-to-video Pass a first frame with `--image-path` to condition the clip on an image: ```bash Command theme={null} sglang generate \ --model-path Rabinovich/LongLive-2.0-5B-Diffusers \ --prompt "A quiet street at dusk" \ --image-path first_frame.png \ --num-frames 61 \ --save-output \ --output-path outputs ``` The image is used as the first-frame condition. ## 5. Notes * `num_frames` must map to a whole number of causal blocks. The latent frame count is `(num_frames - 1) / 4 + 1` and must be divisible by 8. For example, 61, 125, and 189 frames give 16, 32, and 48 latent frames. * SGLang supports T2V sizes 1280x704, 704x1280, 832x480, and 480x832. * I2V request images follow the Wan TI2V preprocessing path in SGLang. This is different from the original LongLive dataset resize path. * For multi-shot runs, set `num_frames` to match `len(shot_prompts) * chunks_per_shot * 8` latent frames, that is `num_frames = (len(shot_prompts) * chunks_per_shot * 8 - 1) * 4 + 1`. ## 6. Run in ComfyUI # MOVA Source: https://docs.sglang.io/cookbook/diffusion/MOVA/MOVA Deploy MOVA with SGLang - simultaneous video and audio generation with asymmetric dual-tower architecture, precise lip-sync, and environment-aware sound effects. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. Set the output directory before starting the server: ```bash theme={null} export SG_OUTPUT_DIR=/root/output_mova mkdir -p "$SG_OUTPUT_DIR" ``` ## 2. Model capabilities [MOVA](https://github.com/OpenMOSS/MOVA) generates video and audio together with an asymmetric dual-tower model connected by bidirectional cross-attention. Its strongest use cases are speaking subjects, visible sound-producing events, and scenes where ambient audio must track the picture rather than be synthesized by a later cascade. The public 360p and 720p checkpoints both generate up to 8 seconds. Choose 360p for the lighter deployment and 720p for output resolution; MOVA is less suitable when the task needs long-form continuity or the richer image/video/audio reference conditioning provided by H3. | Checkpoint | Best fit | Output limit | | - | - | - | | `OpenMOSS-Team/MOVA-360p` | Faster, lower-memory joint audiovisual generation | Up to 8 seconds at 360p | | `OpenMOSS-Team/MOVA-720p` | Higher-resolution lip-sync and environment audio | Up to 8 seconds at 720p | ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration MOVA supports both online serving and CLI generation modes. The recommended launch configurations vary by hardware and resolution. Use the [command builder](#1-quick-start) above. ### 3.2 Configuration Tips See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. * `--num-gpus`: Number of GPUs to use * `--tp`: Tensor parallelism size (should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) * `--ring-degree`: The degree of ring attention-style SP in USP * `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP * `--adjust-frames`: Whether to adjust frames automatically (set to `false` for MOVA) * `--enable-torch-compile`: Enable torch.compile for faster inference ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api). ### 4.1 CLI Generation (sglang generate) ```bash Command theme={null} sglang generate \ --model-path OpenMOSS-Team/MOVA-720p \ --prompt "A man in a blue blazer and glasses speaks in a formal indoor setting, \ framed by wooden furniture and a filled bookshelf. \ Quiet room acoustics underscore his measured tone as he delivers his remarks. \ At one point, he says, \"I would also believe that this advance in AI recently wasn't unexpected.\"" \ --image-path "" \ --adjust-frames false \ --num-gpus 8 \ --ring-degree 2 \ --ulysses-degree 4 \ --num-frames 193 \ --fps 24 \ --seed 67 \ --num-inference-steps 25 \ --enable-torch-compile \ --save-output ``` ### 4.2 Generate a Video ```bash Command theme={null} curl -X POST "http://0.0.0.0:30002/v1/videos" \ -F "prompt=A man in a blue blazer and glasses speaks in a formal indoor setting, framed by wooden furniture and a filled bookshelf. Quiet room acoustics underscore his measured tone as he delivers his remarks. At one point, he says, \"I would also believe that this advance in AI recently wasn't unexpected.\"" \ -F "input_reference=@" \ -F "size=640x352" \ -F "num_frames=193" \ -F "fps=24" \ -F "seed=67" \ -F "guidance_scale=5.0" \ -F "num_inference_steps=25" \ -o create_video.json ``` ### 4.3 Advanced Usage #### 4.3.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit). **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path OpenMOSS-Team/MOVA-720p ``` **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path OpenMOSS-Team/MOVA-720p ``` #### 4.3.2 CPU Offload * `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. * `--vae-cpu-offload`: Use CPU offload for VAE. * `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". ## 5. Benchmark ### 5.1 Speedup Benchmark #### 5.1.1 Generate a video Test Environment: * Hardware: NVIDIA H200 x 8 * git revision: 443b1a8 * Model: OpenMOSS-Team/MOVA-720p **Server Command**: ```bash Command theme={null} sglang serve --model-path OpenMOSS-Team/MOVA-720p --port 30002 \ --adjust-frames false --num-gpus 8 --ring-degree 2 --ulysses-degree 4 \ --tp 1 --enable-torch-compile ``` **Benchmark Command**: ```bash Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --task image-to-video --dataset vbench --num-prompts 1 --max-concurrency 1 \ --port 30002 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: image-to-video Model: OpenMOSS-Team/MOVA-720p Dataset: vbench -------------------------------------------------- Benchmark duration (s): 590.76 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.00 Latency Mean (s): 590.7549 Latency Median (s): 590.7549 Latency P99 (s): 590.7549 -------------------------------------------------- Peak Memory Max (MB): 74996.00 Peak Memory Mean (MB): 74996.00 Peak Memory Median (MB): 74996.00 ============================================================ ``` #### 5.1.2 Generate videos with high concurrency **Server Command**: ```bash Command theme={null} sglang serve --model-path OpenMOSS-Team/MOVA-720p --port 30002 \ --adjust-frames false --num-gpus 8 --ring-degree 2 --ulysses-degree 4 \ --tp 1 --enable-torch-compile ``` **Benchmark Command**: ```bash Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --task image-to-video --dataset vbench --num-prompts 20 --max-concurrency 20 \ --port 30002 ``` ## 6. Run in ComfyUI # MiniMax-H3 Source: https://docs.sglang.io/cookbook/diffusion/MiniMax/MiniMax-H3 Run native MiniMax-H3 video-and-audio generation with SGLang Diffusion. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation), then choose a verified recipe below. Docker is recommended where the picker offers it. Setup changes the deployment; Server and Request expose orthogonal startup and sampling choices. The generated Server command already includes the recommended encoder policy. Change a Server option only for a deliberate trade-off; Request options do not reload the model. The Docker form uses the image's bundled diffusion dependencies without reinstalling them at startup. For conditioned requests, set **Host media directory** under **Variables**; the builder mounts it read-only at `/data/minimax-h3`. AMD and Grace Blackwell currently offer the Python form; the other NVIDIA recipes also offer Docker. To use ModelScope through the same normal `sglang serve` path, prefix the copied command with `SGLANG_USE_MODELSCOPE=true` and replace the model path with `MiniMax/MiniMax-H3`. Keep the selected variant and topology flags unchanged. For platform-specific installation details, see the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). ## 2. Model capabilities [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) is a native joint video-and-audio model for text-to-video-and-audio, first/last-frame control, and multimodal reference conditioning. Its main strength is producing the picture and stereo soundtrack together, so speech, music, ambient sound, and visible events can stay aligned without a separate audio-generation pass. Choose H3 when synchronized audiovisual output or reference-driven generation matters more than a lightweight deployment. The released recipe targets a 768-pixel short edge at 24 fps for 4–15 seconds, and its capabilities are split across two checkpoint partitions; serving every mode therefore requires separate FL2VA and Ref2VA deployments. | Task | `task` value | Conditioning | | - | - | - | | Text to video and audio | `t2va` | Text prompt only | | First/last frame to video and audio | `fl2va` | First frame, last frame, or both | | Reference to video and audio | `ref2va` | Image, video, and audio references, optionally combined with first/last keyframes | Video-to-video (V2V) is a supported `ref2va` use case, not a fourth task value. Run the `Ref2VA` partition and provide a video reference in `conditions`. A hybrid `ref2va` request may also include the same ordered first/last keyframes accepted by `fl2va`, but it must still contain at least one reference condition. Use the selected Hub's root model ID: `MiniMaxAI/MiniMax-H3` on Hugging Face or `MiniMax/MiniMax-H3` on ModelScope. Select the checkpoint variant with `--model-variant`: `fl2va` serves both `t2va` and `fl2va`, while `ref2va` serves reference-conditioned requests. SGLang owns the checkpoint-directory mapping; do not point `--model-path` at a manually downloaded subdirectory. The same FL2VA / Ref2VA split applies in [ComfyUI integrated mode](#comfyui-integrated-mode). Review the license and usage terms in the MiniMax-H3 model card before production or commercial use. SGLang support does not grant additional model usage rights. ## 3. Deployment details The builder accepts legal custom GPU counts and topologies, marking them **Unverified** until the exact recipe has completed end-to-end validation. Static H3 head or partition violations disable Copy before they reach `sglang serve`. ### Grace Blackwell Select **GB300** or **GB200** in the builder. The default is one Linux ARM64 host with four GPUs, Ulysses4, and resident weights. This follows the [four-GPU compute-tray layout](https://docs.nvidia.com/dgx/dgxgb200-user-guide/hardware.html); an NVL72 rack is not a single 72-GPU process host. For two four-GPU hosts, set **Nodes = 2** and **GPUs / node = 4**. The builder emits Ulysses4 inside each host, Ring2 across hosts, and replicated encoders. Set the same reachable **Head address** on both hosts and use node ranks 0 and 1. Run the generated command once per host, with the same SGLang revision and dependencies. Let `sglang serve` resolve and download the selected weights. GB200 is a **derived, unverified recipe**, not a measured GB300-equivalent performance claim. Keep the native precision and platform-default attention for the baseline; the B200/B300 quantization measurements and H200 Cache-DiT quality audit do not establish coverage on Grace Blackwell. GB300 end-to-end coverage currently includes single-host FL2VA text-to-video with audio, native precision, eager execution, and `quality="exact"`. Ref2VA and multi-host commands remain **Unverified**. See the [GB300 measurements](#gb300-single-host) for the tested workload. ### Checkpoint and adapter formats Start with the command emitted by the builder. Every row below is an overlay on the same native SGLang pipeline; component repositories contribute their own config and weights, while weight files retain the base component config. Storage layout and inference behavior are separate contracts: for example, a PEFT file may be either a normal style adapter or a timestep-distilled Turbo adapter. | Scope | Format or variant | Add to the base command | Contract | | - | - | - | - | | Full model | Official mixed BF16/FP32, CFG-distilled | `--model-variant fl2va` or `--model-variant ref2va` | Lossless reference and consistency GT path. CFG distillation removes the negative branch; it is not the few-step timestep distillation used by Turbo releases. | | DiT | Official Diffusers component layout | `--component-paths.transformer MiniMaxAI/MiniMax-H3/transformer` (`fl2va`) or `.../transformer_ref` (`ref2va`) | Loads the official component through the native SGLang graph; no Diffusers runtime fallback. | | DiT | [AdaLN-pruned Diffusers component](https://huggingface.co/multimodalart/MiniMax-H3-Pruned) | `--component-paths.transformer multimodalart/MiniMax-H3-Pruned/transformer` or `.../transformer_ref` | Approximate curve-AdaLN architecture; its config and basis metadata are loaded natively. | | DiT | [Full or AdaLN-pruned](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models), or [LoRA-merged/remixed](https://huggingface.co/FX-FeiHou/MiniMax-H3-Remix) BF16 safetensors | `--component-weights-paths.transformer OWNER/REPO/path/FILE.safetensors` | Weight-only override for a native full/pruned H3 layout. Match the FL2VA/Ref2VA partition; pruned, merged, or dtype-converted exports are approximate, and any author-specific sampler remains a separate requirement. | | DiT | FL/Ref hybrid fine-tunes, including [Singularity v1.3](https://huggingface.co/WarmBloodAban/Minimax-h3_Singularity) (full or pruned INT8) | Replace `--model-variant` with `hybrid`; add `--component-weights-paths.transformer OWNER/REPO/FILE.safetensors` | Explicit merged-weight deployment serving `t2va`, `fl2va`, and `ref2va` on one native pipeline. Quantization and pruning are detected from the file; INT8 requires `comfy-kitchen`. See [Singularity](#singularity-hybrid-weights). | | DiT | Comfy FP8 or self-describing MXFP8 safetensors | `--component-weights-paths.transformer OWNER/REPO/path/FILE.safetensors` | Per-layer metadata selects static/dynamic FP8 or MXFP8 automatically. | | DiT | ConvRot INT8, W4A8, W4A4, or mixed W4A4+INT8 safetensors ([INT8](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models), [W4A8](https://huggingface.co/Winnougan/MiniMax-H3-INT4_Convrot_ComfyUI), [W4A4](https://huggingface.co/Merserk/MiniMax-H3-INT4-ConvRot)) | `--component-weights-paths.transformer OWNER/REPO/path/FILE.safetensors` | Auto-detected. INT8 layers run on SGLang's JIT-compiled ConvRot ops on CC 9.0, 10.0, 12.0 and 12.1 and on `comfy-kitchen` elsewhere; W4A8 and W4A4 layers require `comfy-kitchen`. TP must preserve each file's ConvRot group boundaries. | | DiT | [NVFP4, optionally mixed with INT8 or FP8](https://huggingface.co/Abiray/Minimax-H3-nvfp4-INT4-INT8-Convrot) | `--component-weights-paths.transformer OWNER/REPO/path/FILE.safetensors` | Auto-detected; NVFP4 execution requires NVIDIA compute capability 10.0+. | | DiT | [AutoRound W4A16 component](https://huggingface.co/Ar4ikov/MiniMax-H3-transformer-W4A16-RTN) | `--component-paths.transformer Ar4ikov/MiniMax-H3-transformer-W4A16-RTN` | Self-describing Diffusers component; SGLang reuses the SRT GPTQ/Marlin backend. The linked export is FL2VA. | | DiT | GGUF, full or AdaLN-pruned ([full](https://huggingface.co/leejet/MiniMax-H3-GGUF), [pruned](https://huggingface.co/unsloth/MiniMax-H3-GGUF)) | `--component-weights-paths.transformer OWNER/REPO/FILE.gguf` | CUDA capacity path; aligned TP and layerwise offload are supported, FSDP and LoRA are not. | | Text encoder | Architecture-compatible Qwen3-VL BF16 finetune ([Heretic example](https://huggingface.co/llmfan46/Qwen3-VL-32B-Instruct-ultra-uncensored-heretic)) | `--component-paths.text_encoder OWNER/REPO` | Reuses the native H3 extractor, including its vision tower and layer-50 hidden-state selection. Finetuning changes conditioning, not the sampling schedule. | | Text encoder | [Serialized FP8 component](https://huggingface.co/Qwen/Qwen3-VL-32B-Instruct-FP8) | `--component-paths.text_encoder Qwen/Qwen3-VL-32B-Instruct-FP8` | Only eligible language-model linears use FP8; embeddings, norms, and the vision tower keep their declared precision. | | Text encoder | ConvRot INT8, W4A8, or W4A4 safetensors ([INT8](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/text_encoders), [Heretic INT8](https://huggingface.co/ethanfel/Qwen3-VL-32B-Ultra-Heretic-H3-ComfyUI-INT8-ConvRot), [W4A8](https://huggingface.co/Winnougan/MiniMax-H3-INT4_Convrot_ComfyUI), [W4A4](https://huggingface.co/Merserk/MiniMax-H3-INT4-ConvRot)) | `--component-weights-paths.text_encoder OWNER/REPO/path/FILE.safetensors` | Auto-detected. INT8 layers run on SGLang's JIT-compiled ConvRot ops on CC 9.0, 10.0, 12.0 and 12.1 and on `comfy-kitchen` elsewhere; W4A8 and W4A4 layers require `comfy-kitchen`. INT8/W4A8 files may include a scalar-scale INT8 embedding, which requires `comfy-kitchen`. Unmarked tensors retain their declared precision. | | Text encoder | Comfy NVFP4 with dynamic activation quantization ([Heretic example](https://huggingface.co/sakamakismile/Qwen3-VL-32B-Heretic-MiniMax-H3-NVFP4)) | `--component-weights-paths.text_encoder OWNER/REPO/path/FILE.safetensors` | Requires `comfy-kitchen` and NVIDIA compute capability 10.0+. Per-layer metadata selects native NVFP4 matmul; scalar/row-wise INT8 embeddings remain packed. | | Text encoder | [NVFP4-AWQ](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/text_encoders) or [Quanto qint8](https://huggingface.co/DeepBeepMeep/MiniMax-H3/tree/main/Qwen3-VL-32B-Instruct) safetensors | `--component-weights-paths.text_encoder OWNER/REPO/path/FILE.safetensors` | Memory-oriented formats: compressed storage is restored, then each active matrix uses BF16/FP16 compute. | | Text encoder | [GGUF Qwen3-VL](https://huggingface.co/DeepBeepMeep/MiniMax-H3/tree/main/Qwen3-VL-32B-Instruct) | `--component-weights-paths.text_encoder OWNER/REPO/FILE.gguf` | CUDA capacity path with encoder TP/layerwise support; encoder FSDP is not supported. | | Text encoder | [Compact Qwen3-VL 4B/8B + ClipProj](https://huggingface.co/NicoLab28/ClipProj-MiniMax-H3) | `--component-paths.text_encoder ENCODER_REPO --component-paths.conditioning_projection PROJECTION.safetensors` | Approximate conditioning replacement. A separate weight-only override may quantize the selected small encoder. | | DiT or adapter | Timestep-distilled Turbo, as merged weights or LoRA ([Larry](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora), [LightX2V](https://huggingface.co/lightx2v/Minimax-h3-Turbo), [merged Ref2VA INT8 example](https://huggingface.co/PulpCut/MiniMax-H3-Ref2VA-Turbo-INT8-ConvRot)) | Use `--component-weights-paths.transformer ...` for merged weights, or `--lora-path OWNER/REPO --lora-weight-name FILE --lora-merge-mode auto` for LoRA | Few-step semantic variant. Pin the exact FL2VA/Ref2VA file and its NFE/sigma schedule, scale, and alpha; storage-format detection does not infer sampling behavior. See [LoRA recipes](#5-lora-recipes). | | Adapter | Style, subject, or behavior LoRA ([example](https://huggingface.co/fal/MiniMax-H3-Realism-People-LoRA)) | `--lora-path OWNER/REPO [--lora-weight-name FILE] --lora-merge-mode auto` | Native fused and Diffusers/PEFT layouts are normalized at load time. Keep the base schedule unless the author specifies another one, and preserve any trigger phrase, scale, and alpha metadata. | The rows compose rather than enumerate every cross-product. A Turbo-merged INT8 ConvRot checkpoint, for example, must satisfy both the Turbo sampling contract and the ConvRot storage/backend contract. H3's Qwen3-VL text encoder also handles image understanding; there is no separate `image_encoder` component. Select the conditioning checkpoint, not an optional generation-tail file for prompt rewriting. The `uncensored` or `Heretic` label describes a weight modification, not a separate loader or a guarantee of output quality. For H3, the registered component names are `transformer`, `text_encoder`, `video_vae`, and `audio_vae`. The shorter `--transformer-weights-path` and `--text-encoder-path` aliases remain supported. `conditioning_projection` is an H3 text-encoder sidecar key, not a standalone model component. Plain video/audio VAE safetensors can use `--component-weights-paths.video_vae` or `--component-weights-paths.audio_vae`, but SGLang does not currently advertise a native quantized H3 VAE format. Pre-quantized files are self-describing: do not combine those rows with `--quantization` or `--component-quantizations.*`. Only byte-identical official full weights—whether loaded from the original model, Diffusers component, or a weight-only layout—belong to the consistency GT. Pruned, quantized, compact-encoder, and LoRA routes are outside that baseline and the audited `quality="high"` contract. Packed and per-layer mixed formats reject FSDP unless their row says otherwise; see [Quantization](/docs/sglang-diffusion/quantization) for backend-wide constraints. LoRA tensors alone do not make an execution-coupled release portable. Sparse- attention/SLA adapters and causal-streaming adapters such as RAVEN also require their matching attention or streaming pipeline; they are not standard H3 LoRA overlays in SGLang. Likewise, a remixed checkpoint that prescribes a custom sampler is only covered when that sampler contract can be reproduced—the fact that its safetensors layout loads is not sufficient. For a four-card H200 host, keep the full BF16/FP32 model resident by default. The model fits without FSDP, so this path avoids the per-block parameter all-gathers of the memory-oriented FSDP profile: ```bash 4×H200 resident theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 4 \ --ulysses-degree 4 \ --encoder-parallel auto \ --performance-mode speed \ --port 30010 ``` Pure Ulysses4 is also the faster measured topology on H200, not just a capacity default. The 4×H100 TP2 + Ulysses2 recipe below fits on 141 GB H200 cards, but it replaces the Ulysses all-to-all exchange with two per-block tensor-parallel all-reduces and measured slower end-to-end, at about 30 GB lower peak memory per GPU. See the **H200 topology comparison** in the Benchmarks section for the measured numbers; treat TP2 + Ulysses2 on H200 as a deliberate memory trade, not a latency default. For 4×H100 80 GB, balance the large packed activation with resident weight sharding. TP2 + Ulysses2 was the fastest measured lossless topology while the Qwen encoder still folds across all four GPUs: ```bash 4×H100 fastest theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 4 \ --tp-size 2 \ --ulysses-degree 2 \ --encoder-parallel auto \ --performance-mode speed \ --port 30010 ``` Pure Ulysses4 could not keep the full pipeline resident on 80 GB H100s. Use `--tp-size 4 --ulysses-degree 1` when lower resident memory matters more than the last few percent of latency. FSDP remains a verified capacity option, but its per-block weight all-gathers do not make it the H100 speed default: ```bash 4×H100 FSDP capacity theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 4 \ --ulysses-degree 4 \ --encoder-parallel auto \ --performance-mode speed \ --use-fsdp-inference true \ --port 30010 ``` For a two-card RTX 5090 host, use TP2 and keep 20 DiT blocks resident. Layerwise placement is lossless: it changes parameter placement and transfer scheduling, not the BF16/FP32 denoising or VAE math. This is the fastest measured 32 GB operating point: ```bash 2×RTX 5090 fastest lossless theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 2 \ --tp-size 2 \ --ulysses-degree 1 \ --encoder-parallel auto \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder,vae \ --dit-offload-prefetch-size 1 \ --dit-layerwise-resident-layers 20 \ --enable-torch-compile false \ --port 30010 ``` The DiT residency and prefetch knobs apply only to the repeatedly executed DiT blocks. The text encoder and the video VAE decoder blocks use one-layer prefetch with zero resident layers. The video VAE encoder stays resident because its indexed down blocks cannot host executable layerwise hooks; the roughly 577 MiB audio VAE also stays resident because offloading it only adds transfer overhead. This exact recipe was validated on 2× RTX 5090 (32 GB each) and a 377 GiB host; use a 384 GiB-class machine. The latency and memory comparison is collected in the benchmark section below. For a single 24 GB consumer card (RTX 4090), stream the DiT and text encoder and quantize DiT linear layers online with `convrot_int8`. Keep `vae` out of `--layerwise-offload-components`: putting the VAE decoder in layerwise offload re-streams about 9 GiB on each of 167 decode tiles. Default attention stays `fa` (exact). Approximate backends are opt-in; see [Attention Backends](/docs/sglang-diffusion/attention_backends#sage-then-sol-hybrid). On the RTX 4090 (CC 8.9) `convrot_int8` runs on `comfy-kitchen`, so install it first (`pip install comfy-kitchen`); on CC 9.0, 10.0, 12.0 and 12.1 it uses SGLang's JIT-compiled ConvRot ops instead (built once per machine on first use). ```bash 1×RTX 4090 24GB theme={null} sglang generate \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --quantization convrot_int8 \ --attention-backend fa \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder \ --dit-offload-prefetch-size 1 \ --dit-layerwise-resident-layers 0 \ --enable-torch-compile false \ --prompt "A cat walking on a sunny beach, gentle waves." \ --save-output ``` The same flags work on `sglang serve`. Drop `--quantization` for the BF16 baseline; everything else stays identical. GPU peak stays about 18 GB either way because streaming offload is set by the offload buffers and VAE decode, not the weight dtype. To skip DiT steps on 4090 while keeping `fa`, add Spectrum to the generate command above. Do not combine it with `quality="high"` or Cache-DiT. On this card, 50-step BF16+FA (**1008.6 s**) drops to **285.3 s (3.53×)** with exact attention, up to **219.0 s (4.61×)** with Sage→Sol hybrid + Spectrum. See the 4090 table in Benchmarks. ```bash theme={null} --enable-spectrum \ --spectrum-warmup-steps 11 \ --spectrum-window-size 5 \ --spectrum-flex-window 1.0 ``` The first launch resolves every selected source through the normal Hub path. If a repository requires authentication, export a Hugging Face token in the server environment; no manual pre-download is required. For MiniMax-H3, `--performance-mode speed` deliberately keeps the DiT eager. The current `torch.compile` path changes the model's numerical output, so no recommended lossless preset enables it implicitly. An explicit `--enable-torch-compile true` remains available for controlled experiments, but do not use it to generate consistency ground truth. ### ComfyUI integrated mode The [ComfyUI plugin](/docs/sglang-diffusion/comfyui) can run H3 as a per-step DiT backend. ComfyUI keeps CLIP, VAE, and the sampler loop. SGLang loads only the transformer under `--comfyui-mode`. Copy `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion` into ComfyUI `custom_nodes/`. Load a shipped graph and set `SGLDOptions.model_type` to `minimax_h3`: | Graph | Task | DiT filename | | - | - | - | | `minimax_h3_t2v_sgld.json` | T2V / I2VA / FL2VA | `minimax_h3_fl2va_*.safetensors` | | `minimax_h3_r2v_sgld.json` | Reference-to-video | `minimax_h3_ref2va_*.safetensors` | | `minimax_h3_t2v_sgld_upscaler.json` | Low-res then 3D ×2 | `fl2va` DiT | Pruned BF16 / FP8 files from [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models) are the usual loader inputs. GGUF goes in `SGLDOptions.transformer_weights_path`. On a 24 GB card put the 32B CLIP on CPU and enable DiT layerwise offload; use `kitchen_int8` or GGUF rather than the full unpruned BF16 DiT. ### Singularity hybrid weights [Singularity](https://huggingface.co/WarmBloodAban/Minimax-h3_Singularity) is an FL/Ref fusion fine-tune, not a complete pipeline repository. Keep the official model ID for the encoders and VAEs, and select one exact transformer file: ```bash 2×B300 hybrid checkpoint theme={null} pip install comfy-kitchen WEIGHTS=WarmBloodAban/Minimax-h3_Singularity/Minimax-h3_Singularity_ref2va_Pruned_v1.3_int8.safetensors sglang serve --model-path MiniMaxAI/MiniMax-H3 \ --model-variant hybrid \ --component-weights-paths.transformer "$WEIGHTS" \ --num-gpus 2 --ulysses-degree 2 --performance-mode speed \ --port 30000 ``` For the full INT8 checkpoint, replace the filename with `Minimax-h3_Singularity_ref2va_v1.3_int8.safetensors`. SGLang downloads the selected file automatically. Do not add an online quantization override. `hybrid` requires an explicit weight override; it does not convert the official Ref2VA weights into a multimode model. Request `task` and `conditions` retain their [usual meanings](#4-generate-video-and-audio), including `conditions: []` for `t2va` and keyframes for `fl2va`. Start with `quality: "lossless"` and the standard 50-point schedule. Here `exact` disables additional request-time approximations; it does not undo INT8 quantization, pruning, or fine-tuning. The author's recommended Ref2V Turbo LoRA is a separate optional adapter, not an automatically inferred 4-step schedule. Reference fidelity can differ from the official model; see the [author's guidance](https://huggingface.co/WarmBloodAban/Minimax-h3_Singularity/discussions/1) before applying the [LoRA sampling contract](#5-lora-recipes). ### Advanced: precomputed AdaLN cache The [model card](https://huggingface.co/MiniMaxAI/MiniMax-H3) notes that about 13B H3 parameters are AdaLN branches whose outputs can be precomputed for inference. The public base checkpoint contains the original branches, not a ready-to-use cache. SGLang therefore keeps the standard path as the default. This is an experimental deployment path. It is intentionally disabled unless you provide an explicitly generated cache; end-to-end numerical and peak-memory validation remains required before using it in production. When an inference-only deployment has a fixed sampling schedule, build a cache from the already materialized transformer directory on CUDA, then pass it to the usual `sglang serve` command. This does not alter the denoising formula: the cache stores the BF16 outputs of the original AdaLN linears. ```bash Command theme={null} python -m sglang.multimodal_gen.tools.build_minimax_h3_adaln_cache \ --transformer-path "$TRANSFORMER_PATH" \ --model-variant fl2va \ --mode t2va \ --num-inference-steps 50 \ --flow-shift 12 \ --audio-flow-shift 3 \ --output /models/minimax-h3-fl2va-adaln-50step.safetensors sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --minimax-h3-adaln-cache-path /models/minimax-h3-fl2va-adaln-50step.safetensors \ --num-gpus 4 \ --tp-size 2 \ --ulysses-degree 2 \ --port 30010 ``` `$TRANSFORMER_PATH` is the `FL2VA/transformer` or `Ref2VA/transformer` directory in the normal SGLang/Hugging Face snapshot; the builder never downloads a second copy. A cache only covers the scheduler settings used to create it, including its mode, step count, flow shifts, and condition noise values. SGLang rejects a request outside that coverage instead of silently changing conditioning. Cache mode supports the matching unquantized checkpoint only. ### Advanced: online AdaLN rebuild with a host cache `--minimax-h3-adaln-online true` needs no prebuilt artifact: the server drops the 24.2 GiB of `adaln_proj` weights from the GPU and computes each request's AdaLN outputs from the checkpoint on demand, bit-exact with the resident-weight path. Because a rebuild pass streams the whole 24.2 GiB, the first request of every new `(task shape, num_inference_steps, flow_shift, audio_flow_shift)` combination pays several seconds; after that the plans are served from a 64-slot GPU slab (LRU per plan) backed by a pinned host cache (`--minimax-h3-adaln-host-cache-gb`, LRU per schedule, default 8 GB per rank), so mixed-schedule serving does not re-read the checkpoint. Plan sets that exceed the host budget are simply recomputed on their next occurrence. Expert escape hatches live in environment variables: `SGLANG_DIFFUSION_MINIMAX_H3_ADALN_GPU_PLANS` resizes the GPU slab (needed only beyond 65 inference steps) and `SGLANG_DIFFUSION_MINIMAX_H3_ADALN_FP32` computes the one-time projections in fp32 (experimental; not bit-comparable to resident weights, validate end-to-end before production). LoRA adapters that modify `adaln_proj` are rejected in both cache modes rather than silently ignored. Both cache modes hold values derived from `adaln_proj`, so a runtime weight update is only accepted when the cache can follow it: online mode takes a disk update whose target directory carries native `adaln_proj` safetensors, and rejects everything else (tensor updates, and directories without those tensors) before a single weight is written. A sidecar is built offline and cannot be regenerated in the server, so weight updates are rejected outright; rebuild the sidecar against the new weights and restart. ### Serve MiniMax-H3 on Ascend NPUs For Ascend NPU, follow the [NPU installation guide](/docs/hardware-platforms/ascend-npus/getting-started/installation) before starting the server. The Ascend commands below explicitly enable the Cache-DiT configuration used for the reported performance measurements. Remove these `SGLANG_CACHE_DIT_*` variables to use lossless denoising. See the **Ascend NPU topology comparison** in the Benchmarks section for the measured eight- and four-NPU latency. The measured latency configuration also passes `--dit-cpu-offload false` to keep the transformer resident on the NPUs. Omit this flag when lower device memory usage is more important than avoiding CPU-to-NPU transfer latency. For an eight-NPU host, the validated topology is TP2 + SP4 with Laser Attention. Use Ascend Flash Attention by replacing `laser_attn` with `fa`. ```bash 8-NPU theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ HCCL_BUFFSIZE=256 sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-type diffusion \ --model-variant fl2va \ --dit-cpu-offload false \ --num-gpus 8 \ --tp-size 2 \ --sp-degree 4 \ --attention-backend laser_attn \ --port 30088 \ --component-residency text_encoder=layerwise-offload ``` For a four-NPU host, use TP2 + SP2: ```bash 4-NPU theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ HCCL_BUFFSIZE=256 sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-type diffusion \ --model-variant fl2va \ --dit-cpu-offload false \ --num-gpus 4 \ --tp-size 2 \ --sp-degree 2 \ --attention-backend laser_attn \ --port 30088 \ --component-residency text_encoder=layerwise-offload ``` ## 4. Generate video and audio `num_inference_steps: N` executes **N denoising updates** over N+1 sigma points, including the terminal zero. Reference implementations that count sigma points need N+1 to reproduce this schedule. Older SGLang versions ran N-1 updates for the same request; use N-1 here to preserve their sampling trajectory. Changing the step count changes the trajectory, not just latency. MiniMax-H3 uses the asynchronous OpenAI-compatible video endpoint. Choose a generation mode below, submit a job, poll its status, and then download the completed MP4. MiniMax-H3 supports output durations from 4 through 15 seconds, inclusive. The following request keeps the verified 5-second profile at a 768-pixel short edge. MiniMax-H3 resolves the aligned output canvas and frame count from `target`. ```bash Command theme={null} video_id=$( curl -sS -X POST http://127.0.0.1:30010/v1/videos \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-H3", "prompt": "At night, while their owner sleeps in a bedroom, three cats march in loudly playing tiny brass instruments, then abruptly file out.", "seconds": 5, "task": "t2va", "conditions": [], "target": { "short_edge": 768, "aspect_ratio": "16:9", "duration_seconds": 5.0 }, "num_outputs_per_prompt": 1, "num_inference_steps": 50, "flow_shift": 12.0, "audio_flow_shift": 3.0, "seed": 1101 }' | jq -r '.id' ) while true; do status=$(curl -sS "http://127.0.0.1:30010/v1/videos/${video_id}" | jq -r '.status') [ "$status" = "completed" ] && break [ "$status" = "failed" ] && exit 1 sleep 1 done curl -sS -L "http://127.0.0.1:30010/v1/videos/${video_id}/content" \ -o minimax-h3-t2va.mp4 ``` The output contract is an MP4 containing H.264 video at 24 fps and one AAC stereo audio stream at 32 kHz. For `fl2va`, provide one or two image conditions with role `keyframe`. The supported frame-index sets are `[0]`, `[-1]`, and `[0, -1]`. The following request uses one server-local first frame. Use `frame_index: -1` for a last frame, or include both entries for first-and-last conditioning. Choose FL2VA when the supplied image should be the actual first or last frame of the generated clip. Use image-based Ref2VA instead when the image should guide identity, style, or composition without being preserved as an endpoint; Ref2VA may recompose or crop the reference. ```bash Command theme={null} curl -sS -X POST http://127.0.0.1:30010/v1/videos \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-H3", "prompt": "The supplied frame continues with calm, natural motion and synchronized ambient sound.", "seconds": 5, "task": "fl2va", "conditions": [ { "type": "image", "uri": "file:///data/minimax-h3/first-frame.png", "role": "keyframe", "frame_index": 0 } ], "target": { "short_edge": 768, "aspect_ratio": "auto", "duration_seconds": 5.0 }, "num_outputs_per_prompt": 1, "num_inference_steps": 50, "flow_shift": 12.0, "audio_flow_shift": 3.0, "seed": 2101 }' ``` V2V uses the reference-conditioning weights. Launch the server with `--model-variant ref2va`, keep the request `task` set to `ref2va`, and provide a video reference in `conditions`. There is no separate `v2v` task value. Use `type: "video"` when the input may be silent. If the file has a soundtrack, H3 also uses it as an audio reference. Use `type: "video_audio"` only when both streams are required; that form rejects an input without audio. The prompt tag for the visual stream is ` For `ref2va`, first launch the reference-conditioning capability with `--model-variant ref2va`, then provide conditions with role `reference`. Image, video, and audio references can be combined. Material tags in the prompt use the one-based order for each modality. An image condition here is semantic reference material rather than a pixel-aligned first frame. Use the FL2VA tab when animating a screenshot from that exact starting composition. ```bash Command theme={null} curl -sS -X POST http://127.0.0.1:30010/v1/videos \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-H3", "prompt": "Use as the visual subject and Poll and download any conditioned request with the same job-status and content endpoints used in the T2VA example. Server-local `file://` URIs must refer to files visible inside the SGLang server environment. ## 5. LoRA recipes H3 accepts native fused adapters, standard Diffusers/PEFT adapters, and Kohya `lora_unet_blocks_*` exports from `networks.lora_minimax_h3`. Native adapters target modules such as `blocks.*.attn.qkv_proj`; PEFT adapters may instead provide separate `to_q`, `to_k`, and `to_v` projections and the `default` adapter namespace. SGLang normalizes all three layouts. The following pinned FL2VA adapters have distinct purposes: | Recipe | Repository and pinned file | Request setting | Prompt requirement | | - | - | - | - | | Recommended speed/quality balance | [`larryvrh/MiniMax-H3-Turbo-Lora`](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora), `minimax_h3_turbo_v4_step600_ema.safetensors` | `num_inference_steps: 8` (8 denoiser evaluations), `lora_scale: 1.0` | None | | Most aggressive speed preset (standard PEFT layout) | [`lightx2v/Minimax-h3-Turbo`](https://huggingface.co/lightx2v/Minimax-h3-Turbo), `minimax_h3_fl2v_turbo_4step_v0.1.safetensors` | `num_inference_steps: 4` (4 denoiser evaluations), `lora_scale: 1.0`, `lora_alpha: 8` | None | | Realistic people style | [`fal/MiniMax-H3-Realism-People-LoRA`](https://huggingface.co/fal/MiniMax-H3-Realism-People-LoRA), `h3-realism-people-t2v-i2v-r2v.safetensors` | Keep the normal `num_inference_steps: 50` schedule; start with `lora_scale: 0.7` | Include `r34l1sm` in the prompt | | Spatial and physics | [`Jojocodex/minimax-h3-spatial-physics-lora`](https://huggingface.co/Jojocodex/minimax-h3-spatial-physics-lora), `wushu_spatial_physics_clean_3000_pruned.safetensors` | Keep the normal `num_inference_steps: 50` schedule; start with `lora_scale: 0.4` | None. Describe collisions, stacking, and gravity-consistent motion in the prompt | | Wushu action | [`Jojocodex/minimax-h3-wushu-action-lora`](https://huggingface.co/Jojocodex/minimax-h3-wushu-action-lora), `wushu_action_h3_lora_v5_3000_pruned.safetensors` | Keep the normal `num_inference_steps: 50` schedule; start with `lora_scale: 0.8` | Put `wushu_action` at the start of the prompt | | Camera motion | [`Jojocodex/minimax-h3-yunjing-lora`](https://huggingface.co/Jojocodex/minimax-h3-yunjing-lora), `camera_motion_h3_lora_v1_3000_pruned.safetensors` | Keep the normal `num_inference_steps: 50` schedule; start with `lora_scale: 0.8` | Put `camera motion` at the start of the prompt | | Looping sketch anime | [`Inner-Reflections/MiniMax-H3-Looping-Sketch-Anime`](https://huggingface.co/Inner-Reflections/MiniMax-H3-Looping-Sketch-Anime), `minimax_h3_looping_sketch_anime_v1.safetensors` | Keep the normal `num_inference_steps: 50` schedule; start with `lora_scale: 1.0` | Include the author's hand-drawn sketch style phrase in the prompt | | 3D-to-real detail slider | [`siraxe/3d_to_real_detail_slider_H3`](https://huggingface.co/siraxe/3d_to_real_detail_slider_H3), `3d_to_real_H3_detail_slider_bf16.safetensors` | Keep the normal `num_inference_steps: 50` schedule; start with `lora_scale: 1.0` | None. Sweep the scale; this is a slider, not a fixed style | The H3 request field controls the number of denoiser evaluations. An 8-step adapter uses `8`, and a 4-step adapter uses `4`; the sigma schedule includes one additional terminal zero. Every recipe uses the same launch shape. Pinning the filename is required for repositories that publish multiple revisions, and is also recommended for a reproducible single-file recipe: ```bash Command theme={null} LORA_REPO=larryvrh/MiniMax-H3-Turbo-Lora LORA_FILE=minimax_h3_turbo_v4_step600_ema.safetensors LORA_NAME=h3-turbo-v4 LORA_SCALE=1.0 LORA_ALPHA_ARGS=() # LightX2V only: LORA_ALPHA_ARGS=(--lora-alpha 8) sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 4 \ --ulysses-degree 4 \ --performance-mode speed \ --lora-path "$LORA_REPO" \ --lora-weight-name "$LORA_FILE" \ --lora-nickname "$LORA_NAME" \ --lora-scale "$LORA_SCALE" \ "${LORA_ALPHA_ARGS[@]}" \ --lora-merge-mode auto \ --port 30010 ``` `auto` merges an adapter into ordinary resident weights to avoid per-step LoRA matmuls. It keeps the dynamic path in two cases. One is FSDP-sharded weights, where a full gather can increase peak memory. The other is an adapter whose update would mostly round away in a BF16 merge. Distilled adapters such as the Larry Turbo LoRA sit below a BF16 ulp of most weights, and a merge drops 77–97% of their update in typical layers. Use `dynamic` when one resident server must switch repeatedly between base and LoRA output, and `merge` to force a merge. Use the filename, scale, and request schedule from the table together. The 4-evaluation LightX2V recipe is the more aggressive latency/quality tradeoff. Its checkpoint has rank 128 but omits the training alpha from both the file and repository metadata, so `--lora-alpha 8` is required to reproduce the author's reference implementation. Start with the Larry 8-evaluation recipe when preserving fine visual detail is more important than minimum latency. The pinned files above were trained for the **FL2VA** partition and apply to `t2va` or `fl2va` requests. Some repositories, including LightX2V, publish separate files for Ref2VA/Ref2V; select one explicitly for a `ref2va` server rather than reusing an FL2VA file. Those Ref2VA files are not yet a pinned, validated recipe on this page. Also avoid stacking a distilled adapter with `quality: "high"`: both alter denoising, and that combination has not been quality-validated. AdaLN-pruned Diffusers components that publish `adaln_basis` and `adaln_mean` can also consume a LoRA trained against the released full-width AdaLN modules: SGLang projects those adapter factors onto the pruned coordinates at load time. A structurally modified checkpoint without that metadata still fails closed, and packed GGUF weights remain incompatible with LoRA. ### PDD acceleration LoRAs [Alibaba PAI's MiniMax-H3-Acc-LoRAs](https://huggingface.co/alibaba-pai/MiniMax-H3-Acc-LoRAs) use Parallel Decoding Distillation (PDD) for eight denoiser evaluations. These checkpoints contain backbone LoRA weights and interval-specific output heads. Merge the backbone weights offline, then fuse the output heads before serving. The `--lora-path` recipe above does not perform these preparation steps. Match the adapter and original MiniMax-H3 transformer weights to the server's model variant: | Model variant | Adapter file | Tasks | | - | - | - | | `fl2va` | `MiniMax-H3-FL2VA-Acc-8Step.safetensors` | `t2va`, `fl2va` | | `ref2va` | `MiniMax-H3-Ref2VA-Acc-8Step.safetensors` | `ref2va` | `BASE_TRANSFORMER_DIR` is the directory containing the original model's BF16/FP32 transformer weight files, not the model root or the LoRA directory. If you downloaded `MiniMaxAI/MiniMax-H3` to `/path/to/MiniMax-H3`, use `/path/to/MiniMax-H3/FL2VA/transformer` for T2VA/FL2VA or `/path/to/MiniMax-H3/Ref2VA/transformer` for Ref2VA. This directory contains `model-*.safetensors`, `model.safetensors.index.json`, and `config.json`. The builder reads these files and writes the merged weights to `$PDD_DIR/transformer`, leaving the original weights unchanged. The following example prepares FL2VA weights and serves them on eight NVIDIA CUDA GPUs with TP2 + Ulysses4: ```bash Command theme={null} BASE_TRANSFORMER_DIR=/path/to/MiniMax-H3/FL2VA/transformer LORA_DIR=/path/to/h3-acc-loras PDD_DIR=/path/to/h3-fl2va-pdd hf download alibaba-pai/MiniMax-H3-Acc-LoRAs \ MiniMax-H3-FL2VA-Acc-8Step.safetensors --local-dir "$LORA_DIR" python3 -m sglang.multimodal_gen.tools.build_minimax_h3_pdd_weights \ "$BASE_TRANSFORMER_DIR" \ "$LORA_DIR/MiniMax-H3-FL2VA-Acc-8Step.safetensors" \ "$PDD_DIR" python3 -m sglang.multimodal_gen.tools.fuse_minimax_h3_pdd_heads \ "$PDD_DIR" --video-shift 12.0 --audio-shift 3.0 SGLANG_DIFFUSION_MINIMAX_H3_PDD_HEADS="$PDD_DIR/pdd_fused_heads.safetensors" \ sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --transformer-weights-path "$PDD_DIR/transformer" \ --num-gpus 8 \ --tp-size 2 \ --ulysses-degree 4 \ --performance-mode speed \ --port 30010 ``` Keep `pdd_config.json` beside `pdd_fused_heads.safetensors`, outside the `transformer/` directory. For Ref2VA, use its adapter and base transformer, a separate output directory, and `--model-variant ref2va`. Use the requests in [Generate video and audio](#4-generate-video-and-audio) with `num_inference_steps: 8`, `flow_shift: 12.0`, and `audio_flow_shift: 3.0`. The nine sigma grid points give eight denoiser evaluations. These settings must match the fused heads; the server rejects a different step count or shift. Use `quality: "lossless"` in requests. ## 6. FastH3: 8-step distilled [FastVideo/FastVideo-FastH3-8-Step-V2](https://huggingface.co/FastVideo/FastVideo-FastH3-8-Step-V2) is an 8-step DMD2 distillation of MiniMax-H3, trained data-free with Video Sparse Attention (VSA) at 0.8 sparsity and 64-token tiles, with video / audio scheduler shifts 10 / 3. Only the T2VA capability was distilled: requests must use `task: "t2va"`, and `fl2va` / `ref2va` requests are rejected. The checkpoint inherits the MiniMax-H3 Community License. Pass the repository directly to `--model-path`. The flat native-Diffusers upload is materialized into the base-H3 layout through a registered model overlay; the only non-symlink step is a one-time re-serialization of the roughly 10 GB video VAE on first launch. ```bash 4×GB300 VSA-H3 theme={null} sglang serve \ --model-path FastVideo/FastVideo-FastH3-8-Step-V2 \ --num-gpus 4 \ --component-attention-backends transformer=video_sparse_attn_h3 \ --port 30010 ``` Requests use the same asynchronous video endpoint as the base model, with `task: "t2va"`, `conditions: []`, and a target such as `{"short_edge": 768, "aspect_ratio": "16:9", "duration_seconds": 5.0}`. The request default is `num_inference_steps: 8`: nine sigma grid points, i.e. the eight distilled DiT evaluations on the checkpoint's trained rungs `[999, 874, 749, 624, 500, 375, 250, 125]` (unshifted timesteps over 1000, shifted once per modality, as FastVideo runs them). Any other step count is rejected. The trained sparsity 0.8 is the default; `--attention-backend-config '{"VSA_sparsity": ...}'` overrides it. FastH3 requests default to `quality: "lossless"`, which fuses the VAE decoder's per-head QK RMSNorm and RoPE and runs its attention on cuDNN SDPA; `quality: "exact"` keeps the reference decoder kernels. At every quality level FastH3 decodes each GPU's spatial VAE tiles as one batch rather than one at a time, which changes GEMM shapes and so the output at rounding level; base MiniMax-H3 decodes tile by tile. `video_sparse_attn_h3` (VSA-H3) is the trained sparse policy: in-tree block-sparse kernels (FastVideo's native tcgen05 forward on SM100 / SM103, a Triton tile-64 kernel on SM90) over segment-pure prefix tiles and (4, 4, 4) video tiles, driven by the checkpoint's trained `to_gate_compress` compression branch. Select it for the transformer only, as with cube sparse attention: the packed DiT blocks run sparse while the token refiner falls back to dense FA inside the backend, and the text encoder and VAEs keep their own backends. Ulysses sequence parallelism is supported. See [Attention Backends](/docs/sglang-diffusion/attention_backends) for `VSA_sparsity`, `vsa_mode`, `vsa_dense_first_n_steps`, and `vsa_dense_layers`. On CUDA, FastH3 selects VSA-H3 for the transformer when no transformer backend is given, unless torch.compile, the breakable CUDA graph or ring attention is on (VSA-H3 serves none of them). Every dense backend that runs on base H3 also runs on the FastH3 weights, without the compression branch: select it with `--component-attention-backends transformer=fa` (or `torch_sdpa`, ...). On 4× H200 dense FA is about 25% slower end to end than VSA-H3 (12.6 s against 10.0 s). `sglang generate` takes the same flags as `sglang serve`. Measured latencies for the 4× GB300 recipe are in [FastH3 on GB300](#fasth3-on-gb300). FastH3 rejects deployment options that do not apply to the distilled checkpoint instead of silently ignoring them: `--model-variant`, `quality: "high"`, `fl2va` / `ref2va` requests, and, with VSA-H3, `--ring-degree` greater than 1, `torch.compile`, and breakable CUDA graph execution. Quality gaps versus base H3 on hard motion and fine detail are properties of the released distillation, not of the SGLang port. Use base MiniMax-H3 when output quality matters more than latency. ## 7. VDN-H3: hybrid attention, 8-step distill [Video DeltaNet](https://openvdn.github.io/) (VDN) is a hybrid attention architecture that approaches the speed of linear attention while maintaining near-lossless quality. It splits the video-video attention into two complementary branches: a sliding-window softmax branch for local frame pairs (five consecutive latent frames form a chunk, and each chunk attends to itself, the previous chunk, and the following chunk; the first and last frames, text and audio stay dense) and a bidirectional linear branch, Video Delta Attention, for long-range context. [OpenVDN/vdn-minimax-h3](https://huggingface.co/OpenVDN/vdn-minimax-h3) applies it to MiniMax-H3: the released `stage-dmd-step-250` checkpoint adds a 4.3 GB linear branch and OpenVDN's own 8-step DMD2 LoRA on the unchanged H3 backbone. It is the FL2VA partition, so it serves `t2va` and `fl2va` (first, last, or both keyframes); `ref2va` is rejected. The weights inherit the MiniMax-H3 Community License. Pass the repository directly to `--model-path`. On first launch a registered overlay prefuses both adapters into the transformer weights (a 62 GB write) and attaches the linear branch; the Qwen3-VL conditioner and the VAEs are hard-linked from `MiniMaxAI/MiniMax-H3`. Keep `SGLANG_DIFFUSION_CACHE_ROOT` on the same filesystem as the Hugging Face cache with at least 90 GB free. ```bash 8×B200 theme={null} sglang serve \ --model-path OpenVDN/vdn-minimax-h3 \ --num-gpus 8 \ --attention-backend hybrid_window_attn_h3 \ --performance-mode speed \ --warmup-num-frames 345 \ --warmup-resolutions 1344x768 \ --port 30010 ``` `--num-gpus 4`, `2` or `1` gives the smaller Ulysses recipes in the benchmark table; `sglang generate` takes the same flags. `--warmup-num-frames` / `--warmup-resolutions` warm up at the clip length and canvas you will serve; without them the first long request pays 2 to 3 s of allocator growth and kernel setup. Requests use the same video endpoint as the base model with `task: "t2va"` and `conditions: []`, or `task: "fl2va"` with the [keyframe conditions](#4-generate-video-and-audio), and a target such as `{"short_edge": 768, "aspect_ratio": "16:9", "duration_seconds": 14.375}` (the 345-frame paper workload). The default `num_inference_steps: 8` is the only accepted value: nine sigma grid points, eight DiT forwards. Each keyframe adds about 2,000 dense rows, so an `fl2va` forward costs a little more than `t2va`. `hybrid_window_attn_h3` is required: a dense backend would silently skip the linear branch and the gates and produce the wrong model. On Blackwell (SM100 / SM103 / SM120) the transformer defaults to online MXFP8; `--quantization bf16` opts out. Before SM100, `--quantization fp8` is base H3's per-channel fp8 path, and SM80 runs the bf16 DiT (62 GB; budget for `--layerwise-offload-components`). Ampere and Ada are enabled but not benchmarked. Ulysses sequence parallelism is supported; `--model-variant`, `quality: "high"`, `--ring-degree` greater than 1, `torch.compile`, and breakable CUDA graph execution are rejected. The backend's `--attention-backend-config` keys are listed in [Attention Backends](/docs/sglang-diffusion/attention_backends). Measured latencies for 1, 2, 4 and 8 B200s are in [VDN-H3 on B200](#vdn-h3-on-b200); single-card and PCIe multi-card numbers for the RTX PRO 6000 are in [VDN-H3 on RTX PRO 6000](#vdn-h3-on-rtx-pro-6000). ## 8. Sampling and output controls MiniMax-H3 supports more than one output per prompt. The video API accepts `num_outputs_per_prompt` (or OpenAI-compatible `n`) from 1 through 10. Offline generation accepts `--num-outputs-per-prompt N`; `--num-outputs N` is the short alias. A scalar seed is expanded deterministically as `seed + output_index`, so the outputs do not reuse the same noise. Same-prompt fan-out reuses text conditioning. On the verified 2× RTX 5090 recipe, a 5-step two-output request completed in 155.39 seconds versus 78.11 seconds for one output, while producing two distinct valid MP4 files. The independent denoise and decode passes remain sequential on this 32 GB profile to keep peak memory bounded; the grouped path adds essentially no orchestration overhead. Use server replicas when lower wall-clock latency for many variants matters more than per-server memory efficiency. For example, set `"num_outputs_per_prompt": 2` in any request above. After the job completes, download both outputs by selecting each zero-based variant: ```bash Command theme={null} video_id="" for variant in 0 1; do curl -sS -L \ "http://127.0.0.1:30010/v1/videos/${video_id}/content?variant=${variant}" \ -o "minimax-h3-${variant}.mp4" done ``` ### Choose the quality level `quality` is a cumulative request-scoped optimization parameter with three levels: * `"exact"`: the exact reference path. Output is bit-exact against the reference implementation and the CI ground truth. * `"lossless"` (default): includes the global fusion-only tier but does not enable Cache-DiT or another approximate optimization. Each DiT block runs its RMSNorm and AdaLN modulation as one kernel that rounds once instead of after every op, and VAE decode fuses the ViT decoder's per-head QK RMSNorm and RoPE into one launch and runs its attention on cuDNN SDPA (rounding-level differences only). * `"high"`: the audited accelerated path. Quality is guaranteed (the audited Cache-DiT configuration measures SSIM 0.931 / PSNR 28.16 dB against `exact`), but output is no longer bit-identical to the reference. One resident server serves all three levels; a `quality: "high"` request mounts its audited Cache-DiT policy at the batch boundary, and a later `quality: "exact"` or `quality: "lossless"` request removes the hooks before denoising. Start the validated server once: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 4 \ --tp-size 1 \ --sp-degree 4 \ --ulysses-degree 4 \ --ring-degree 1 \ --encoder-parallel auto \ --performance-mode speed \ --use-fsdp-inference false \ --enable-torch-compile false \ --port 30010 ``` Then choose a request level: Native denoising and VAE decode with no feature-cache approximation or quality-gated VAE fusions. ```json Request field theme={null} { "quality": "exact" } ``` The global fusion-only tier. It does not enable MiniMax-H3 Cache-DiT. It differs from `exact` only in rounding: the DiT blocks' fused RMSNorm + AdaLN modulation, and the VAE decoder's fused QK RMSNorm + RoPE and cuDNN SDPA attention. Omitting the field selects this level. ```json Request field theme={null} { "quality": "lossless" } ``` The audited accelerated path. Use it when you can trade bit-exactness for latency while keeping output closest to the same-seed exact trajectory. ```json Request field theme={null} { "quality": "high" } ``` The measured trade-off is: | `quality` | Mean
inference
latency | Speedup | SSIM vs
exact | PSNR vs
exact | Expected
trade-off | | - | -: | -: | -: | -: | - | | `exact` | 75.10 s | 1.00× | 1.000 | exact | Native reference path | | `lossless` | Not separately measured | — | Same H3 denoise path | Same H3 denoise path | Fusion-only tier; VAE decode fuses QK RMSNorm + RoPE and uses cuDNN SDPA (rounding-level differences) | | `high` | 53.70 s | 1.40× | 0.931 | 28.16 dB | Smallest same-seed visual change | These numbers use 1344×768, 124-frame, 24 fps T2VA with 50 inference steps, video flow shift 12, audio flow shift 3, and three fixed prompt/seed pairs on 4×H200. The prompts cover a quiet detailed scene, fast multi-subject action, and a moving close-up portrait. `inference_time_s` is averaged across the three prompts; the quiet-scene point is itself the mean of two repeats. SSIM and PSNR compare decoded, frame-aligned output with the `exact` result for the same prompt and seed. They measure trajectory deviation, not absolute perceptual quality: the `high` path can produce a different but still plausible realization. It also changes the joint audio-video denoise trajectory, while these two metrics cover video only. `quality: "high"` currently accepts only the exact workload and 4×H200 deployment above; other hardware, task modes, request shapes, step counts, or flow shifts fail before denoising. Offline generation uses the same level name, for example `sglang generate --quality high`. `quality` selects a model sampling level and can change generated content. `output_quality` controls only output-file compression; it is a separate field. For manually tuned Cache-DiT experiments outside that validated path, omit the request `quality` field and set `--enable-cache-dit` or the process-wide `SGLANG_CACHE_DIT_*` defaults. Any explicit `quality`, including `"exact"` and `"lossless"`, takes H3 off the generic Cache-DiT path. The 24 GB layerwise recipe above can use the same switch; skipped blocks are not streamed. ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=1 \ SGLANG_CACHE_DIT_BN=0 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.12 \ SGLANG_CACHE_DIT_MC=2 \ sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant ref2va \ --num-gpus 8 \ --ulysses-degree 8 \ --encoder-parallel auto \ --performance-mode speed \ --port 30010 ``` Cache-DiT skips selected block computation and is approximate. It cannot be combined with FSDP inference. DiT layerwise offload is compatible: skipped blocks are not streamed, and the first layer after a skip may sync-load. Breakable CUDA graph execution takes precedence and leaves Cache-DiT disabled. Tune the cache thresholds only after comparing both video and audio quality on the target task profile. A real B200 request has completed, but the `quality: "high"` path above remains fail-closed to the audited 4×H200 workload. ## 9. Feature contracts and advanced recipes The generated command already contains the recommended topology and encoder setting. Use the detailed reference below only when applying an optional override or checking its installation, topology limits, and validation evidence. The recommended `speed` launch already combines resident components with Ulysses sequence parallelism. Validation status below applies only to the listed hardware and topology; it is not inherited by a similar GPU family. | Feature | Validation status | Notes | | - | - | - | | Ulysses sequence parallelism | Verified: 8× B200, 4× H200, 4× H100, and Ulysses1/2/4/8 on MI300X and MI355X | Use `--ulysses-degree`. Combine with Ring for cross-node scaling; see the next row. | | Ring sequence parallelism (cross-node) | Verified: 2 nodes of 8× H200 each (Ulysses8 × Ring2) | Use `--ring-degree` together with `--nnodes`/`--node-rank`/`--dist-init-addr`. Ring shards the sequence across nodes while Ulysses shards heads within a node; H3's packed multi-segment attention only supports Ring across the node boundary, not within a single node's Ulysses group. Requires `--encoder-parallel replicate` — `auto`'s fold decision is not node-boundary aware. See the benchmark section below. | | Tensor parallelism | Verified: B200 TP2 + Ulysses4; H100 TP2 + Ulysses2 and TP4 + Ulysses1 | `--tp-size` may be combined with Ulysses when the TP-local head count remains divisible by the Ulysses degree. On 4×H100, TP2 + Ulysses2 is the measured speed default. | | FSDP inference | Verified: 4× B200 and 4× H100 + Ulysses4 | Preserves H3's mixed BF16/FP32 parameter policy. B200 completed the exact eager comparison; H100 completed consecutive real requests at about 57 GB peak memory per GPU. | | Resident components | Verified: B200, H200, 4×H100 with TP, and 1/2/4/8× MI300X and MI355X | This is the recommended single-request latency path when the complete workload fits. | | CPU and layerwise offload | Verified: 2× RTX 5090 TP2; 1× RTX 4090 24 GB | The 5090 lossless recipe keeps 20 DiT blocks plus both VAE encoders resident, streams the remaining DiT blocks, text encoder, and video VAE decoder blocks, and leaves the small audio VAE resident. The 4090 recipe streams DiT and the text encoder with zero resident DiT layers and **omits `vae`** from `--layerwise-offload-components`. Compatible with Cache-DiT; skipped blocks are not streamed. | | Breakable CUDA graph | Verified: B200 Ref2VA, opt-in | Matching eager output was observed for the captured signature, without a measured speedup. Re-capture for other shapes and reference sets. | | `torch.compile` | Measured: H200, opt-in | Steady-state benefit was below measurement noise, while startup increased and numerical output changed. Do not use it for consistency ground truth. | The verified parallel, placement, and matching-signature BCG paths keep the BF16/FP32 weights and denoising math. `torch.compile` is the exception called out above. Always use the eager BF16/FP32 launch when producing CI consistency ground truth. For the validated 1344×768 Ref2VA profile, use a 5504-row text bucket so both the server warmup and reference-conditioned requests share the captured signature: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant ref2va \ --num-gpus 8 \ --ulysses-degree 8 \ --encoder-parallel auto \ --performance-mode speed \ --enable-breakable-cuda-graph true \ --warmup-resolutions 1344x768 \ --bcg-text-buckets 5504 \ --port 30010 ``` BCG is lossless for a matching captured signature, but capture reserves extra GPU memory. Re-measure the live H3 text length before reusing this bucket for a different task profile, reference set, resolution, or prompt template. Leave `--attention-backend` unset for the platform default. Use `--attention-backend fa` only for an explicit FlashAttention comparison. SageAttention uses quantized attention math and is not a consistency mode. To select H3's native packed-varlen Sage path, install the dependency and add `--attention-backend sage_attn`. On Hopper, install the upstream SM90 binding fix rather than the PyPI 2.2.0 build: ```bash Command theme={null} pip install --force-reinstall \ git+https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4 \ --no-build-isolation ``` The backend is a server-wide default. Use `--component-attention-backends` only when a measured component needs a different kernel, and keep the platform default for every component not named in the override. On the verified 8× B200 topology, quantize the BF16 transformer at server load: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant ref2va \ --num-gpus 8 \ --ulysses-degree 8 \ --encoder-parallel auto \ --performance-mode speed \ --quantization fp8 \ --port 30010 ``` H3 automatically keeps its video/audio patch projections, timestep MLP, and final video/audio heads in FP32. All other linear layers have stable full module prefixes, so additional layers can be kept unquantized: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant ref2va \ --num-gpus 8 \ --ulysses-degree 8 \ --encoder-parallel auto \ --quantization fp8 \ --quantization-ignored-layers blocks.0.attn token_refiner \ --port 30010 ``` Online FP8 is approximate and is not a consistency ground-truth mode. It can be combined with Cache-DiT, but the two approximations compound. Validate visual quality, audio quality, memory use, and latency on the target workload. This recipe is limited to the resident B200 and B300 topologies used for real H3 validation runs. On a single 24 GB card, use `convrot_int8` instead of FP8. It quantizes the four GEMMs per DiT block online from the Hub BF16 weights (data-free, no calibration) and runs them on `comfy_kitchen.int8_linear` on the 4090 (CC 8.9) or on SGLang's JIT-compiled ConvRot ops on CC 9.0, 10.0, 12.0 and 12.1 (see [Quantization](/docs/sglang-diffusion/quantization#convrot-int8)). Quantization happens after H3's grouped `qkv` reorder, so do not load an externally pre-quantized INT8 checkpoint here. ```bash Command theme={null} sglang generate \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --quantization convrot_int8 \ --attention-backend fa \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder \ --dit-offload-prefetch-size 1 \ --dit-layerwise-resident-layers 0 \ --enable-torch-compile false \ --prompt "A cat walking on a sunny beach, gentle waves." \ --save-output ``` `fa` keeps exact attention. For a faster, approximate DiT path, use `--attention-backend sol_attn` with `--attention-backend-config dense_backend=sage_attn,dense_steps=10` and `--component-attention-backends text_encoder=torch_sdpa,transformer=sol_attn`. See [Quantization](/docs/sglang-diffusion/quantization#convrot-int8) and [Attention Backends](/docs/sglang-diffusion/attention_backends#sage-then-sol-hybrid). `convrot_int8` changes Linear numerics. `sol_attn` / `sage_attn` also change the attention algorithm. Neither is a consistency ground-truth mode. The BF16 path never depends on `comfy-kitchen`; `convrot_int8` needs it only on GPUs the JIT ops do not cover (any CC other than 9.0, 10.0, 12.0 and 12.1) or when no CUDA toolchain is available to build them. Pre-quantized and compact H3 text encoders are listed once in [Checkpoint and adapter formats](#checkpoint-and-adapter-formats). They use component-local paths and never inherit the DiT's `--quantization` setting. SGLang reads their metadata before constructing the native Qwen3-VL encoder and fails closed when the selected format, projection, or topology is incompatible. Cube sparse attention applies TopK sparsity only to H3's 3D visual streams. Text, audio, standalone reference images, and the text-only token refiner stay dense. It runs on pure PyTorch plus FlexAttention, so it has no third-party kernel dependency. Select it for the H3 transformer with `--component-attention-backends transformer=cube_sparse_attn` and pass `--attention-backend-config` with both `local_cube_size` and `topk_ratio_list`. Scoping the backend leaves the text encoder on its native backend: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant ref2va \ --num-gpus 8 \ --ulysses-degree 8 \ --performance-mode speed \ --component-attention-backends transformer=cube_sparse_attn \ --attention-backend-config '{"local_cube_size": [4, 4, 4], "topk_ratio_list": [1.0, 1.0, 0.8, 0.7, 0.6, 0.5, 0.5]}' \ --port 30010 ``` * `local_cube_size` is the `(T, H, W)` cube that groups neighboring latent tokens into one attention block. It must have exactly three entries. * `topk_ratio_list` sets the per-step keep ratio and must have exactly one entry per denoise step, each in `(0, 1]`. A ratio of `1.0` keeps a step on H3's native dense attention path; smaller values select the FlexAttention sparse path and drop more blocks. The example above matches a request with `num_inference_steps: 7`, whose endpoint-inclusive sigma schedule has seven denoise updates. Cube labeling is coordinate-driven. FL2VA keyframes share the target video's position grid, so a keyframe token and a target token at the same `(T, H, W)` coordinate receive the same semantic cube label. Duplicate coordinates do not extend the temporal grid; a semantic cube can therefore span multiple physical attention blocks. In Ref2VA, standalone reference images remain dense, while reference videos and the target video contribute to one global TopK candidate pool rather than receiving separate per-stream quotas. Cube sparse attention is an approximate backend and is not a consistency ground-truth mode. `topk_ratio_list` length must equal the denoise step count or the server rejects the request. Cube sparse attention does not support Ring parallelism; use `--ulysses-degree` without `--ring-degree`. FlexAttention's routing overhead can outweigh sparse-kernel savings on short sequences, so benchmark latency as well as visual and audio quality on the target workload. SubBlock sparse attention applies training-free block sparsity to MiniMax-H3's long, non-causal DiT self-attention. The BF16 path supports CUDA SM90, SM100, and SM120. Other compute capabilities, including SM103/B300, are rejected. Ulysses sequence parallelism is supported; Ring parallelism is not. The global backend selects the H3 DiT lazily, so keep the Qwen text encoder on its compatible dense backend with a component override: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 8 \ --ulysses-degree 8 \ --attention-backend subblock_sparse_attn \ --component-attention-backends text_encoder=fa \ --attention-backend-config '{"sparsity": 0.75, "skip_first_steps": 10}' \ --port 30010 ``` Use `text_encoder=torch_sdpa` on SM120. The default configuration drops at most 75% of key blocks after the first 10 denoise forwards; short sequences, cross-attention, and unsupported shapes continue on the dense path. Tune `sparsity` and `skip_first_steps` together with visual and audio quality checks. On H100/H200 (SM90), `compute_mode=sage_fp8` switches the sparse kernel to online INT8 Q/K and FP8 P/V compute. Install the optional kernel and add the compute mode: ```bash Command theme={null} pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 8 \ --ulysses-degree 8 \ --attention-backend subblock_sparse_attn \ --component-attention-backends text_encoder=fa \ --attention-backend-config '{"compute_mode": "sage_fp8", "sparsity": 0.75}' \ --port 30010 ``` Both SubBlock routing and `sage_fp8` are approximate. Use dense BF16 attention for consistency ground truth, and validate the selected sparsity on the target resolution, duration, task, and checkpoint. The picker explicitly writes `--encoder-parallel auto` in every single-node recipe. At the default request batch size of one, H100/H200/B200/B300 servers with peer-to-peer access fold the Qwen encoder across otherwise idle Ulysses ranks. A pure-TP recipe keeps the encoder inside its TP group, while a PCIe-only host can avoid an expensive world fold. Keep `auto` unless one of the cases below applies. Encoder DP is a throughput policy for compatible request batches. It requires TP1 and DiT DP1, replicates the encoder weights, and does not improve a batch of one: ```bash Overlay theme={null} --encoder-parallel dp \ --batching-max-size 2 ``` The cross-node picker recipe already uses replication because the automatic fold decision is not node-boundary aware: ```bash Overlay theme={null} --encoder-parallel replicate ``` ## 10. Configuration notes * MiniMax-H3 produces the canonical 24 fps output; request duration is expressed through `target.duration_seconds`. * `target.duration_seconds` must be between 4 and 15 seconds, inclusive. The command picker defaults to the verified 5-second profile. * Use a 768-pixel short edge for the released quality recipe. The aligned output dimensions are derived from `target.aspect_ratio`. * `flow_shift` controls video diffusion and `audio_flow_shift` controls audio diffusion. * V2V uses `task: "ref2va"` with a `video` or `video_audio` reference; it is served by the `Ref2VA` partition and is not a separate public task value. * `conditions[].start_time_seconds` selects a non-negative offset for a video reference. Its visual and audio streams are always sought together. * Ref2VA condition order is semantic and must match the one-based material tags in the prompt. For Ref2VA, `target.aspect_ratio: "auto"` resolves to the model's 16:9 fallback rather than inheriting a reference asset's geometry. * The distilled pipeline uses a single denoising branch, so CFG parallelism does not apply. Keep it disabled: a CFG degree greater than 1 is accepted for request compatibility, but the extra ranks only recompute the same branch and do not accelerate the request. Explicitly disabling CFG, or setting its size to 1, remains a valid no-op. * The released visual VAE quality recipe uses overlapping tiled decode. SGLang keeps that recipe by default and distributes complete tiles across the decode group; this changes scheduling, not the computation inside each tile. * H3 rejects `--vae-config.parallel-decode-mode spatial` and `spatial_shard`: validation found output mismatches. Use the default released tiled recipe. * Keep the default `--encoder-parallel auto`. With the server’s default `batching_max_size` of 1, single-node H100/H200/B200/B300 recipes with peer-to-peer access fold the Qwen text encoder over otherwise idle Ulysses ranks. This is separate from DiT tensor parallelism. A pure-TP recipe already shards the encoder over its TP group and does not add a world fold. * Single-node Ulysses recipes pipeline each block's all-to-all against dense attention (FlashAttention on Hopper, cuDNN SDPA on Blackwell) over the copy engine: one head group's q/k/v moves while the previous group attends, every copy (the merge into the output included) runs on the copy engine, and the output is bit-identical to the sequential exchange. It needs peer-to-peer access and stream memory operations on every rank and about 6 × (rows per rank) × heads × head\_dim extra activations per rank (2.1 GB for Ref2VA at Ulysses 2), one set shared by every resolution; when that does not fit on some rank, every rank keeps the sequential exchange. `SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS=0` restores the sequential exchange. * For throughput-oriented serving, select **DP (batched throughput)**. The picker pairs `--encoder-parallel dp` with an editable `--batching-max-size` greater than 1. Encoder DP stays inside each DiT replica and composes with encoder TP: the H100 TP2 + Ulysses2 recipe has two TP-sharded encoder copies that can split a batch, while the RTX 5090 pure-TP2 recipe has one encoder copy and therefore no additional batch-DP degree. It provides no benefit for a batch of one and is not bitwise-identical to the unsplit deployment. * Use explicit **Fold** to prioritize single-request latency and encoder memory on a measured high-bandwidth single-node topology. Use **Replicate** as the compatibility path when folding or encoder DP is unsuitable. * `--use-fsdp-inference true` shards only the DiT. MiniMax-H3 preserves the original FP32 dtype of its patch, time, and output projections during FSDP all-gather, so this path does not trade numerical correctness for memory. On 4×H100, prefer TP2 + Ulysses2 for speed; use FSDP as an explicit capacity policy rather than assuming it is faster. * `speed` keeps model components resident, while `auto` applies the model-aware 120 GiB residency threshold. `memory` prioritizes avoiding OOM and includes the executable VAE decoder in its default layerwise set. A measured recipe with sufficient headroom can opt into `--component-residency vae=resident`; the 2×H100 CI recipe does this because the VAE's 4.8 GiB/GPU cost avoids repeated decoder transfers during tiled decode. DiT residency and prefetch knobs remain scoped to the DiT. Use `speed` only after confirming that the complete target workload fits. * Breakable CUDA graph execution is an explicit opt-in, not part of the recommended `speed` preset. It requires `--enable-breakable-cuda-graph`, every served size in `--warmup-resolutions`, and `--bcg-text-buckets` that cover the live H3 condition sequence. The validated 1344×768 Ref2VA recipe uses 5504; other task profiles and reference sets may need a different value. It preserves eager output for matching captured signatures, but graph capture consumes additional GPU memory and may provide little latency benefit when Ulysses attention and collectives dominate, so benchmark it on the target topology before enabling it. * `--enable-spectrum` skips selected DiT steps. On H3 it cannot combine with Cache-DiT / `quality="high"` or SP > 1. 50-step 4090 recipe is in §3. ## 11. Benchmarks The picker exposes resident and FSDP profiles on NVIDIA datacenter GPUs. GPU counts are properties of the selected recipes, not a claim that every platform requires that many GPUs. The detailed tables below report performance only for the configurations with collected measurements: | Hardware | Default resident recipe | Other profile or topology | | - | - | - | | B300 | 8× Ulysses8 resident | 8× FSDP + Ulysses8; the 8-GPU sweep is not a minimum-GPU claim. | | B200 | 8× Ulysses8 resident | 4× FSDP + Ulysses4 | | GB300 | 4× Ulysses4 resident (FL2VA T2VA) | 2 nodes × 4× Ulysses4×Ring2 is unverified. | | GB200 | 4× Ulysses4 resident, derived/unverified | No performance measurements. | | H200 | 4× Ulysses4 resident | 4× FSDP + Ulysses4; 4× TP2 + Ulysses2; 2 nodes × 8× Ulysses8×Ring2 cross-node | | H100 | 4× TP2 + Ulysses2 resident | 4× TP4 + Ulysses1; 4× FSDP + Ulysses4 | | Ascend NPU | 8 NPUs, TP2 + SP4, Laser Attention | 4 NPUs, TP2 + SP2, Laser Attention | | MI300X / MI355X | 8× Ulysses8 resident | 1×, 2×, and 4× scaling runs | | RTX 5090 | 1× layerwise offload, measured on a physical desktop | 2× TP2 + layerwise offload | | RTX 4090 24 GB | 1× layerwise offload + `convrot_int8` | Spectrum and approximate attention backends are opt-in | ### GB300 single-host Four requests on **1 host × 4 NVIDIA GB300**, using Linux ARM64, SGLang main [`15d2cbcc90fc`](https://github.com/sgl-project/sglang/commit/15d2cbcc90fc66b4d08745994cc707a45b067ffc), PyTorch 2.13.0+cu130, and NCCL 2.29.7. The source checkout was installed with `python3 -m pip install -e "python[diffusion]" --upgrade` inside `lmsysorg/sglang:latest`; this is a source-validation environment, not a claim that the image's bundled release contains that revision. The command matches the GB300 default in the builder: Ulysses4, resident weights, automatic encoder folding, native BF16/FP32, platform-default Dynamic cuDNN/FA attention, and eager execution. The model was downloaded by `sglang serve` from `MiniMaxAI/MiniMax-H3`. The workload uses the builder's cat-band prompt, FL2VA T2VA, a 5-second request at a 768-pixel short edge, 50 inference steps, `quality="exact"`, one output, and seed 1101. Every MP4 contained video and audio and passed a full decode check. | First full request after server warmup | Following three requests | Warm median | | -: | - | -: | | 35.16 s | 33.11 / 33.10 / 33.10 s | **33.10 s** | These are client-observed completion times, including up to one second of status-polling delay. Download, model loading, and server warmup are excluded. This is a deployment smoke/latency measurement, not a cross-framework quality audit, a minimum-GPU claim, or an extrapolation to GB200/NVL72 scaling. ### Ascend NPU topology comparison Both topologies used Laser Attention and the explicit Cache-DiT configuration from the Ascend launch commands, with `--dit-cpu-offload false` keeping the DiT resident. The measured workload was one 5-second T2VA request at 1344×768, 124 frames, 24 fps, and 50 inference steps. | NPU count | Topology | End-to-end latency | | -: | - | -: | | 8 | TP2 + SP4 | **55.07 s** | | 4 | TP2 + SP2 | **103.57 s** | These are individual end-to-end measurements for each topology, not averages. The eight-NPU topology had 46.8% lower end-to-end latency than the four-NPU topology. ### B300 precision and encoder placement A 12-configuration sweep on a single 8× B300 host, covering both checkpoint partitions, both transformer precisions, and all three text-encoder placements. It answers one question — *how long does one request take, and how much memory does it need*. ### What was measured **Hardware.** 8× NVIDIA B300 SXM6, single node. **Model.** `MiniMaxAI/MiniMax-H3`, both released weight partitions. **Serve command.** Exactly the recipe the picker emits for B300, plus the one or two overlay flags under test: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 8 \ --ulysses-degree 8 \ --performance-mode speed \ --host 0.0.0.0 \ --port 30010 ``` The swept axes are `--model-variant` (`fl2va` / `ref2va`), `--quantization` (unset for BF16 / `fp8`), and `--encoder-parallel` (`auto` / `fold` / `replicate`). Nothing else differs between the 12 servers. This is a single-request latency sweep (`batching_max_size: 1`), so encoder DP is intentionally excluded: it cannot distribute a batch of one. Use the **DP for a request batch** setting above for a compatible multi-request deployment; the table below does not claim a measured H3 DP speedup. **Driver.** ```bash Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --host 127.0.0.1 --port 30010 \ --model MiniMaxAI/MiniMax-H3 \ --dataset vbench --task text-to-video \ --num-prompts 1 --max-concurrency 1 \ --warmup-requests 1 --warmup-inference-steps 50 \ --extra-body '{"task":"t2va","conditions":[],"target":{"short_edge":768,"aspect_ratio":"16:9","duration_seconds":5.0},"seconds":5,"flow_shift":12.0,"audio_flow_shift":3.0}' ``` **Workload** | Property | Value | | - | - | | Output duration | 5.167 s | | Resolution | 1344×768 | | Frames | 124 @ 24 fps | | Denoising steps | 50 | | `flow_shift` / `audio_flow_shift` | 12.0 / 3.0 | | Requests in flight | 1 (`--max-concurrency 1`, server at `batching_max_size: 1`) | | Requests measured | 1 per cell, after 1 warmup request | ### Results | Weights | Precision | Encoder | Load | Warmup | Latency | Peak/GPU | | - | - | - | -: | -: | -: | -: | | FL2VA | BF16 | auto | 118.1 s | 29.65 s | **19.04 s** | 83,578 MB | | FL2VA | BF16 | fold | 114.0 s | 28.72 s | **19.04 s** | 83,578 MB | | FL2VA | BF16 | replicate | 116.0 s | 28.33 s | **19.04 s** | 124,158 MB | | FL2VA | FP8 | auto | 116.0 s | 27.16 s | **18.03 s** | 51,926 MB | | FL2VA | FP8 | fold | 116.0 s | 25.99 s | **18.04 s** | 51,926 MB | | FL2VA | FP8 | replicate | 118.0 s | 27.97 s | **18.04 s** | 92,506 MB | | Ref2VA | BF16 | auto | 114.0 s | 38.69 s | **29.12 s** | 83,968 MB | | Ref2VA | BF16 | fold | 118.0 s | 36.58 s | **29.13 s** | 83,968 MB | | Ref2VA | BF16 | replicate | 116.0 s | 35.17 s | **29.13 s** | 124,490 MB | | Ref2VA | FP8 | auto | 124.0 s | 34.30 s | **27.12 s** | 52,816 MB | | Ref2VA | FP8 | fold | 112.0 s | 34.44 s | **27.12 s** | 52,816 MB | | Ref2VA | FP8 | replicate | 116.0 s | 33.42 s | **27.12 s** | 93,396 MB | ### FastH3 on GB300 A 4× GB300 host served [FastH3](#6-fasth3-8-step-distilled) with the VSA-H3 recipe above (1344×768 at 24 fps with audio, `task: "t2va"`, eight DiT forwards, the default `quality: "extra-high"` and sparsity 0.8, eager BF16, Ulysses4, `NCCL_NVLS_ENABLE=0`), measured with the MP4 encoded while the VAE decodes ([sgl-project/sglang#41819](https://github.com/sgl-project/sglang/pull/41819)). The workload is 72 requests at concurrency 1 after one warm request per shape: six scene families, 1,000- and 10,000-token prompts, 5 / 10 / 15 s, two seeds, every prompt unique so no conditioning cache hits. E2E is the client wall clock from submission to the last byte of the MP4; the stage columns are the server timings of the same requests. Each cell is the median of the 12 requests of that shape: | Prompt / requested / aligned | Text encoder | Denoise (8 forwards) | Decode (video + audio) | E2E | Peak/GPU | | - | -: | -: | -: | -: | -: | | 1K / 5 s / 124 | 0.10 s | 3.84 s | 0.56 s | **5.20 s** | 94,468 MB | | 1K / 10 s / 243 | 0.10 s | 8.45 s | 1.12 s | **10.74 s** | 97,628 MB | | 1K / 15 s / 362 | 0.10 s | 14.85 s | 1.72 s | **18.24 s** | 100,246 MB | | 10K / 5 s / 124 | 0.18 s | 6.31 s | 0.55 s | **7.90 s** | 94,510 MB | | 10K / 10 s / 243 | 0.18 s | 12.35 s | 1.12 s | **14.98 s** | 98,128 MB | | 10K / 15 s / 362 | 0.18 s | 20.28 s | 1.67 s | **24.01 s** | 100,408 MB | Over all 72 requests the E2E p50 / p90 / p95 is 12.94 / 23.94 / 24.08 s, and every MP4 carries 1344×768 H.264 at 24 fps with a 32 kHz stereo AAC track. ### VDN-H3 on B200 An 8× B200 (SM100, 183 GB) host served [VDN-H3](#7-vdn-h3-hybrid-attention-8-step-distill) on the paper workload: 1344×768 at 24 fps with audio for 14.375 s (345 frames, 102 latent frames, about 104k packed rows), `task: "t2va"`, nine sigma grid points (8 DiT forwards), seed 1000, `hybrid_window_attn_h3`, eager, and the warmup run at the served clip shape (`--warmup-num-frames 345 --warmup-resolutions 1344x768`), so every forward of the served request is steady state; without those flags the first forward pays 2 to 3 s of allocator growth and kernel setup. "Steady s/NFE" is the mean of forwards 2 to 8. The OpenVDN reference rows ran the released inference stack (`8nfe_tuned_fp8.yaml`, `infer_ulysses.py`) on the same host and the same clip length, with its Ulysses layout (`parallel.softmax_ranks`) swept at every GPU count rather than assumed. Every row below comes from one campaign, one arm at a time on an otherwise idle node; a repeated arm moves by about 3% across nodes, so treat smaller differences as noise: | Config | Steady s/NFE | Denoise (8 forwards) | Decode | Peak/GPU | | - | -: | -: | -: | -: | | OpenVDN published, FP8, 8× B200 Ulysses 5+3 ([openvdn.github.io](https://openvdn.github.io/)) | 1.40 | 11.2 s | — | — | | OpenVDN reference, BF16 (`8nfe_tuned.yaml`), 1× B200 | 7.66 | 61.2 s | — | — | | OpenVDN reference, FP8, 1× B200 | 6.25 | 50.0 s | — | — | | OpenVDN reference, FP8, 2× B200 Ulysses (standard / 1+1 branch-parallel) | 4.87 / 5.04 | 39.0 / 40.3 s | — | — | | OpenVDN reference, FP8, 4× B200 Ulysses (3+1 branch-parallel) | 2.59 | 20.7 s | — | — | | OpenVDN reference, FP8, 8× B200 Ulysses (5+3 / 4+4 / standard) | 1.40 / 1.47 / 1.52 | 11.2 / 11.7 / 12.2 s | — | — | | SGLang, `--quantization bf16`, 1× B200 | 7.45 | 58.8 s | 8.8 s | 156,298 MB | | SGLang, `--quantization fp8` (→ online `mxfp8` on SM100+), 1× B200 | 6.03 | 47.6 s | 8.7 s | 128,418 MB | | SGLang default (online `mxfp8`), 2× B200 Ulysses2, `--performance-mode speed`, served-shape warmup | **3.30** | **25.9 s** | 4.7 s | 106,738 MB | | SGLang, `--quantization bf16`, 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | 2.06 | 16.2 s | 2.3 s | 108,434 MB | | SGLang default (online `mxfp8`), 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | **1.67** | **13.1 s** | 2.4 s | 85,244 MB | | SGLang default (online `mxfp8`), 8× B200 Ulysses8, `--performance-mode speed`, served-shape warmup | **0.88** | **6.9 s** | 1.5 s | 79,972 MB | | SGLang, per-channel fp8 weight scales (the online `fp8` path of other models; on SM100+ `--quantization fp8` resolves to `mxfp8`, so this arm selects the per-channel path explicitly), 8× B200 Ulysses8 | 0.98 | 7.7 s | 1.5 s | 79,504 MB | On 8× B200 the SGLang Ulysses8 path runs the paper workload at 0.88 s/NFE (7.0 GPU-seconds per NFE) against the published 1.40 s/NFE of OpenVDN's 5+3 branch-parallel layout; the whole request (text encoding, 8 forwards, joint decode) completes in about 8.7 s after warmup. At this GPU count the step is launch- and copy-bound in the linear branch rather than FLOP-bound, and the transformer runs online `mxfp8` by default (e4m3 with one E8M0 scale per 32 elements into cuBLASLt's block-scaled GEMM, with the activation quant fused into the adaLN and SwiGLU producers). Against the published 8× B200 headline (1.40 s/NFE, 11.2 GPU-seconds per NFE), the SGLang 4× B200 run spends 6.7 GPU-seconds per NFE, with half the GPUs and half the all-to-all fan-out. At equal GPU count SGLang is 1.48× faster at two cards (3.30 vs 4.87 s/NFE), 1.55× at four (1.67 vs 2.59) and 1.60× at eight (0.88 vs 1.40), while on one card the two stacks are within 4% of each other (6.03 vs 6.25 s/NFE quantized, 7.45 vs 7.66 in BF16). The advantage is therefore parallel scaling rather than per-GPU kernel work: SGLang holds 86-91% parallel efficiency from 2 to 8 cards against the reference stack's 56-64%, and sweeping the reference stack's own layout does not close it (its best split is 8% better than standard Ulysses at eight cards, 1.40 against 1.52 s/NFE). On one B200 both single-GPU rows run with the DiT resident under the auto memory policy (128 GB peak quantized, 156 GB in BF16, of the card's 183 GB), so no `--layerwise-offload-components` flag is needed at this clip length and the single-GPU rows are compute-bound. ### VDN-H3 on RTX PRO 6000 The same paper workload (1344×768, 345 frames, 8 DiT forwards, seed 1000, `hybrid_window_attn_h3`, `--performance-mode speed`, served-shape warmup) on RTX PRO 6000 Blackwell Server Edition cards (SM120, 96 GB, PCIe, no NVLink). The window runs on FA4's sm120 kernel and the transformer defaults to online `mxfp8` (a cutlass sm120 block-scaled GEMM); `--quantization fp8` maps to it. One card needs `--layerwise-offload-components text_encoder`: the fp8 DiT (33 GB) and the bf16 text encoder (48 GB) do not both fit, and the whole-module `--text-encoder-cpu-offload` re-homes the encoder in one piece and runs out of memory when it is used. Multi-card Ulysses goes over PCIe; the all-to-all is the bulk of the step there (54% of GPU time at 8 cards), so the per-card efficiency drops with count. A 2-card run should use a pair on one PCIe switch (`nvidia-smi topo -m` shows `PIX`): a pair across the CPU root complex measured 15.7 to 17.0 s/NFE on a shared host. | Config | Steady s/NFE | Denoise (8 forwards) | Decode | Peak/GPU | | - | -: | -: | -: | -: | | 1× RTX PRO 6000, `--layerwise-offload-components text_encoder` (default `mxfp8`) | **20.27** | **160.0 s** | 20.6 s | 95,790 MB | | 1× RTX PRO 6000, same, per-channel `fp8` path (before this mapping) | 22.85 | 180.3 s | 20.7 s | 95,160 MB | | 2× RTX PRO 6000 Ulysses2, one PCIe switch (default `mxfp8`) | 13.61 | 107.4 s | 10.6 s | 93,846 MB | | 2× RTX PRO 6000 Ulysses2, one PCIe switch, per-channel `fp8` | 14.97 | 118.1 s | 10.6 s | 94,700 MB | | 4× RTX PRO 6000 Ulysses4 (default `mxfp8`) | 7.98 | 63.0 s | 5.6 s | 82,178 MB | | 4× RTX PRO 6000 Ulysses4, per-channel `fp8` | 8.23 | 65.1 s | 5.6 s | 82,422 MB | | 8× RTX PRO 6000 Ulysses8 (default `mxfp8`) | 5.43 | 43.1 s | 3.5 s | 77,704 MB | | 8× RTX PRO 6000 Ulysses8, per-channel `fp8` | 5.47 | 43.4 s | 3.5 s | 77,236 MB | For comparison the same code on B200 measures 6.03 / 3.30 / 1.67 / 0.88 s/NFE at 1 / 2 / 4 / 8 cards (NVLink, 86% parallel efficiency at 8). ### H200 topology comparison The same four-card H200 host completed both lossless resident placements with the standard 1344×768, 5-second, 50-step T2VA request (fixed prompt and seed, eager BF16/FP32, back-to-back runs on an otherwise idle host). Latency is the warmed-up request; the first pair uses the default warmup request, the second pair adds `--warmup-resolutions 1344x768` so warmup already covers the served resolution: | Topology | Warmup | Denoise | Decode | E2E | Peak/GPU | | - | - | -: | -: | -: | -: | | Ulysses4 | default | 79.04 s | 3.77 s | **84.14 s** | 94,288 MB | | TP2 + Ulysses2 | default | 81.17 s | 2.97 s | 85.51 s | 63,490 MB | | Ulysses4 | `--warmup-resolutions 1344x768` | 71.73 s | 1.32 s | **74.38 s** | 94,290 MB | | TP2 + Ulysses2 | `--warmup-resolutions 1344x768` | 75.52 s | 1.29 s | 78.33 s | 63,490 MB | Ulysses4 stays the H200 latency default: 5.0 % faster end-to-end than TP2 + Ulysses2 once warmup covers the served resolution (1.6 % with the default warmup, where first-request cold start masks the topology gap). TP2 + Ulysses2 shards the DiT weights and holds peak memory about 30 GB per GPU lower, which is why it remains the 80 GB H100 recipe. Matching the warmup request to the served resolution removes the cold first-request cost on both topologies (about 10 s end-to-end on this workload). ### H200 cross-node scaling Long references and long durations grow the packed sequence length, and Ulysses alone cannot scale sequence parallelism past the GPU count of one node without either violating head-count divisibility or exposing all-to-all traffic across the slower inter-node link. H3 combines node-local Ulysses with cross-node Ring: Ring's point-to-point KV rotation is designed to overlap with attention compute, which fits a slower cross-node link better than an all-to-all does. **Hardware.** 2 nodes × 8× NVIDIA H200 SXM, same cluster, InfiniBand between nodes. **Serve command.** The cross-node cell the picker emits for H200, run identically on both nodes with `--node-rank` set to 0 and 1: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant ref2va \ --num-gpus 16 \ --nnodes 2 \ --node-rank {{NODE_RANK}} \ --dist-init-addr {{NODE0_IP}}:20000 \ --sp-degree 16 \ --ulysses-degree 8 \ --ring-degree 2 \ --encoder-parallel replicate \ --performance-mode speed \ --host 0.0.0.0 \ --port 30010 ``` **What was measured.** A controlled denoise-stage comparison on identical hardware: 8× H200 single-node (Ulysses8, no Ring) versus the same 16-GPU cross-node command above (Ulysses8 × Ring2), holding prompt, seed, and step count fixed: | Task | Single-node (Ulysses8) | Cross-node (Ulysses8 × Ring2) | Change | | - | -: | -: | -: | | T2VA denoise/step | 0.749 s | 0.477 s | −36.3% | | Ref2VA/V2V denoise/step | 2.572 s | 1.494 s | −41.9% | The gain grows with sequence length because Ring's per-hop communication cost stays roughly constant while attention compute grows quadratically with sequence length, so V2V's longer packed sequence benefits more than T2VA's shorter one. With the point-to-point KV rotation pipelined against attention compute, one V2V request's full denoise stage completed in 68.1–68.3 seconds versus 128.6 seconds on the single-node 8-GPU baseline (−47.0%), with byte-identical output to the unpipelined cross-node path. Cross-node determinism was confirmed separately: the same request run twice against the same cross-node deployment produced byte-identical output. A cross-node run's output is not expected to bit-match a single-node run of the same prompt and seed — Ring's online-softmax merge across hops accumulates floating-point operations in a different order than single-node attention, which is an expected source of bit-level difference, not a correctness regression. `--encoder-parallel auto`'s fold decision is not yet node-boundary aware and attempts to fold the text encoder across nodes, which crashes the Ref2VA reference-conditioned encoder. Always pass `--encoder-parallel replicate` explicitly for cross-node H3 deployments. ### H100 topology comparison The same four-card H100 host completed three lossless placements. TP2 with Ulysses2 was the fastest; TP4 used the least memory: | Topology | Pipeline latency | Peak/GPU | | - | -: | -: | | TP2 + Ulysses2 | 13.25 s | 66.04 GB | | FSDP + Ulysses4 | 13.36 s | 57.01 GB | | TP4 + Ulysses1 | 13.86 s | 49.80 GB | ### RTX 5090 capacity run (two cards) The single-card recipe is a different one — see *RTX 5090 single card, physical desktop* below; the twenty resident layers here are half a layer per GPU under TP2 and do not transfer to one card. The verified two-card RTX 5090 host used TP2 with layerwise offload. The full 50-step, 1344×768, 5-second request completed in 559.67 seconds: 525.05 seconds of denoising and 33.61 seconds of decoding, with a 26.3 GiB sampled peak per GPU. | DiT settings | 5-step denoise | Inference | Peak/GPU | Result | | - | -: | -: | -: | - | | prefetch 1, resident 20 | 43.48 s | 78.11 s | 26.3 GiB | Selected recipe | | prefetch 2, resident 20 | 43.37 s | 78.06 s | 27.5 GiB | No measurable gain | | Ulysses2, prefetch 2, resident 10 | Did not reach warmup | — | — | Rejected | ### Consumer GPU tuning On consumer hardware the binding question is not which card you have but how much host RAM sits behind it. H3's weights are about 108 GB — 61.73 GB of DiT and 46.18 GB of text encoder — so no consumer configuration holds them all, and where the shortfall lands decides the throughput. **The command** — most consumer machines need exactly one flag beyond the model: ```bash consumer single GPU, lossless theme={null} sglang serve --model-path MiniMaxAI/MiniMax-H3 --model-variant fl2va \ --layerwise-offload-components dit,text_encoder,vae ``` With 16 GB of VRAM or more, add `--layerwise-resident-layers video_vae=36` for the 13 s decode; with \~96 GB of host RAM and 16 GB+ of VRAM, add `--dit-layerwise-resident-layers 4` for the 6 s step. That is the whole flag surface. The [builder at the top of this page](#1-quick-start) has consumer cards and a Host RAM selector: pick your budget and it emits this command with your tier's measured expectations attached as comments. The table below is the same data in one view. **Unified-memory machines are the exception**: on a DGX Spark (GB10, 128 GB shared between CPU and GPU) launch with **no flags at all**. The deployment still exceeds the pool, automatic offload engages on its own, and its placement beat the explicit recipe above by 2.1× on the denoise (12.1 vs 25.8 s/it measured on the same box) — the VRAM/host split that justifies every flag in this section does not exist there. Expect \~12 min of load, a \~5.5 min text encoding stage per request (steady-state compute on this chip, not a stall), and \~12 min per warm 480P request. **Two budgets, and what each one buys** | | 12 GB VRAM + 32 GB host | host free, VRAM 16 GB | | - | -: | -: | | Recipe | A | B | | Peak VRAM | ≤ 12 GiB | ≤ 16 GiB (OOMs at 12) | | Host anonymous (must fit) | 24.5 GiB | 116.7 GB pinned | | Denoise, 864×480 / 124 frames / 20 NFE | 16.8 - 18.7 s/it | **6.01 s/it** | | Runs at all | yes | yes | The left column is one configuration measured twice, at 318.94 s and 356.37 s; the 12% spread tracked host load on a shared machine, so treat smaller differences than that as unresolved. The right column is 120.92 s at a 16 GiB allocator cap. Four resident DiT layers is what Recipe B buys its speed with, and it is also why 12 GiB is not enough for it. Read the host row carefully, because the two numbers are not the same kind of memory. *Anonymous* host memory — pinned buffers and pageable copies — has to fit, and the kernel cannot reclaim it. Page cache backing a file mapping is *droppable*, so it does not count against the budget even though it shows up in `VmRSS`; use `RssAnon` from `/proc//status` when checking. Likewise measure VRAM with `torch.cuda.set_per_process_memory_fraction` and let the allocator fail, rather than reading `nvidia-smi`, which reports the caching allocator's reserved pool and overstates the requirement. Inside 32 GB the weights cannot be pinned, so each denoise step copies about 60 GiB from the checkpoint mapping, and a mapped source is synchronous however the copy is requested: the driver stages it through its own buffer, so the transfer neither overlaps compute nor runs at pinned bandwidth. That is where the step goes, and giving the host room to pin the weights instead is what takes it to 6.01 s. Two caveats on the constrained number, both from instrumenting the run rather than from arithmetic. The machine it was measured on has 2 TB of host memory, so the kernel kept all 107.7 GiB of mapped checkpoint pages resident: major faults across a whole request were 6, and `read_bytes` was zero. Nothing was read from disk. A real 32 GB host cannot cache 107.7 GiB, so it will fault and re-read, and should be expected to be slower than the figures here rather than equal to them — an NVMe is a requirement, not a recommendation. Measure your own machine with major faults (`/proc//stat`) on the worker process, not on the launcher, which holds no weights. **Recipe A — fits 12 GB VRAM + 32 GB host** ```bash 12 GB + 32 GB, lossless theme={null} PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ sglang serve --model-path MiniMaxAI/MiniMax-H3 --model-variant fl2va \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder,vae \ --layerwise-resident-layers video_vae=36 ``` **Recipe B — host memory is free (the fast path)** ```bash unconstrained host, lossless theme={null} sglang serve --model-path MiniMaxAI/MiniMax-H3 --model-variant fl2va \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder,vae \ --dit-layerwise-resident-layers 4 \ --layerwise-resident-layers video_vae=36 ``` Recipe B pins \~112 GB of host memory (DiT 61.56 GB, text encoder 46.18 GB, VAE \~4.5 GB in its decode dtype). Do not reach for it on a 32 GB machine. **What not to change, and why** * `video_vae=36` holds every decoder block for the decode only — residency arms at the decoder's first block and releases when it finishes, so the denoise still runs on an empty card. It fits 12 GB because decoder weights are held in their decode compute dtype (fp16) from load, which halves them to \~4.9 GiB; the rounding was already part of every output (the decode computes in fp16 autocast), so the result is bit-identical, and the decode drops from 60 s streamed (or 209 s on a busy host) to \~10 s. The `expandable_segments` line stays: the decode sits close enough to the cap that fragmentation otherwise tips it over. * Leave `--enable-torch-compile` off, as elsewhere on this page. Layerwise offload rebinds `param.data` on every layer, so compiled graphs do not get the benefit they would on resident weights. * Recipe A's flags are what the automatic policy should choose on its own. Until the model declares its own placement, `--performance-mode memory` plus the explicit component list is what makes it happen; pass them. **Reading the startup log** The server prints the memory decisions it made; checking three lines against your budget catches a mis-set machine in the first minute instead of the first request. * `Layerwise offload: host memory available: N GiB` — what the runtime sees after loading, not your DIMM size. On a 32 GB host expect single digits here; a much larger number means another process's memory accounting (or a container limit) is in play. * `leaving N GiB of weights on the checkpoint mapping` — the expected line on a 32 GB host: the DiT streams from the checkpoint file. If instead the log reports pinned weights, the runtime decided your host has room — which is faster, and means the 32 GB figures above do not apply to you. * `Loaded video_vae: ... host mmap` vs `host pageable` — where the VAE landed (decoder weights are \~4.9 GiB once held in their decode dtype). `Loaded ` lines carry the same buckets for every component. If a request dies after the denoise finishes, it is the decode colliding with the cap: keep the `expandable_segments` line, and if it persists drop to `video_vae=24` and take the partially streamed decode. **Against ComfyUI, on the same weights** Same unpruned bf16 checkpoints, same card, same sampler settings (cfg 1.0, euler\_ancestral, sigma shift 12.0/3.0, seed 1101), 864×480 / 124 frames / 20 NFE: When host memory is free, the engines are close and sglang is ahead: | | denoise | host anonymous | peak VRAM | | - | -: | -: | -: | | sglang, Recipe B | **6.01 s/it** | 116.7 GB pinned | ≤ 16 GiB | | ComfyUI KSampler | 6.58–6.59 s/it | 116.5 GiB | 13048 MiB | Inside 12 GB, both engines run these weights, and one measurement convention matters on each side. ComfyUI's memory manager reads system RAM and adapts, so the rows below patch `psutil` to a pretend host size — the same convention the sglang rows use. Its `--reserve-vram` is also soft: told to keep 12 GiB free it still peaked at 13.5 GiB, a figure a real 12 GB card cannot give it, so both engines here run under the same hard allocator cap (`set_per_process_memory_fraction`), where its peak stays at 12.1–12.3 GiB. Under that cap, Recipe A wins the whole request at every host size: | 12 GB VRAM, both engines hard-capped | sglang Recipe A (TE + denoise + decode) | ComfyUI, bf16 (warm) | | - | -: | -: | | 32 GB host | 12.4 + 212.8 + 9.4 ≈ **235 s** | 276–302 s | | 48 GB host | 15.8 + 192.1 + 10.0 ≈ **218 s** | 246–267 s | | 64 GB host | 7.5 + 162.4 + 10.3 ≈ **180 s** | 194–195 s | Same GPU, same load window, unpruned bf16 checkpoints, outputs verified. All figures are anchored at 480P — activations grow with the pixel count, so at 768P drop the resident DiT layers to 0 first, then `video_vae` to 24 if the decode still collides. One convention behind that table is the one a home machine does not share: the host sizes above are `psutil` patches on a 2 TB lab machine, so the kernel kept the whole checkpoint in page cache and nothing was read from disk. A physical host of that size re-reads the streamed weights every step, and what that costs is computable — `bytes per step = (50 - resident DiT layers - layers the host pin budget covered) x 1.23 GiB`. The courier overlaps that read with compute, so the drive stops being the denoise clock once its sequential rate exceeds those bytes divided by the card's step time. A 32 GB host covers roughly 13 layers, which puts a 24 GB card at \~38 GiB/step and a 12/16 GB card at \~46 GiB/step: a PCIe 4.0 NVMe (6–7 GiB/s) clears both, a PCIe 3.0 drive (\~3.2 GiB/s) becomes the clock, and a SATA SSD is not usable for this model. Resident DiT layers remove 1.23 GiB/step each, but only while the streamed set is larger than the pin budget — once the read fits under the step's compute time, more of them buy nothing and only cost VRAM. An earlier revision of this page recorded 38 s/step on a physical 4090 with a 990 Pro, reading 52.9 GB per step — about 1.4 GB/s, and faulted sequentially, so almost none of it showed in `majflt` (measure `read_bytes`, not major faults). That rate was the old path, not the drive: the pinned-store pool rounded every block up to a power of two, so the pin budget over-committed and the machine had to run with pinning off, and the layers that stayed mapped faulted in 4 KiB at a time. Exact-size pinning and O\_DIRECT reads through the courier removed both, and the physical 5090 run below shows the drive fully hidden behind compute. A 4090 on a 32 GB host is derived to return to its compute wall (\~8.5 s/step) on a PCIe 4.0 drive; that machine has not been re-measured. The VRAM axis holds too: capped at 16 GiB the same recipe wins \~250 vs 292–301 s, and at 24 GiB (with `--dit-layerwise-resident-layers 6` — measured at a 22 GiB cap so a desktop's own allocations fit; a headless card can raise it to 10 for under 1% more) \~8.5 s/step vs ComfyUI's 249–260 s requests. Four changes carry it: the VAE staying on its checkpoint mapping (#35862, root fix \#35946), per-layer pinning with net-cost accounting (#35867), the courier thread that ships still-mapped layers through pinned slots (#35882), and decoder weights held in their decode dtype from load (#35967) — which is what lets `video_vae=36` fit and turns the decode from the slowest stage (54–96 s streamed) into the fastest (\~10 s, faster than ComfyUI's own 15–25 s). Output equivalence is bit-level: the fp16-held decode reproduced the fp32-store run's video byte for byte, and the audio stream is bit-identical. Stage by stage under the cap: text encoding is even (both stream the same 48 GB Qwen3VL), the denoise leads at 32–48 GB hosts and sits within run-to-run variance of ComfyUI at 64 GB (162 vs 159 s), and the decode leads everywhere. Two ComfyUI notes that still matter: `--fast-disk` measured no faster than its default on this lab host — though on a physical 60 GB desktop its default is OOM-killed while loading (55.6 GB of anonymous memory) and `--fast-disk` is what makes it run at all — and stacking `--novram --cache-none --disable-pinned-memory` made things strictly worse (69.1 GiB anonymous, 750 s requests) — the adaptive default is the right configuration on a small host. The path ComfyUI ships for 12 GB cards uses `minimax_h3_fl2va_pruned_int8_convrot` and `qwen3vl_32b_minimax_h3_nvfp4_awq`, i.e. an int8 DiT and an NVFP4 text encoder, and its pruned bf16 file is 40.2 GB against the unpruned 66.3 GB. Those are different weights, so it is not a like-for-like comparison with the recipes above. ### RTX 5090 single card, physical desktop Every consumer figure above this point was measured with at least one axis simulated: a hard allocator cap standing in for VRAM, a `psutil` patch standing in for host RAM, and a lab machine whose page cache held the whole checkpoint. This run had none of that — an RTX 5090 (32 GB) in a Ryzen 9 9950X desktop with 60 GB of RAM, an 8 GB swap file and a Samsung 9100 PRO on PCIe 5.0, on Ubuntu 24.04. The workload matches the comparison above: FL2VA T2VA, 864×480, 124 frames, 20 steps, cfg 1.0, euler\_ancestral, sigma shift 12.0/3.0, same seeds and keyframes on both engines, unpruned bf16 weights on both sides. ```bash 32 GB VRAM + 60 GB host, lossless theme={null} PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ sglang serve --model-path MiniMaxAI/MiniMax-H3 --model-variant fl2va \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder,vae \ --dit-layerwise-resident-layers 14 \ --layerwise-resident-layers video_vae=36 \ --warmup-resolutions 864x480 --warmup-num-frames 124 ``` | | text encoding | denoise step | decode | request | | - | -: | -: | -: | -: | | sglang | **4.2 s** | 5.14–5.17 s | **6.4 s** | **112.2 / 112.1 s** | | ComfyUI `--fast-disk --cache-none` | 11.3 s | 5.17–5.30 s | 7.2 s | 145.9 / 145.6 / 140.9 s | Three things a capped run cannot settle: **The denoise is at the card's compute wall, not the drive's.** The two engines are within noise of each other per step, and prefetch depth 2 and cuDNN SDPA each measured as no change. The 51.3 GiB the DiT reads from the drive per request is fully overlapped — across a whole request the compute thread waited 0.9 s for the courier. **Host RAM decides how much traffic there is to hide.** At 60 GB the pin budget covers 34 of the 50 DiT layers, so only about two layers per step come off the disk. The 46 GiB text encoder is never pinned — streamed layers take the budget first — and is read whole, once per request, at the drive's sequential rate: that is the 4.2 s stage, against 12.1–12.9 s when the same bytes go through the page cache. **Neither engine's default command survives this machine.** ComfyUI's is OOM-killed during load (55.6 GB of anonymous memory) and needs `--fast-disk`; sglang's was too, until pinned stores were allocated at their exact size instead of the allocator's next power of two. Even now the automatic placement keeps no resident DiT layers and streams the VAE on every decode, so the flags above are what reach the numbers in the table. Two caveats to carry into your own machine. The run finished with its 8 GB swap file full, because the pin budget takes 95% of the memory available after loading and there is no smaller setting yet — on a machine that is also your desktop, expect that pressure and give the card more resident layers, since a resident layer is one the host no longer has to pin. And these are 480P figures: at 768P drop the resident DiT layers first, then `video_vae` to 24. ### RTX 4090 24 GB single-GPU run One RTX 4090 D 24 GB completed the 1344×768, 107-frame, 20-NFE T2VA workload (euler, `torch.compile` and step caching disabled) with DiT and text-encoder layerwise offload. Same process: load → warmup (seed 0) → timed (seed 42); only the timed pass is reported. GPU peak stayed about 18 GB. | Config | Timed e2e | Denoise | vs BF16 | PSNR vs BF16 | | - | -: | -: | -: | -: | | BF16 + FlashAttention | 405.6 s | 370.2 s | 1.00× | — | | `convrot_int8` + FA | 303.3 s | 273.7 s | 1.34× | 24.81 dB | | `convrot_int8` + `sol_attn` | 223.9 s | 203.9 s | 1.81× | 24.44 dB | | `convrot_int8` + `sage_attn` | 174.9 s | 154.2 s | 2.32× | 23.51 dB | | `convrot_int8` + Sage→Sol hybrid | 163.8 s | 143.1 s | 2.48× | 23.04 dB | `convrot_int8` + FA changes Linear numerics only. The `sol_attn` / `sage_attn` / hybrid rows also change the attention algorithm, so speed and pixel fidelity rank in opposite orders there. Default remains `convrot_int8` + `fa`. Cache-DiT can share this layerwise recipe; omit `quality` and see the quality-level section. Same card, 50-step run. Speedup is vs BF16+FA. PSNR is vs same-tree INT8+FA (Spectrum off). | Config | e2e | vs BF16+FA | PSNR mean / min | | - | -: | -: | -: | | BF16 + FlashAttention | 1008.6 s | 1.00× | — | | `kitchen_int8` + FA | 718.5 s | 1.40× | — | | `kitchen_int8` + FA + Spectrum `11 / 5 / 1.0` | **285.3 s** | **3.53×** | **27.4 / 25.3** | | `kitchen_int8` + `sage_attn` + Spectrum `15 / 5 / 0` | 245.3 s | 4.11× | 23.6 / 21.0 | | `kitchen_int8` + Sage→Sol hybrid + Spectrum `15 / 5 / 0.5` | **219.0 s** | **4.61×** | 20.6 / 17.8 | Prefer FA + `11 / 5 / 1.0` when the bar is FA-like. Sage / hybrid + Spectrum is faster; quality then follows the attention backend, not the skip schedule. ### AMD Instinct task and scaling runs The AMD recipes keep the released BF16/FP32 precision policy and use AITER packed attention. The picker emits the fastest measured topology, 8 GPUs with Ulysses degree 8. All runs below completed full H.264/AAC decoding and representative-frame inspection. | Hardware | Task | Denoise | Decode | Peak/GPU | | - | - | -: | -: | -: | | MI355X | T2VA | 55.2907 s | 9.5344 s | 97,444 MB | | MI355X | FL2VA | 53.7978 s | 9.4477 s | 96,922 MB | | MI355X | Ref2VA | 41.3812 s | 6.8247 s | 94,518 MB | | MI300X | T2VA | 167.4878 s | 25.3244 s | 97,272 MB | | MI300X | FL2VA | 150.2311 s | 12.5684 s | 96,750 MB | | MI300X | Ref2VA | 107.6232 s | 11.3768 s | 94,268 MB | The task matrix used 8 GPUs and 50 denoising steps. The scaling matrix uses one 1344×768, 209-frame T2VA request and changes only the GPU count and matching Ulysses degree: | Hardware | GPUs | Denoise | Decode | Peak/GPU | | - | -: | -: | -: | -: | | MI355X | 8 | 55.2907 s | 9.5344 s | 97,444 MB | | MI355X | 4 | 104.2294 s | 11.1824 s | 103,350 MB | | MI355X | 2 | 223.0246 s | 15.5330 s | 115,250 MB | | MI355X | 1 | 288.7968 s | 24.0472 s | 137,676 MB | | MI300X | 8 | 167.4878 s | 25.3244 s | 97,272 MB | | MI300X | 4 | 297.3727 s | 26.5067 s | 103,436 MB | | MI300X | 2 | 585.5401 s | 29.4909 s | 115,010 MB | | MI300X | 1 | 978.0886 s | 36.0142 s | 137,626 MB | For a measured lower-count AMD deployment, set both `--num-gpus` and `--ulysses-degree` to 4, 2, or 1. AITER packed attention matched segment-wise BF16 SDPA at cosine similarity `0.9999991655` on MI355X and `0.9999991059` on MI300X. ## 10. Run in ComfyUI # Qwen-Image Source: https://docs.sglang.io/cookbook/diffusion/Qwen-Image/Qwen-Image Deploy Qwen-Image with SGLang - community contribution guide for Qwen's image generation model. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [Qwen-Image](https://huggingface.co/Qwen/Qwen-Image) is a 20B text-to-image model built for strong prompt following and precise rendering of English and Chinese text. It is especially useful for posters, signs, diagrams, and dense layouts where typography and spatial relationships matter as much as general image quality. This page covers generation rather than editing: use Qwen-Image-Edit when an existing image, subject identity, or untouched region must be preserved. The full checkpoint is memory-heavy, while the validated ModelOpt NVFP4 release provides a supported low-precision deployment option with an expected quality tradeoff. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration Qwen-Image is a text-to-image model. The recommended launch configurations vary by hardware. SGLang supports serving Qwen-Image on NVIDIA B200, B300, H200, H100, AMD MI300X, MI325X, MI355X GPUs and Ascend A2/A3 Series NPUs. Use the [command builder](#1-quick-start) above. For the validated ModelOpt NVFP4 checkpoint on Blackwell, load the published Qwen-Image-2512 NVFP4 repo directly: ```bash Command theme={null} sglang serve \ --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \ --ulysses-degree=1 \ --ring-degree=1 ``` For high-resolution B200 generations, the FlashInfer CUTLASS FP4 GEMM backend can be faster than the default TensorRT-LLM FP4 GEMM backend: ```bash Command theme={null} SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass \ sglang generate \ --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \ --width 2048 --height 2048 \ --prompt "A tiny astronaut reading a book under a glass greenhouse" \ --save-output ``` ### 3.2 Fixed-resolution latency on two H200 GPUs For `Qwen/Qwen-Image-2512` at 1024x1024, use breakable CUDA graph (BCG) to reduce launch overhead across graph-safe DiT segments while retaining explicit breakpoints around unsupported operations. This recipe was validated on two NVIDIA H200 GPUs with 50 denoising steps and no classifier-free guidance: ```bash Command theme={null} sglang serve \ --model-path Qwen/Qwen-Image-2512 \ --model-type diffusion \ --num-gpus 2 \ --tp-size 2 \ --performance-mode speed \ --dit-layerwise-offload false \ --enable-torch-compile false \ --enable-breakable-cuda-graph \ --warmup-mode server \ --warmup-resolutions 1024x1024 ``` Declare every production resolution in `--warmup-resolutions`. A request at an uncaptured resolution runs eagerly, so omitting `1024x1024` removes the gain from this recipe. Graph capture used about 5 GB more peak memory per GPU in the validation run. On CUDA, the TP path dispatches supported collectives through SRT CustomAllReduceV2. At 1024x1024, Qwen-Image reduces 24 MiB row-parallel outputs; the diffusion runtime reserves a 32 MiB V2 workspace so these collectives do not fall back to NCCL. If profiling shows large NCCL all-reduce kernels again, first confirm that V2 is enabled and the requested shape fits the workspace. BCG changed floating-point execution order but not the sampling algorithm. The fixed-seed output measured 0.984 SSIM and 39.7 dB PSNR against eager output; use eager execution when you require bit-exact output. Regional `torch.compile` was also tested on this profile and did not improve steady-state latency. ### 3.3 Two 32 GB GPUs (RTX 5090) With `--tp-size 2`, the adaLN modulation projections and the text-stream FFN are sharded, so each rank holds about 19 GiB of DiT weights and the DiT stays resident on 32 GB cards: ```bash Command theme={null} sglang serve \ --model-path Qwen/Qwen-Image \ --num-gpus 2 \ --tp-size 2 \ --ulysses-degree 1 \ --performance-mode manual \ --text-encoder-cpu-offload true \ --attention-backend torch_cudnn_sdpa ``` On 2x RTX 5090 (PCIe, no P2P) this measured 19.7 s at 1024x1024 with 50 steps and no CFG. Add `--dit-cpu-offload true` to cut peak memory to about 22 GB per GPU at 21.2 s. ### 3.4 Configuration Tips See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. * `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. * `--num-gpus`: Number of GPUs to use * `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) * `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs) * `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP * `--ring-degree`: The degree of ring attention-style SP in USP **AMD ROCm Notes**: Requires SGLang >= v0.5.8. ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](../../../docs/sglang-diffusion/api/openai_api). ### 4.1 Generate an Image ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1") response = client.images.generate( model="Qwen/Qwen-Image", prompt="A logo With Bold Large text: SGL Diffusion", n=1, response_format="b64_json", ) # Save the generated image image_bytes = base64.b64decode(response.data[0].b64_json) with open("output.png", "wb") as f: f.write(image_bytes) ``` ### 4.2 Advanced Usage #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](../../../docs/sglang-diffusion/cache_dit). **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Qwen/Qwen-Image ``` **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Qwen/Qwen-Image ``` #### 4.2.2 CPU Offload * `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. * `--vae-cpu-offload`: Use CPU offload for VAE. * `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". #### 4.2.3 Known LoRA examples Use `--lora-path` at startup or the [LoRA management API](/docs/sglang-diffusion/api/openai_api#lora-management) to load an adapter. Known Qwen-Image examples include: * [`lightx2v/Qwen-Image-Lightning`](https://huggingface.co/lightx2v/Qwen-Image-Lightning) * [`flymy-ai/qwen-image-realism-lora`](https://huggingface.co/flymy-ai/qwen-image-realism-lora) * [`prithivMLmods/Qwen-Image-HeadshotX`](https://huggingface.co/prithivMLmods/Qwen-Image-HeadshotX) * [`starsfriday/Qwen-Image-EVA-LoRA`](https://huggingface.co/starsfriday/Qwen-Image-EVA-LoRA) ## 5. Benchmark Test Environment: * Hardware: AMD Instinct MI300X GPU (1x) * Model: Qwen/Qwen-Image * Docker Image: lmsysorg/sglang:v0.5.8-rocm700-mi30x * sglang diffusion version: 0.5.8 ### 5.1 Speedup Benchmark #### 5.1.1 Generate an image **Server Command**: ```shell Command theme={null} sglang serve --model-path Qwen/Qwen-Image \ --ulysses-degree=1 --ring-degree=1 --port 30000 ``` **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Qwen/Qwen-Image Dataset: vbench -------------------------------------------------- Benchmark duration (s): 29.04 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.03 Latency Mean (s): 29.0378 Latency Median (s): 29.0378 Latency P99 (s): 29.0378 -------------------------------------------------- Peak Memory Max (MB): 48018.83 Peak Memory Mean (MB): 48018.83 Peak Memory Median (MB): 48018.83 ============================================================ ``` **Server Command**: ```shell Command theme={null} #One A3 Series card has 2 npu chips sglang serve --tp-size 2 --sp-degree 1 --model-path Qwen/Qwen-Image --num-gpus 2 ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Qwen/Qwen-Image Dataset: vbench -------------------------------------------------- Benchmark duration (s): 36.26 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 Completed outputs: 1 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.03 Output throughput (outputs/s): 0.03 Latency Mean (s): 36.26 Latency Median (s): 36.26 Latency P90 (s): 36.26 Latency P95 (s): 36.26 Latency P99 (s): 36.26 -------------------------------------------------- Peak Memory Max (MB): 36984.00 Peak Memory Mean (MB): 36984.00 Peak Memory Median (MB): 36984.00 ------------------------------------------------------------ ``` #### 5.1.2 Generate images with high concurrency **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 --port 30000 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Qwen/Qwen-Image Dataset: vbench -------------------------------------------------- Benchmark duration (s): 300.79 Request rate: inf Max request concurrency: 20 Successful requests: 14/20 -------------------------------------------------- Request throughput (req/s): 0.05 Latency Mean (s): 154.5368 Latency Median (s): 154.8363 Latency P99 (s): 285.4603 -------------------------------------------------- Peak Memory Max (MB): 48030.31 Peak Memory Mean (MB): 48030.30 Peak Memory Median (MB): 48030.29 ============================================================ ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Qwen/Qwen-Image Dataset: vbench -------------------------------------------------- Benchmark duration (s): 300.81 Request rate: inf Max request concurrency: 20 Successful requests: 8/20 Completed outputs: 8 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.03 Output throughput (outputs/s): 0.03 Latency Mean (s): 166.61 Latency Median (s): 167.02 Latency P90 (s): 270.80 Latency P95 (s): 283.48 Latency P99 (s): 293.64 -------------------------------------------------- Peak Memory Max (MB): 36984.00 Peak Memory Mean (MB): 36984.00 Peak Memory Median (MB): 36984.00 ------------------------------------------------------------ ``` ## 6. Run in ComfyUI # Qwen-Image 2.1 and Turbo Source: https://docs.sglang.io/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1 Run Qwen-Image 2.1 and its eight-step Turbo variant with SGLang Diffusion: text-to-image, reference-image editing, and transparent RGBA output. ## 1. Quick start Use a nightly Docker image containing this integration, or install from source, following the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). Run the commands below inside that environment. Select **Checkpoint weights** for `Qwen/Qwen-Image-2.1` or `Qwen/Qwen-Image-2.1-Turbo`; you can also set a local checkpoint directory under **Variables**. The recipes target NVIDIA CUDA on Linux; the picker marks which workloads have been verified with the full checkpoint. Use **Setup** to select text-to-image, single-image editing, or multi-image editing. **Server** controls placement, attention, encoder scheduling, VAE tiling, graph execution, and request batching. **Request** controls the background, resolution, steps, and output count. Set reference PNG paths under **Variables**; edits upload files from the machine running cURL, so they need not exist on the server. Hardware selection applies the recommended placement for that GPU. H200, B200, RTX PRO 6000 96GB, and DGX Spark keep weights resident; RTX 5090 and RTX 4090 offload selected components to fit the full pipeline. Untested topologies and feature combinations remain selectable and are labeled **Unverified**. Invalid topology combinations disable Copy. This integration currently uses the Python/source command; no published Docker image is verified. Both request modes return base64 PNGs. To save all returned images, append `> response.json` to the request command, then run: ```bash Command theme={null} python - <<'PY' import base64 import json from pathlib import Path for i, item in enumerate(json.loads(Path("response.json").read_text())["data"]): Path(f"output-{i}.png").write_bytes(base64.b64decode(item["b64_json"])) PY ``` ### Recommended hardware settings The picker defaults to native BF16/FP32 precision, exact attention, eager execution, and full-image VAE decoding. Commands omit default values, including one GPU, encoder auto scheduling, and batch size one. Explicit placement and attention overrides preserve each recipe. | GPU | Placement / attention | Generation | Edit | Peak VRAM | | - | - | - | - | - | | H200 141GB | Resident / FlashAttention | 4.48 s | 5.29 s | 38.4 GiB | | B200 192GB | Resident / FlashAttention | 2.46 s | 3.02 s | 38.5 GiB | | RTX PRO 6000 96GB | Resident / Torch SDPA | 8.03 s | 9.63 s | 38.4 GiB | | RTX 4090 24GB | DiT and VAE resident, encoder layerwise offload / FlashAttention | 18.68 s | 21.68 s | 22.7 GiB | | DGX Spark 128GB unified | Resident / Torch SDPA | 35.36 s | 42.23 s | — (unified) | Measured with the original checkpoint on 2026-09-20 at 1024×1024, 40 steps, CFG 1, and one RGBA PNG per request. Times are median HTTP latency after warmup, including PNG serialization and excluding startup; VRAM is the sampled request-phase peak. Prompts and software versions affect both latency and memory use. On one RTX 5090, native-precision, eager text-to-image or single-reference edits with one output and batching off keep the DiT and VAE resident and stream the encoder with Torch SDPA. Multiple reference images, multiple outputs, or request batching select DiT layerwise offload instead; use the updated Server command when changing these settings. The picker identifies which exact combinations are verified. Both RTX 5090 and RTX PRO 6000 use SDPA when FlashAttention is selected in this runtime. CPU offload requires host RAM. ### DGX Spark Select **DGX Spark** for one GB10 GPU on Linux ARM64 with CUDA 13. Use the source installation above. The recommended configuration keeps all components resident, uses native BF16/FP32 precision, and lets the runtime select Torch SDPA: ```bash Command theme={null} sglang serve \ --model-path Qwen/Qwen-Image-2.1 \ --performance-mode speed ``` The [128 GB unified memory](https://docs.nvidia.com/dgx/dgx-spark/hardware.html) is shared by the CPU and GPU. CPU offload is unnecessary for the verified single-image 1024×1024 workload. Keep full-image VAE decoding and eager execution. Generation, editing, transparent generation, and transparent editing were verified with PyTorch 2.13.0+cu130. Spark reports no separate VRAM usage in `nvidia-smi`. This recipe covers one Spark; multi-node deployment and batching remain unverified. ### Batching Keep **Request batching → Off** and **Outputs → 1** for interactive use. Batching increases individual request latency and does not guarantee higher throughput. Measure your workload before enabling it. Cross-request batching merges compatible text-to-image requests. Image edits run separately; **Outputs** controls multiple images within one request. On RTX 4090, selecting multiple outputs or request batching switches to DiT layerwise offload for memory headroom. Restart with the updated **Server** command. Batching preserves native precision but can change floating-point rounding and output pixels, even with the same seed. See [Inference batching](/docs/sglang-diffusion/dynamic_batching) for admission rules and metrics. ### Turbo sampling Select **Qwen-Image 2.1 Turbo** under **Setup → Checkpoint weights** and restart with the generated Server command. Requests use the checkpoint's preset eight-step sigma grid automatically; omit `num_inference_steps`. Setting that field to 8 on the original checkpoint does not reproduce Turbo. For offline generation: ```bash Command theme={null} sglang generate --model-path Qwen/Qwen-Image-2.1-Turbo \ --prompt "A ceramic teapot on a wooden table in soft morning light" ``` The preset comes from `sample_sigmas` in `model_index.json`, with scheduler settings loaded from `scheduler/scheduler_config.json`. Preserve both when copying or exporting the checkpoint. Changing only `num_inference_steps` does not override the preset. Custom grids are experimental and can reduce quality; the picker keeps Turbo on its published schedule. The default CFG scale is 1 for both checkpoints. See the [Turbo model card](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo). Turbo was verified at 1024×1024 with native precision and eager execution on H200: single-GPU generation, editing, transparent generation and editing, multi-image editing, and two outputs; TP×2 generation and editing also passed. The picker distinguishes these from untested Turbo hardware and combinations. The hardware latency table above describes the original 40-step checkpoint. Turbo has the same architecture and weight-memory footprint; fewer denoising steps do not eliminate encoder or VAE costs. Existing quantization and acceleration results for the base checkpoint do not establish Turbo quality or speed. ## 2. Model capabilities Qwen-Image 2.1 supports text-to-image generation, single- and multi-image editing, and RGBA output. Both the original and eight-step Turbo checkpoints support all these modes. Choose Turbo for shorter denoising runs; compare the two checkpoints on your prompts when fidelity is the priority. For multi-round editing, send the previous output as the next reference image. The server does not retain conversation state. Repeated prompts and reference images can reuse exact encoder outputs and VAE posteriors through the default [conditioning cache](/docs/sglang-diffusion/caching-acceleration#conditioning-cache). Changing the prompt or image recomputes their joint embedding; unchanged images can still reuse vision features. Condition-prefix KV reuse stays within each request. Use `--disable-conditioning-cache` to turn off cross-request reuse. ## 3. Checkpoint layout The checkpoint directory must contain `model_index.json` and the `processor`, `text_encoder`, `transformer`, `vae`, and `scheduler` subdirectories. The processor includes the Qwen3-VL tokenizer assets; no separate tokenizer directory is needed. Use `--model-id Qwen-Image-2.1` when your local checkpoint directory has another name. Older Qwen-Image and Qwen-Image-Edit transformer/VAE weights are incompatible. Keep SGLang's installed dependencies. Its native encoder preserves the reference's Transformers 4.57.3 conditioning semantics without requiring a runtime-wide downgrade. ### Transparent PNG output Choose **Transparent / alpha** under Request and describe an isolated subject on a transparent background in the prompt. The picker adds this instruction and selects PNG. `background: "transparent"` alone does not change conditioning or remove the background; JPEG cannot retain alpha. PNG references retain their alpha channel during editing; RGB references receive an opaque alpha channel. The model predicts continuous alpha values, including partly transparent edges, without thresholding or background removal. ## 4. Offline requests Defaults are 1024×1024, CFG 1, and seed 42; output saving is enabled. The original checkpoint defaults to 40 steps; Turbo uses its preset 8-step grid. For GPUs that need offload, also pass the placement flags from the picker. ### Text-to-image ```bash Command theme={null} sglang generate \ --model-path Qwen/Qwen-Image-2.1 \ --prompt "A capybara reading a book by candlelight" ``` ### Image-conditioned editing ```bash Command theme={null} sglang generate \ --model-path Qwen/Qwen-Image-2.1 \ --image-path /path/to/input.png \ --prompt "Move the scene to a snowy mountain at sunrise" ``` Height and width must be positive multiples of 32. Reference images preserve their aspect ratio and are resized to approximately the requested output area; the same resized image feeds the VLM and VAE. Image labels are deterministic (`Picture 1`, `Picture 2`, and so on). Multiple outputs receive independent noise seeds and independent prefix caches. ## 5. Runtime features Both checkpoints use Euler flow matching with CFG disabled by default: 40 steps for the original, or the preset 8-step grid for Turbo. For CFG, provide `--negative-prompt` and `--guidance-scale` greater than one. The API requires a text prompt; precomputed embeddings alone are insufficient. * **Parallelism:** TP, Ulysses, Ring, CFG parallelism, and encoder folding are available in the picker. The target token count, `(height / 16) × (width / 16)`, must be divisible by the SP degree. Ring requires FlashAttention or SageAttention. * **Memory:** use the hardware's recommended placement. **All components layerwise** also streams encoder and VAE blocks, trading transfers for lower device memory. * **VAE:** full-image decoding is the default except on AMD gfx1151 (Strix Halo). Tiling can change pixels near boundaries. With two or more GPUs, **Spatial shard** distributes full-image decoding without enabling tiling; floating-point rounding can still differ. On gfx1151, VAE tiling defaults to enabled to avoid reported full-frame decode hangs at 896px and higher. Explicit CLI, pipeline config, and Python API settings take precedence: `--vae-tiling false` disables tiling but may encounter the hang. Defaults on NVIDIA CUDA and other platforms are unchanged. See the [performance guide](/docs/sglang-diffusion/performance-optimization) for shared runtime options. ### Validation boundaries The picker marks exact tested HTTP deployments, not every combination of the features below. Functional checks do not establish quality for lossy settings. The feature checks below cover the original checkpoint; see [Turbo sampling](#turbo-sampling) for the Turbo validation scope. * **Parallelism and offload:** H200/B200 full-checkpoint checks cover TP, Ulysses, Ring with FlashAttention, CFG parallelism, encoder folding, and all-component layerwise offload. Spatial VAE sharding was checked on two B200 GPUs; floating-point rounding can differ from full-image decoding. * **Disaggregation:** encoder, denoiser, and decoder roles were checked on three B200 GPUs with same-host Mooncake TCP at 512px/4 steps. Multi-host RDMA and multi-rank roles remain unverified for this model. * **Quantized exports:** serialized FP8, native-name Q4\_0 GGUF, and calibrated ModelOpt NVFP4 component exports were checked on B200, including TP2 with encoder folding and separate offload checks. A community ComfyUI-GGUF Q4\_K\_M DiT, which also quantizes the projections outside the blocks, was checked for text-to-image on one RTX 5090. These results do not establish support for arbitrary community exports, other GGUF types, or other hardware. The NVFP4 export used six calibration requests; assess image and alpha quality on your workload. * **LoRA:** synthetic Diffusers-format adapters were checked on one B200 and TP2 with encoder folding. Trained-adapter quality remains unverified. ### Quantization Native precision is the default. Quantization changes image and alpha values; check quality on your own prompts and reference images. Set compatible component paths under **Variables** when choosing an exported format. Adding quantization metadata to native weights does not convert them. For online FP8, use `--component-quantizations.transformer fp8`, `--component-quantizations.text_encoder fp8`, or both. ### Serialized FP8 components Select a **Serialized FP8** option and set the exported component directories. Each directory needs its architecture `config.json`, weights, and quantization metadata. Use `--component-paths.transformer` and/or `--component-paths.text_encoder`; omit online quantization flags. See the [quantization guide](/docs/sglang-diffusion/quantization) for formats. ### GGUF components Select a **GGUF** option and set the `.gguf` files. The picker uses `--component-weights-paths.transformer` and/or `--component-weights-paths.text_encoder`, retaining architecture configs from the base checkpoint. Each file must contain the entire component with native tensor names. GGUF reduces weight storage but does not guarantee lower latency. See the [GGUF guide](/docs/sglang-diffusion/quantization#gguf). ### NVFP4 components NVFP4 requires Blackwell and compatible ModelOpt exports. Select the component directories using `--component-paths.transformer` and/or `--component-paths.text_encoder`. Keep the FlashInfer backend at `auto` on RTX 5090, RTX PRO 6000, and DGX Spark: TensorRT-LLM FP4 GEMM does not support SM12.x. These GPUs remain unverified for this model's NVFP4 exports. See the [NVFP4 guide](/docs/sglang-diffusion/quantization#modelopt-nvfp4). ### LoRA and execution options Use `--lora-path` and `--lora-merge-mode dynamic|merge` or the runtime adapter APIs. Diffusers adapter keys prefixed with `transformer.` map to the native DiT. Keep eager execution as the default. Breakable CUDA Graph replay requires matching resolution and condition-prefix length; unseen shapes run eagerly. Text buckets alone do not guarantee replay. SageAttention and Cache-DiT can change numerical results and require quality checks for your workload. ### Cache-DiT Enable `--enable-cache-dit true` or `SGLANG_CACHE_DIT_ENABLED=true`. 2.1 prefix KV is per layer: each block slices caches by `_layer_id`. Cache-DiT wraps `transformer_blocks` and forwards the same extras to every layer; without that slice, later layers reuse layer 0 and the image collapses to color noise. See the [Cache-DiT guide](/docs/sglang-diffusion/cache_dit). # Qwen-Image-Edit-2511 Source: https://docs.sglang.io/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit Deploy Qwen-Image-Edit-2511 with SGLang - 20B image editing model with text rendering, character consistency, and geometric reasoning. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [Qwen-Image-Edit-2511](https://huggingface.co/Qwen/Qwen-Image-Edit-2511) is the 20B editing counterpart to Qwen-Image. It is strongest at changing text, materials, lighting, viewpoint, or composition while reducing drift in regions that were not requested to change. Choose it for identity-sensitive portrait edits, multi-person composition, typography replacement, and geometry-aware design work. It is substantially heavier than small specialist editors, and consistency is improved rather than guaranteed; evaluate untouched-region drift on the actual editing workload. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration Qwen-Image-Edit-2511 is a 20B parameter model optimized for image editing tasks. The recommended launch configurations vary by hardware. Use the [command builder](#1-quick-start) above. ### 3.2 Configuration Tips See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. * `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. * `--num-gpus`: Number of GPUs to use * `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) * `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs) * `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP * `--ring-degree`: The degree of ring attention-style SP in USP ### 3.3 Decompose an image into layers on H200 `Qwen/Qwen-Image-Layered` returns separate RGBA images. For this model, `--num-frames 4` requests four output layers. The CLI saves all four PNGs, and `DiffGenerator.generate()` returns one result per layer. On Linux with NVIDIA CUDA and two H200 GPUs, you can run the conditional and unconditional branches on separate GPUs: ```bash Command theme={null} CUDA_VISIBLE_DEVICES=0,1 sglang generate \ --model-path Qwen/Qwen-Image-Layered \ --num-gpus 2 \ --cfg-parallel-size 2 \ --tp-size 1 \ --ulysses-degree 1 \ --quality exact \ --enable-torch-compile false \ --warmup-mode request \ --width 640 --height 640 --num-frames 4 \ --num-inference-steps 50 --guidance-scale 4.0 --seed 42 \ --image-path https://raw.githubusercontent.com/QwenLM/Qwen-Image-Layered/main/assets/test_images/4.png \ --prompt "a high quality, cute halloween themed illustration, consistent style and lighting" \ --output-path outputs/qwen-layered \ --save-output ``` For a single H200, set `CUDA_VISIBLE_DEVICES=0`, `--num-gpus 1`, and `--cfg-parallel-size 1`. Both configurations use eager execution. Layered does not currently support breakable CUDA graph; enabling BCG falls back to eager execution. The Layered CFG policy gathers the branch predictions before applying the single-GPU arithmetic order, preserving BF16 rounding and alpha values in the validated fixed-seed example. Each GPU still holds a full DiT replica, so CFG parallelism reduces request latency without reducing the model memory needed on each GPU. ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api). ### 4.1 Edit an Image ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:3000/v1") response = client.images.edit( model="Qwen/Qwen-Image-Edit-2511", image=open("input.png", "rb"), prompt="Change the color of the taxi to black.", n=1, response_format="b64_json", ) # Save the edited image image_bytes = base64.b64decode(response.data[0].b64_json) with open("output.png", "wb") as f: f.write(image_bytes) ``` ### 4.2 Advanced Usage #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit). **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Qwen/Qwen-Image-Edit-2511 ``` **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Qwen/Qwen-Image-Edit-2511 ``` #### 4.2.2 CPU Offload * `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. * `--image-encoder-cpu-offload`: Use CPU offload for image encoder inference. * `--vae-cpu-offload`: Use CPU offload for VAE. * `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". #### 4.2.3 Known LoRA examples Use `--lora-path` at startup or the [LoRA management API](/docs/sglang-diffusion/api/openai_api#lora-management) to load an adapter. Known Qwen-Image-Edit examples include: * [`ostris/qwen_image_edit_inpainting`](https://huggingface.co/ostris/qwen_image_edit_inpainting) * [`lightx2v/Qwen-Image-Edit-2511-Lightning`](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning) ## 5. Benchmark Test Environment: * Hardware: NVIDIA B200 GPU (1x) * Model: Qwen/Qwen-Image-Edit-2511 * sglang diffusion version: 0.5.6.post2 ### 5.1 Speedup Benchmark #### 5.1.1 Edit a image **Server Command**: ```shell Command theme={null} sglang serve --model-path Qwen/Qwen-Image-Edit-2511 --port 30000 ``` **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Model: Qwen/Qwen-Image-Edit-2511 Dataset: vbench Task: image-to-image -------------------------------------------------- Benchmark duration (s): 35.31 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.03 Latency Mean (s): 35.3053 Latency Median (s): 35.3053 Latency P99 (s): 35.3053 -------------------------------------------------- Peak Memory Max (MB): 47959.35 Peak Memory Mean (MB): 47959.35 Peak Memory Median (MB): 47959.35 ============================================================ ``` #### 5.1.2 Edit a image with high concurrency **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-image --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Model: Qwen/Qwen-Image-Edit-2511 Dataset: vbench Task: image-to-image -------------------------------------------------- Benchmark duration (s): 286.11 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 -------------------------------------------------- Request throughput (req/s): 0.07 Latency Mean (s): 150.0428 Latency Median (s): 150.0600 Latency P99 (s): 283.3843 -------------------------------------------------- Peak Memory Max (MB): 47971.82 Peak Memory Mean (MB): 47971.49 Peak Memory Median (MB): 47971.29 ============================================================ ``` ## 6. Run in ComfyUI # SANA-Video Source: https://docs.sglang.io/cookbook/diffusion/SANA-Video/SANA-Video Generate videos with native SANA-Video 2B and SANA-Video 2.0 5B pipelines. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [SANA-Video 2B 480p](https://huggingface.co/Efficient-Large-Model/SANA-Video_2B_480p_diffusers) is a lightweight text-to-video model served through a native SGLang Diffusion pipeline. Its main advantage is deployment cost: at 2B parameters it serves 480p video from a single GPU with a plain `sglang serve` invocation, no parallelism flags required. The released generation profile produces 832×480 output at 81 frames and 16 FPS over 50 inference steps; a compact 17-frame, 8-step profile is covered by server CI for quick validation. Motion strength can be steered directly from the prompt with an optional `motion score: N.` suffix. | Model ID | Task | Default output | | - | - | - | | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers` | Text to video | 832x480, 81 frames at 16 FPS | ## 3. Serve SANA-Video ```bash Command theme={null} sglang serve \ --model-path Efficient-Large-Model/SANA-Video_2B_480p_diffusers \ --port 30010 ``` ## 4. Generate a video The following request uses the compact 17-frame, 8-step profile covered by server CI. Use the model defaults of 81 frames and 50 steps for the released generation profile. ```python Python theme={null} import time from pathlib import Path import requests base_url = "http://127.0.0.1:30010" response = requests.post( f"{base_url}/v1/videos", json={ "model": "Efficient-Large-Model/SANA-Video_2B_480p_diffusers", "prompt": ( "A red tram moves slowly through a sunlit city square while " "pedestrians cross behind it. motion score: 30." ), "size": "832x480", "num_frames": 17, "fps": 16, "num_inference_steps": 8, "guidance_scale": 6.0, "seed": 42, }, timeout=60, ) response.raise_for_status() video_id = response.json()["id"] while True: job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json() if job["status"] == "completed": break if job["status"] == "failed": raise RuntimeError(job.get("error") or "Video generation failed") time.sleep(1) video = requests.get( f"{base_url}/v1/videos/{video_id}/content", timeout=300, ) video.raise_for_status() Path("sana_video.mp4").write_bytes(video.content) ``` ## 5. Request constraints * The default profile uses `832x480`, 81 frames, 50 inference steps, and 16 FPS. * Frame counts are aligned to `4n+1`; for example, a request for 80 frames is adjusted to 77. * Use width and height values divisible by 16. * The prompt supports an optional `motion score: N.` suffix to express the desired amount of motion. ## 6. SANA-Video 2.0 The separate `Efficient-Large-Model/SANA-Video_2.0_5B_720p` checkpoint supports text-to-video and first-frame image conditioning. Its default profile is 1280×736, 193 frames at 24 FPS, 50 steps, and guidance scale 8. This integration loads the official checkpoint directly and uses Gemma 2 and the LTX 2.3 VAE. On a single NVIDIA CUDA GPU, generate a short video with: ```bash Command theme={null} sglang generate \ --model-path Efficient-Large-Model/SANA-Video_2.0_5B_720p \ --prompt "A red tram moves slowly through a sunlit city square." \ --width 832 --height 480 --num-frames 17 --fps 24 \ --num-inference-steps 50 --guidance-scale 8 --seed 42 \ --output-file-path sana_video2.mp4 ``` Add `--image-path first_frame.png` to condition the video on an image. Frame counts align to `8n+1`; width and height must be divisible by 32. The 4-step preview is a different checkpoint and is not supported by this pipeline. For offline weights, keep the model directory name, such as `/path/to/SANA-Video_2.0_5B_720p`, and pass it as `--model-path`. That directory must contain `config.yaml` and `checkpoints/SANA_Video_2.0_5B_720p.pth`. Set `--component-paths.text_encoder /path/to/gemma-2-2b-it` and `--component-paths.vae /path/to/ltx-2.3-vae`. The latter directory may contain the `vae` subdirectory or the VAE files directly. Without these overrides, the component checkpoints are downloaded automatically. # SANA-WM Source: https://docs.sglang.io/cookbook/diffusion/SANA-WM/SANA-WM Deploy SANA-WM with SGLang - a camera-controlled text+image-to-video world model with WASD/IJKL 6-DoF camera control, served three ways: dense bidirectional and chunk-causal batch streaming over /v1/videos (SanaWMTwoStagePipeline), and live over a realtime WebSocket API (SanaWMRealtimePipeline, /v1/realtime_video/generate). ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. For realtime mode, use the [WebSocket client](#7-realtime-websocket-api); the Request tab only checks model metadata in that mode. SGLang-diffusion offers multiple installation methods depending on your hardware platform. Please refer to the [SGLang Diffusion installation guide](../../../docs/sglang-diffusion/installation). SANA-WM adds the `SanaWMTransformer3DModel` + GDN kernels, the `SanaWMTwoStagePipeline` (dense bidirectional + chunk-causal streaming), and the `SanaWMRealtimePipeline` with the `/v1/realtime_video` WebSocket router. Use `sglang serve` to launch the diffusion server. ## 2. Model capabilities [SANA-WM](https://huggingface.co/Efficient-Large-Model/SANA-WM_bidirectional) is NVLabs' 2.6B text-and-image-to-video world model for 720p, minute-scale generation with explicit per-frame 6-DoF camera control. Its hybrid recurrent/softmax attention keeps long causal histories bounded, while an LTX-2 refiner supplies the detail that the fast Stage-1 world model does not produce on its own. Choose the dense checkpoint for the best bounded-clip quality and the streaming checkpoint for long-running or interactive control. Streaming and realtime trade global bidirectional context for bounded state and lower response latency; the realtime WebSocket path is not bit-identical to offline batch streaming. | Serving mode | Checkpoint and API | Best fit | Tradeoff | | - | - | - | - | | Dense bidirectional | `SANA-WM_bidirectional` via `/v1/videos` | Highest single-clip quality | Bounded one-shot generation; full attention cost | | Batch streaming | `SANA-WM_streaming` via `/v1/videos` | Long clips returned as one job | Chunk-causal context rather than global bidirectional context | | Live realtime | `SANA-WM_streaming` via WebSocket | Interactive camera events and incremental output | Stateful session; not bit-identical to batch streaming | **Architecture & components** | Component | Value | | - | - | | Stage-1 DiT | 2.6B; 20 layers, hidden 2240, 20 heads (head\_dim 112); \~10 GB | | Attention | frame-wise Gated DeltaNet + softmax every 4th block (hybrid linear) | | Camera | dual-branch, UCPE + PRoPE (raymap + Plücker), 6-DoF | | VAE | LTX-2 causal, strides (T, H, W) = (8, 32, 32); \~2 GB | | Refiner | LTX-2 Stage-2 distilled; \~41 GB | | Output | up to 720p (704×1280) @ 16 fps, minute-scale | For architecture and training details, see the [SANA-WM paper](https://arxiv.org/abs/2605.15178) and [model card](https://huggingface.co/Efficient-Large-Model/SANA-WM_bidirectional) (Apache-2.0). ## 3. Model Setup Both SANA-WM checkpoints are **public** (Apache-2.0, no gating, no token) and load **directly** — there is no manual assembly step. Pass the HuggingFace repo id to `--model-path` and SGLang downloads, materializes, validates, and loads it: | Mode | `--model-path` | | - | - | | Dense bidirectional (§4) | `Efficient-Large-Model/SANA-WM_bidirectional` | | Batch streaming (§5) / realtime (§6) | `Efficient-Large-Model/SANA-WM_streaming` | Both repo ids are registered in SGLang's **built-in model-overlay registry**, so on first load the overlay transparently materializes the official release into a runnable Diffusers directory — for the streaming checkpoint this converts the DMD self-forcing checkpoint (`sana_dit/model.pt`) into a Diffusers `transformer/` and wires the LTX-2 causal VAE, the LTX-2 refiner, and the Gemma encoders. No environment variable or `build_model_dir.sh` step is needed. (You may also pass a local, already-materialized Diffusers directory.) The materialized checkpoint is a Diffusers directory whose `model_index.json` declares the loadable components: | Component (`model_index.json`) | Class | | - | - | | `transformer` (Stage-1 DiT) | `diffusers.SanaWMTransformer3DModel` | | `vae` | `diffusers.AutoencoderKLCausalLTX2Video` | | `text_encoder` | `transformers.Gemma2Model` | | `tokenizer` | `transformers.GemmaTokenizer` | | `scheduler` | `diffusers.FlowMatchEulerDiscreteScheduler` | How loading works: * The server resolves the checkpoint via `maybe_download_model(model_path, force_diffusers_model=True)` and verifies it contains a `model_index.json` plus the required component subdirectories (`transformer/`, `vae/`). * If `text_encoder` / `tokenizer` are not provided as component paths, the pipeline falls back to the default Stage-1 text encoder **`Efficient-Large-Model/gemma-2-2b-it`** (`DEFAULT_SANA_WM_TEXT_ENCODER`). * **Pick the path with `--pipeline-class-name`.** The checkpoint's `model_index.json` `_class_name` selects the default pipeline (`SanaWMTwoStagePipeline`). Pin it explicitly to choose: `--pipeline-class-name SanaWMTwoStagePipeline` for the `/v1/videos` paths (§4–5) or `--pipeline-class-name SanaWMRealtimePipeline` for live realtime (§6). Pinning is also required if you point `--model-path` at a bare safetensors file instead of a Diffusers directory. * **The Stage-2 LTX-2 refiner** lives under `refiner/` in the checkpoint: `refiner/transformer` (`transformer_2`), `refiner/connectors` (`connectors`), and `refiner/text_encoder` (the Gemma-3 encoder for `text_encoder_2`, whose tokenizer also serves as `tokenizer_2`). The refiner is **optional**: it is skipped (Stage-1-only output) when the env flag `SGLANG_SANA_WM_SKIP_REFINER` (or a `skip_refiner` request extra) is set, or when no `refiner/` is present (`transformer_2` unloaded). On the batch path it runs chunk-wise with `--refiner-chunked` (the official streaming path, default on) or whole-clip without it; on the realtime path the pipeline builds a `SanaWMChunkedRefinerChainStage` only when a refiner is available, and otherwise streams Stage-1 frames. Throughout this cookbook, `` stands for the appropriate SANA-WM repo id from the table above (or a local materialized Diffusers directory). ## 4. Dense bidirectional (offline `/v1/videos`) The **bidirectional** checkpoint generates the whole clip in **one shot** (full bidirectional attention, not chunked) followed by a dense LTX-2 refiner — the highest single-clip quality, matching the NVlabs dense reference. Launch with the two-stage pipeline and **no** `--streaming` flag (dense is the default — `streaming` defaults to `False`): ```bash Command theme={null} sglang serve \ --model-path Efficient-Large-Model/SANA-WM_bidirectional \ --pipeline-class-name SanaWMTwoStagePipeline \ --host 127.0.0.1 --port 30000 ``` Then POST to **`/v1/videos`** exactly as in §5, but pass the NVlabs dense sampling defaults for closest parity — the dense path is denser than the distilled streaming few-step schedule: ```bash Command theme={null} curl -s http://127.0.0.1:30000/v1/videos \ -H 'content-type: application/json' -d '{ "prompt": "a camera moving forward and turning left", "input_reference": "/path/to/first_frame.png", "num_frames": 321, "seed": 42, "fps": 16, "num_inference_steps": 60, "guidance_scale": 5.0, "diffusers_kwargs": { "action": "w-80,wl-80,l-80,wj-80", "intrinsics": "/path/to/intrinsics.npy" } }' ``` * `num_inference_steps` / `guidance_scale` — the dense path uses CFG; NVlabs' reference defaults to **60 steps, guidance 5.0** (the `SanaWMSamplingParams` defaults are the lighter 20 / 4.5 — pass 60 / 5.0 explicitly for dense parity). * The dense refiner drops the leading sink frame, so a `num_frames=321` request yields 320 output frames. ## 5. Batch streaming (offline `/v1/videos`) The **streaming** checkpoint generates a **full camera-controlled clip in one request** — no websocket. This is SGLang's offline streaming path: the whole clip is generated chunk-by-chunk internally, refined, decoded, and returned as one video. Launch with the two-stage pipeline + the streaming flags: ```bash Command theme={null} sglang serve \ --model-path Efficient-Large-Model/SANA-WM_streaming \ --pipeline-class-name SanaWMTwoStagePipeline \ --streaming --refiner-chunked \ --host 127.0.0.1 --port 30000 ``` * `--streaming` — chunk-causal `forward_long` Stage-1 (vs the dense one-shot path of §4). * `--refiner-chunked` — chunk-wise streaming LTX-2 refiner (**on by default**). To use the whole-clip dense refiner instead (also valid, higher peak memory), pass `--refiner-chunked false` — simply omitting the flag keeps the default chunked refiner. * `--num-frame-per-block N` — latent frames per chunk (default `3`). Then POST to **`/v1/videos`** (JSON body shown below; multipart/form-data with an uploaded `input_reference` file also works). Camera control goes in `diffusers_kwargs` — the action-DSL string (§8) and the intrinsics: ```bash Command theme={null} curl -s http://127.0.0.1:30000/v1/videos \ -H 'content-type: application/json' -d '{ "prompt": "a camera moving forward and turning left", "input_reference": "/path/to/first_frame.png", "num_frames": 321, "seed": 42, "fps": 16, "diffusers_kwargs": { "action": "w-80,wl-80,l-80,wj-80", "intrinsics": "/path/to/intrinsics.npy" } }' ``` | Field | Notes | | - | - | | `prompt` | text prompt | | `input_reference` | first-frame image — a server-side path, or (multipart) an uploaded file. For an `http(s)://` URL in a JSON body, use the separate `reference_url` field (the server downloads it and assigns it to `input_reference`) | | `num_frames` | total pixel frames (e.g. `321` → 41 latent frames, 13 chunks; output 704×1280) | | `seed` | RNG seed (default `42`) | | `fps` | output frame rate — **pass `16`** (SANA-WM's native rate). The generic `/v1/videos` default is `24`, which would encode the same frames at 24 fps and make the clip play \~33% shorter (16/24 of the duration) | | `diffusers_kwargs.action` | camera action-DSL string (§8) | | `diffusers_kwargs.intrinsics` | path to a camera-intrinsics `.npy` (per-frame `(T,3,3)`) or an inline 3×3 / `(T,3,3)` list | The response is a `VideoResponse`; fetch the rendered MP4 via the returned reference or `GET /v1/videos/{id}/content`. The streaming hyperparameters (`num_frame_per_block`, `denoising_step_list`, `sink_size`, `num_cached_blocks`, `streaming_cfg_scale`) are **pipeline-config** defaults on `SanaWMPipelineConfig`, not request fields — see §9. ## 6. Launch the Realtime Server Launch with the realtime pipeline **pinned** — the checkpoint defaults to `SanaWMTwoStagePipeline`, so realtime must be selected explicitly (see §3). The `/v1/realtime_video` router is always mounted and becomes functional once the realtime config is active, because `SanaWMRealtimeConfig` has a registered realtime adapter (`SanaWMRealtimeAdapter`). ```bash Command theme={null} sglang serve \ --model-path Efficient-Large-Model/SANA-WM_streaming \ --pipeline-class-name SanaWMRealtimePipeline \ --host 127.0.0.1 --port 30000 ``` Common launch variants: ```bash Command theme={null} # recommended multi-GPU realtime profile sglang serve \ --model-path Efficient-Large-Model/SANA-WM_streaming \ --pipeline-class-name SanaWMRealtimePipeline \ --num-gpus 8 --sp-degree 8 \ --host 127.0.0.1 --port 30000 # single GPU sglang serve \ --model-path Efficient-Large-Model/SANA-WM_streaming \ --pipeline-class-name SanaWMRealtimePipeline \ --num-gpus 1 --host 127.0.0.1 --port 30000 # offload DiT + text encoder to CPU (tight VRAM) sglang serve \ --model-path Efficient-Large-Model/SANA-WM_streaming \ --pipeline-class-name SanaWMRealtimePipeline \ --host 127.0.0.1 --port 30000 \ --dit-cpu-offload --text-encoder-cpu-offload ``` Notes on launch behavior: * **Default endpoint** is `127.0.0.1:30000` (`--host` / `--port` override). * **CPU offload flags are optional.** `--dit-cpu-offload`, `--text-encoder-cpu-offload`, and `--image-encoder-cpu-offload` are available; defaults are auto-adjusted from GPU memory (GPUs under 30 GB get more aggressive offloading). * **Multi-GPU realtime.** Prefer explicit sequence parallelism (`--sp-degree` equal to the number of GPUs for a single session). Do not enable CFG parallel for the realtime profile: the default request uses `guidance_scale=1.0`, so the extra CFG ranks only recompute the same branch. * **FSDP.** Use `--use-fsdp-inference` only when you specifically need weight sharding for memory. For the low-latency realtime profile, prefer keeping components resident and using SP first. * **Warmup.** Server warmup is **automatically skipped** for the realtime pipeline — a synthetic warmup request has no WebSocket session, so the server detects the registered realtime adapter and skips it. No explicit `--warmup-mode` setting is needed. Once up, the realtime WebSocket endpoint lives at `ws://127.0.0.1:30000/v1/realtime_video/generate` (use the Python client in §7 to connect — plain `curl` does not speak the `ws://` upgrade). ## 7. Realtime WebSocket API The realtime API is a single WebSocket at **`/v1/realtime_video/generate`**. All messages — client → server and server → client — are **msgpack** (`msgspec.msgpack.encode` / `decode`), not JSON. The lifecycle is: The client opens the WebSocket and sends exactly one **init** message (`type: "init"`), carrying the prompt, the required `first_frame`, output/sampling options, and optional camera conditions in `condition_inputs`. While generation runs, the client may push **event** messages (`type: "event"`) to steer the camera — either `kind: "camera_actions"` (frame-by-frame lists or state transitions) or `kind: "action"` (an action-DSL string). The server streams **frame batches** back. Each chunk arrives as one or more `frame_batch` messages (header fields + payload bytes); `is_final_frame_batch: true` marks the end of a chunk. The server also emits `chunk_stats` timing messages. ### INIT message `RealtimeVideoGenerationsRequest` (`type` is the literal `"init"`). Key fields: | Field | Type | Notes | | - | - | - | | `type` | `"init"` | Required literal | | `prompt` | str | Text prompt | | `first_frame` | bytes \| str | **Required by the SANA-WM adapter** (`on_init` raises if absent), though the generic request schema defines it as optional. Raw image bytes, a server-side path, or an `http(s)://` URL (downloaded & cached) | | `condition_inputs` | dict | Camera/conditioning inputs (see below) | | `num_frames` | int | Total frames to generate. **Omit it for an open-ended, continuous session** — the adapter leaves `num_frames` unset and flags an open-ended run (`condition_inputs["sana_wm_open_ended"] = True`), generating uniform chunks indefinitely (until `max_chunks` or the client disconnects). Provide an integer for a fixed-length clip | | `seed` | int | RNG seed (default `42`) | | `size` | str | `"WIDTHxHEIGHT"`; realtime requests default to `"832x480"` for latency. Pass `"1280x704"` for the native landscape resolution | | `max_chunks` | int | Optional cap on total chunks generated | | `num_inference_steps` | int | Default `4` for SANA-WM (realtime adapter) | | `guidance_scale` | float | Default `1.0` | | `realtime_output_format` | `"raw"` \| `"webp"` \| `"jpeg"` | Frame encoding for output (see below) | | `realtime_causal_sink_size` | int | Optional override | | `realtime_causal_kv_cache_num_frames` | int | Optional override | `condition_inputs` accepts (all optional; pass **only one** of `action` / `camera_actions`): | Key | Type | Meaning | | - | - | - | | `camera_actions` | `list[list[str]]` or `{mode: "state", transitions: [...]}` | Frame-by-frame camera actions, or state-based transitions | | `action` | str | Action-DSL string, e.g. `"w-10,none-5,a-8"` (see §8) | | `intrinsics_path` | str | Server-side path to a camera-intrinsics **`.npy`** file (loaded via `np.load`; shapes `(4,)`, `(3,3)`, or `(F,3,3)`) | | `intrinsics` | list | Inline intrinsics with shape `(4,)`, `(3,3)`, `(F,4)`, or `(F,3,3)` | If you omit both `intrinsics_path` and `intrinsics`, SGLang uses a centered heuristic intrinsic matrix derived from the first-frame size. Pass explicit intrinsics when you need closer camera parity with a prepared trajectory. ```json INIT (msgpack dict) — open-ended (omit num_frames) theme={null} { "type": "init", "prompt": "beautiful landscape video", "first_frame": "", "size": "832x480", "seed": 42, "max_chunks": 10, "realtime_output_format": "raw", "num_inference_steps": 4, "guidance_scale": 1.0, "condition_inputs": { "camera_actions": [["w"], [], ["a", "s"]], "intrinsics_path": "/path/to/intrinsics.npy" } } ``` ### Live EVENT messages `RealtimeEvent` (`type: "event"`). Use `kind` + `payload` (optional `event_id` correlates the response back to this event). ```json EVENT - camera_actions (frame-by-frame list[list[str]]) theme={null} { "type": "event", "kind": "camera_actions", "event_id": 1, "payload": [["w"], ["w"], ["a"], []] } ``` ```json EVENT - camera_actions (state-based transitions) theme={null} { "type": "event", "kind": "camera_actions", "event_id": 2, "payload": { "mode": "state", "transitions": [ {"actions": ["w"], "client_ts_ms": 1000}, {"actions": ["a", "w"], "client_ts_ms": 1500} ] } } ``` ```json EVENT - action (DSL string) theme={null} { "type": "event", "kind": "action", "event_id": 3, "payload": "w-10,none-5,a-8,d-10" } ``` ### Server frame output The server streams **frame batches**. Every batch arrives as a **single** msgpack message with `type: "frame_batch"` — the header fields below plus an inline `payload` bytes field (the wire `type` is always `"frame_batch"`; there is no separate header-then-bytes message). Header fields: | Field | Meaning | | - | - | | `type` | `"frame_batch"` (always) | | `request_id` | Generation id | | `chunk_index` | Chunk index | | `content_type` | `application/x-raw-rgb`, `application/x-raw-rgb-delta-gzip`, `image/webp`, or `image/jpeg` | | `num_frames` | Frames in this batch | | `total_size` | Payload size in bytes (`len(payload)` — the compressed size for delta-gzip) | | `width`, `height`, `channels` | Frame geometry (`channels: 3`) | | `bytes_per_frame` | Bytes per uncompressed frame (`width*height*3`) | | `format` | `rgb24` for raw | | `encoding` | `raw`, `delta-gzip`, `webp`, or `jpeg` | | `delta_reference` | `previous-frame` (present for delta-gzip) | | `event_id` | Echoes the steering event id; **omitted** from the header for INIT-only chunks | | `frame_batch_index`, `num_frame_batches` | Sequence multiple batches within a chunk | | `is_final_frame_batch` | `true` ends the chunk | ```json Server output - frame_batch (msgpack dict) theme={null} { "type": "frame_batch", "request_id": "uuid-string", "chunk_index": 0, "content_type": "application/x-raw-rgb-delta-gzip", "num_frames": 3, "total_size": 1048576, "width": 1280, "height": 704, "channels": 3, "bytes_per_frame": 2703360, "format": "rgb24", "encoding": "delta-gzip", "delta_reference": "previous-frame", "event_id": 1, "frame_batch_index": 0, "num_frame_batches": 1, "is_final_frame_batch": true, "payload": "" } ``` **Encodings.** `application/x-raw-rgb` is uncompressed RGB24 (3 × uint8, `bytes_per_frame = width*height*3`). `application/x-raw-rgb-delta-gzip` is the zlib-compressed **per-frame XOR delta** against the preceding frame (each frame in the batch is XOR'd against the previous one; sent by default). `realtime_output_format: "raw"` forces uncompressed RGB; `"webp"` / `"jpeg"` send preview-encoded frames. delta-gzip must be restored **frame-by-frame**: decompress the payload, then for each frame XOR it against the already-restored previous frame (the first frame of a batch references the last frame of the previous batch). See `restore_delta_gzip_raw_rgb_payload` in `runtime/realtime/video.py`. The `"raw"` format below avoids this. ### Minimal client example ```python Python theme={null} import msgspec import numpy as np import websockets # pip install websockets WS_URL = "ws://127.0.0.1:30000/v1/realtime_video/generate" async def run(): async with websockets.connect(WS_URL, max_size=None) as ws: # 1) INIT — omit num_frames for an open-ended session; "raw" = uncompressed RGB24 with open("first_frame.png", "rb") as f: first_frame = f.read() await ws.send(msgspec.msgpack.encode({ "type": "init", "prompt": "a camera moving forward and turning right", "first_frame": first_frame, "size": "832x480", "seed": 42, "max_chunks": 10, "realtime_output_format": "raw", "num_inference_steps": 4, "guidance_scale": 1.0, "condition_inputs": { "action": "w-100,wd-50,d-30", "intrinsics_path": "/path/to/intrinsics.npy", # optional; centered heuristic if omitted }, })) # 2) optional: steer mid-stream await ws.send(msgspec.msgpack.encode({ "type": "event", "kind": "camera_actions", "event_id": 1, "payload": [["w"], ["w"], ["a"], []], })) # 3) receive frame batches (raw RGB24) async for message in ws: msg = msgspec.msgpack.decode(message) if msg.get("type") != "frame_batch": continue # skip chunk_stats etc. n, h, w, c = msg["num_frames"], msg["height"], msg["width"], msg["channels"] frames = np.frombuffer(msg["payload"], dtype=np.uint8).reshape(n, h, w, c) # ... display/save frames ... if msg.get("is_final_frame_batch") and msg.get("chunk_index", 0) >= 9: break # asyncio.run(run()) ``` ## 8. Camera Action DSL Camera trajectories are described by a compact string of comma-separated `-` segments, e.g. `"w-100,wd-50,d-30,none-10"`. This is the format accepted by `condition_inputs.action` at init and by `kind: "action"` events. Parsing rules (`parse_action_string`): * Each segment is `-`; `` must be a positive integer. * `none` means no motion for that span: `none-10` = 10 static frames. * Keys are case-insensitive; combined keys apply simultaneously (`wd` = forward + right strafe). Allowed keys are exactly `wasdijkl`. | Key | Motion | | - | - | | `w` / `s` | move forward / backward | | `a` / `d` | strafe left / right | | `i` / `k` | look (pitch) up / down | | `j` / `l` | look (yaw) left / right | Pose generation (`action_string_to_c2w`): * **Translation** (`w`/`s`/`a`/`d`) moves at `translation_speed` (default `0.04` world-units/frame). * **Rotation** (`i`/`k` pitch, `j`/`l` yaw) turns at `rotation_speed_deg` (default `1.2`°/frame); pitch is clamped to ±85°. * **Strafe-yaw coupling** (coefficient `0.4`): a `d` (right) strafe also nudges yaw right and `a` (left) nudges yaw left, so `wd` traces a curving arc rather than a pure sidestep. * Produces `(F+1, 4, 4)` camera-to-world matrices; the realtime stage pads the trajectory to the requested frame count. Example: `"w-100,wd-50,d-30,none-10"` = 100 frames forward → 50 frames forward + sweep right → 30 frames right strafe → 10 frames static. ## 9. Configuration Reference SANA-WM's defaults live in three places: **request-time** sampling params, the **pipeline config** (streaming/refiner knobs), and the **realtime adapter** (init-time overrides). ### Request-time — `SanaWMSamplingParams` (`configs/sample/sana_wm.py`) | Field | Default | Purpose | | - | - | - | | `height` | `704` | Output height | | `width` | `1280` | Output width | | `num_frames` | `49` | Total pixel frames (must satisfy `(num_frames - 1) % 8 == 0`) | | `fps` | `16` | Output frame rate (overrides the base default of 24) | | `num_inference_steps` | `20` | Stage-1 step count | | `guidance_scale` | `4.5` | Dense-path CFG scale | | `negative_prompt` | `""` | Negative prompt | | `camera_to_world` | `None` | In-memory `(T,4,4)` c2w extrinsics (mutually exclusive with `action`) | | `intrinsics` | `None` | In-memory `(T,3,3)` pinhole intrinsics | | `action` | `None` | Action-DSL string (see §8) | | `translation_speed` | `0.04` | World-units/frame for W/S/A/D | | `rotation_speed_deg` | `1.2` | Degrees/frame for I/K/J/L | | `pitch_limit_deg` | `85.0` | Pitch clamp | `generator_device` is inherited from the base `SamplingParams` (default `None` = use the pipeline/model default). On the `/v1/videos` HTTP API the camera fields are passed inside `diffusers_kwargs` (`action` / `intrinsics`, as in §4–5). ### Pipeline config — `SanaWMPipelineConfig` (`configs/pipeline_configs/sana_wm.py`) These are server-launch knobs (set via the `--streaming` / `--refiner-chunked` / `--num-frame-per-block` CLI flags or a pipeline-config override), **not** request fields: | Field | Default | Purpose | | - | - | - | | `streaming` | `False` | Chunk-causal `forward_long` (§5) vs dense one-shot (§4) | | `refiner_chunked` | `True` | Chunk-wise streaming refiner vs whole-clip dense refiner | | `num_frame_per_block` | `3` | Latent frames per Stage-1 / refiner chunk | | `num_cached_blocks` | `2` | Rolling KV-cache history window | | `denoising_step_list` | `(1000, 960, 889, 727, 0)` | 4-step streaming self-forcing timesteps (must end in 0) | | `streaming_cfg_scale` | `1.0` | CFG scale for the distilled streaming path (1.0 = off) | | `sink_size` | `1` | Sink (unrefined context) frames | | `refiner_block_size` | `3` | Refiner block size | | `refiner_kv_max_frames` | `11` | Refiner sliding KV window | ### Realtime adapter init overrides — `SanaWMRealtimeAdapter` At WebSocket `init` the realtime adapter fills SANA-WM defaults that differ from the request/sampling defaults above: | Field | Realtime default | Note | | - | - | - | | `size` | `832x480` | Realtime request default; pass `1280x704` for native landscape output | | `num_frames` | *(unset)* | Omitting → open-ended continuous session (§7) | | `num_inference_steps` | `4` | Distilled few-step | | `guidance_scale` | `1.0` | CFG off | | `fps` | `16` | Native rate | `guidance_scale` applies to the dense path (§4) only; the distilled streaming path uses `streaming_cfg_scale` (default `1.0`, i.e. no CFG) so a `guidance_scale` override never accidentally enables CFG on the streaming stage. `denoising_step_list = (1000, 960, 889, 727, 0)` is the official 4-step streaming schedule (it must end in 0). ## 10. Run in ComfyUI # SenseNova-U1.5-8B-MoT Source: https://docs.sglang.io/cookbook/diffusion/SenseNova/SenseNova-U1.5-8B-MoT Run SenseNova-U1.5-8B-MoT text-to-image generation with SGLang's native multimodal generation runtime. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [SenseNova-U1.5-8B-MoT](https://huggingface.co/sensenova/SenseNova-U1.5-8B-MoT) is a text-to-image model in the SenseNova-U1 family. It is useful for high-resolution poster, infographic, and enterprise visual-generation workloads where prompt following, layout structure, and Chinese text rendering are important. Built on [NEO-Unify](https://huggingface.co/blog/sensenova/neo-unify), this release improves composition, color harmony, material rendering, and local visual detail for native high-resolution generation. It also focuses on clearer Chinese and English text rendering, stronger infographic layout structure, and more reliable execution of complex prompts with object counts, spatial relationships, styles, and multiple constraints. ## 3. Model Deployment ### 3.1 Offline Image Generation SenseNova-U1.5-8B-MoT is a text-to-image model. You can run a single offline generation job with `sglang generate`: ```bash Command theme={null} sglang generate \ --model-path sensenova/SenseNova-U1.5-8B-MoT \ --prompt "A cinematic mountain lake at sunrise, realistic photography." \ --width 2048 \ --height 2048 \ --guidance-scale 4.0 \ --num-inference-steps 50 \ --seed 42 \ --output-file-path output_sensenova_u15.png ``` ### 3.2 Recommended Resolutions SenseNova-U1 was trained around the 2K-pixel resolution tier. The SGLang pipeline only supports positive `--width` and `--height` values that are divisible by 32, and generation quality may degrade for untrained size combinations. Recommended resolution tiers: | Aspect ratio | Resolution | | - | - | | 1:1 | `2048 x 2048` | | 16:9 / 9:16 | `2720 x 1536` / `1536 x 2720` | | 3:2 / 2:3 | `2496 x 1664` / `1664 x 2496` | | 4:3 / 3:4 | `2368 x 1760` / `1760 x 2368` | | 2:1 / 1:2 | `2880 x 1440` / `1440 x 2880` | | 3:1 / 1:3 | `3456 x 1152` / `1152 x 3456` | | 9:16 4K | `2304 x 4096` | ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](../../../docs/sglang-diffusion/api/openai_api). ### 4.1 Generate an Image ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1") response = client.images.generate( model="sensenova/SenseNova-U1.5-8B-MoT", prompt="A clean technology poster with structured layout and clear typography.", n=1, response_format="b64_json", ) image_bytes = base64.b64decode(response.data[0].b64_json) with open("output_sensenova_u15.png", "wb") as f: f.write(image_bytes) ``` ## 5. Advanced usage ### 5.1 Dynamic request batching Start the server with dynamic batching enabled: ```bash Command theme={null} sglang serve \ --model-path sensenova/SenseNova-U1.5-8B-MoT \ --port 30000 \ --batching-max-size 2 \ --batching-delay-ms 100 ``` Submit requests concurrently so they arrive within the batching delay. Requests in one batch must use the same resolution, inference steps, guidance settings, and output options. Requests with multiple outputs or `think_mode=true` are executed sequentially. ### 5.2 Ascend NPU optimizations On Ascend NPU, SenseNova-U1 automatically uses fused inference attention, RMSNorm, and SwiGLU MLP operators for supported inputs. FIA and SwiGLU fall back when their requirements are not met. Run one complete warmup request before benchmarking because the fused MLP packs its gate and up projection weights on first use. ## 6. Benchmark ### 6.1 Speedup benchmark #### 6.1.1 Single-run profile This is a single-run profile for smoke validation rather than a formal benchmark. Environment: NVIDIA A800; workload: vbench text-to-image, 1 prompt, concurrency 1, 1 output per prompt; warmup: none; repetitions: 1; dispersion: not reported; baseline: none; revisions: PR checkout and resolved model snapshot at run time; output validation: request completed and returned one image. **Server Command**: ```shell Command theme={null} sglang serve \ --model-path sensenova/SenseNova-U1.5-8B-MoT \ --port 30000 ``` **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench \ --task text-to-image \ --num-prompts 1 \ --max-concurrency 1 \ --port 30000 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: /workspace/models/ModelScope/SenseNova/SenseNova-U1.5-8B-MoT Dataset: vbench -------------------------------------------------- Benchmark duration (s): 13.10 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 Completed outputs: 1 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.08 Output throughput (outputs/s): 0.08 Latency Mean (s): 13.10 Latency Median (s): 13.10 Latency P90 (s): 13.10 Latency P95 (s): 13.10 Latency P99 (s): 13.10 -------------------------------------------------- Peak Memory Max (MB): 34326.00 Peak Memory Mean (MB): 34326.00 Peak Memory Median (MB): 34326.00 ------------------------------------------------------------ ``` Environment: one Ascend 910C; workload: four concurrent 2048 x 2048 text-to-image requests, 50 denoising steps, CFG 4, BF16. | Batch size | NPU operators | Duration (s) | Throughput (images/s) | Mean latency (s) | Peak reserved memory (MB) | | -: | - | -: | -: | -: | -: | | 1 | Disabled | 234.28 | 0.01707 | 146.47 | 35698 | | 2 | Disabled | 234.83 | 0.01703 | 176.06 | 37870 | | 1 | FIA + RMSNorm + SwiGLU | 205.94 | 0.01942 | 128.80 | 35718 | | 2 | FIA + RMSNorm + SwiGLU | 195.02 | 0.02051 | 146.33 | 37850 | Optimized B1 improves throughput by 13.76% over unoptimized B1. Optimized B2 improves throughput by 20.42% over unoptimized B2 and by 5.60% over optimized B1. # HunyuanImage-3.0-Instruct Source: https://docs.sglang.io/cookbook/diffusion/Tencent/HunyuanImage3 Deploy HunyuanImage-3.0-Instruct for text-to-image generation and reference-image editing. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). The command builder below targets Linux/CUDA; use the [Ascend installation guide](/docs/hardware-platforms/ascend-npus/getting-started/installation) for NPU environments. Exact deployment recipes remain unverified unless explicitly marked otherwise. ```bash theme={null} uv pip install -e 'python[diffusion]' ``` ## 2. Model capabilities [HunyuanImage-3.0-Instruct](https://huggingface.co/tencent/HunyuanImage-3.0-Instruct) combines a text-and-image transformer with diffusion image generation. It supports text-to-image requests and instruction-guided editing with reference images. The native pipeline reuses the transformer's text prefix during denoising and uses a SigLIP2 encoder plus a convolutional VAE for image conditioning. Choose it when one checkpoint must handle both generation and reference-based editing. Its 80B total parameters, despite approximately 13B active per token, make weight residency a major deployment constraint. This recipe covers the Instruct checkpoint, not the separate distilled release. Start with tensor parallelism and the default 50-step, guidance-2.5 sampling profile before testing acceleration options. ## 3. Reference-image editing Use the same server with the images edit endpoint. Repeat `image[]` for multiple reference images. The file paths below are local to the client. ```bash theme={null} curl -sS http://localhost:30010/v1/images/edits \ -F model=tencent/HunyuanImage-3.0-Instruct \ -F 'prompt=Change the background to a quiet seaside town; preserve the subject.' \ -F 'image[]=@reference.png' \ -F num_inference_steps=50 \ -F guidance_scale=2.5 \ -F seed=42 \ -F response_format=b64_json ``` For offline generation, pass the same model and topology to `sglang generate`: ```bash theme={null} sglang generate --model-path tencent/HunyuanImage-3.0-Instruct \ --num-gpus 2 --tp-size 2 --ulysses-degree 1 \ --prompt "Change the background to a quiet seaside town; preserve the subject." \ --image-path reference.png --num-inference-steps 50 --guidance-scale 2.5 \ --seed 42 --save-output ``` ## 4. Request semantics * **Resolution:** `width` and `height` select an aspect ratio by default, not an exact generation canvas. The processor selects a native bucket; decoding crops or pads it. Editing without an explicit size uses the first reference image's dimensions. Use `output_size_mode="exact_size"` when exact output dimensions matter; this resamples the decoded result rather than changing its native bucket. * **Prompting:** `bot_task="image"` and `system_prompt="en_unified"` are the defaults. Other `bot_task` values change the response prefix. They do not run a separate autoregressive reasoning or recaptioning loop. `cot_text` accepts an already-generated reasoning/recaption prefix. * **Multiple outputs:** `n` on the HTTP API or `--num-outputs-per-prompt` offline controls output count. More outputs require additional activation memory. ## 5. Deployment boundaries For an Ascend environment with four visible NPU devices and sufficient aggregate memory, the equivalent TP4 launch is: ```bash theme={null} sglang serve --model-path tencent/HunyuanImage-3.0-Instruct \ --num-gpus 4 --tp-size 4 --ulysses-degree 1 --port 30010 ``` Here `--num-gpus` counts visible accelerator devices, including NPUs. Follow the platform installation guide for compatible PyTorch/CANN packages; do not install the CUDA extra into an NPU environment. Use TP to shard transformer weights; keep Ulysses and Ring at 1. This pipeline does not implement sequence-parallel attention or CFG-parallel execution. The vision encoder and VAE remain replicated, so increasing TP does not divide every component's memory usage. Approximately 160 GB of BF16 transformer weights is only a capacity estimate, not a peak-VRAM measurement: leave room for those components, activations, and loading buffers. Compatible requests can be grouped with `--batching-max-size` and `--batching-delay-ms`. Begin at batch size 1, then validate peak memory at the largest reference-image count and output size you intend to serve. Different prompt/seed values can share a batch; incompatible conditioning and sampling settings cannot. The pipeline has a model-specific Cache-DiT adapter. Caching reuses intermediate results and is not numerically lossless; measure quality and latency on your workload before enabling it. See [Cache-DiT](/docs/sglang-diffusion/cache_dit). SP, CFG parallel, quantization, and full-checkpoint CUDA performance are not claimed as verified by this recipe. # Wan-Animate-2 Source: https://docs.sglang.io/cookbook/diffusion/Wan/Wan-Animate-2 Run Wan-Animate-2 character animation (reference image + reference video) with SGLang Diffusion. ## 1. Quick start Follow the [installation guide](/docs/sglang-diffusion/installation) on Linux with NVIDIA CUDA, then select a verified recipe. Provide both a reference image and a server-readable reference video. See [HTTP invocation](#4-2-serve-and-request-over-http) for polling and downloading the finished video. ## 2. Model capabilities [Wan-Animate-2](https://huggingface.co/Wan-AI/Wan2.2-Animate-2-14B-Diffusers) is a 14B character-animation model in the Wan2.2 family. Instead of the text- or image-only conditioning of the other Wan2.2 checkpoints, it animates a **reference image** so that it follows the motion of a **reference video**, producing a new clip of that subject performing the reference video's motion. Use **`Wan-AI/Wan2.2-Animate-2-14B-Diffusers`** as `--model-path`. The distilled checkpoint (`Wan-AI/Wan2.2-Animate-2-14B-Distilled-Diffusers`, `log_scale=-1.3`, 10 steps, no CFG) is not supported in this release. Unlike the A14B T2V/I2V checkpoints, Wan-Animate-2 is a **single-expert** model: it has one `guidance_scale` (default `3.0`), no low-noise expert (`guidance_scale_2`) and no boundary switching. Diffusers >= 0.40.0 serves Wan-Animate-2 as a modular pipeline (`WanAnimate2ModularPipeline`); SGLang implements the model natively and loads the same `Wan-AI/Wan2.2-Animate-2-14B-Diffusers` checkpoint. There is no diffusers backend fallback. Wan-Animate-2 is served through the native `WanAnimate2Pipeline`, selected automatically from the checkpoint's `model_index.json`. It takes **two** inputs, a reference image and a reference video, so it is invoked differently from the other Wan2.2 models. See [section 4](#4-model-invocation). ### 2.1 Inputs | Input | Field | Passed via | | - | - | - | | Reference image (the subject to animate) | `image_path` / `input_reference` | `--config` YAML for `sglang generate`; the `input_reference` part of a `POST /v1/videos` request | | Reference video (the motion to follow) | `video_path` | `--video-path` or `--config` YAML for `sglang generate`; the `video_path` field in the `/v1/videos` request body or `extra_body` | Both inputs are the base `image_path` / `video_path` sampling fields shared with the other video models; Wan-Animate-2 adds no model-specific input field. Offline runs pass both through a `--config` file (or `--image-path` / `--video-path`), and the online API forwards `video_path` into the pipeline while the reference image rides `input_reference`. Local paths must be readable by the server process. HTTP(S) video URLs use SGLang's shared media loader before decoding, without a disk cache. ### 2.2 Defaults | Parameter | Default | Notes | | - | - | - | | Resolution (`width`×`height`, the API's `size`) | `640`×`800` | A pixel-area budget, not the output dimensions: the frame keeps the reference image's aspect ratio and is the largest 16-aligned canvas with at most `width × height` pixels (see **Output resolution** below). `720`×`1280` is the larger supported budget. | | `clip_len` | `37` | Per-clip denoising chunk length; must be `4k+1` (VAE temporal stride 4), e.g. `37`, `65`, `81`. Other values are rounded to `4k+1` with the same `k` and the rounding is logged (`40` becomes `41`; `38` and `39` become `37`); values below `5` are rejected. | | `fps` | `16` | The reference video is resampled to this rate and the output MP4 is written at it; duration is preserved. | | `num_inference_steps` | `40` | Fewer steps (e.g. `20`) run faster. | | `guidance_scale` | `3.0` | Single expert; there is no `guidance_scale_2`. | | `prompt_ref` | `人物动作的参考视频` ("reference video of the character's motion") | Text for the reference-video branch. The official demo conditions that branch on this fixed description of the reference video rather than on the scene prompt; leave it unchanged unless reproducing a different reference setup. | | `enable_audio` | `true` | Keep the reference video's audio track on the output (see **Audio** below). Per request, like `clip_len`. | **Output length.** The output has as many frames as the (resampled) reference video, not `clip_len`. The video is generated clip by clip with `clip_len` frames per chunk; a larger `clip_len` improves temporal coherence at the cost of VRAM. `clip_len=81` roughly doubles the attention tokens of the default and OOMs a single 80 GB GPU. `num_frames` (and the API's `seconds`, which maps to it) does not control the length: it is ignored and the server logs that it was, so leave it unset. **Output resolution.** `width`×`height` (the API's `size`) is a pixel-area budget, not the output dimensions. The reference image's aspect ratio is kept, the frame is the largest canvas with at most `width × height` pixels whose sides are multiples of 16, and the reference image and the driving-video frames are letterboxed onto it; the output MP4 has the canvas size. At the default `640`×`800` (512,000 pixels) a 9:16 reference gives `528`×`944`, a 1:1 reference gives `704`×`704`, and only a 4:5 reference gives exactly `640`×`800`. To obtain a specific output size, request a budget with the same aspect ratio as the reference image (for a 9:16 image, `576`×`1024` maps onto itself). This differs from the OpenAI Videos API and from the other SGLang diffusion pipelines, where `size` is the literal output resolution. **Audio.** The output MP4 carries the reference video's audio track, as the official pipeline does. Set `enable_audio=false` on a request (a `--config` field offline, a form field or `extra_body` entry online) for a silent output. If the track cannot be decoded, the output is silent and the server logs one warning per process; a reference video without an audio track also gives a silent output. **Lossless conditioning reuse.** Reference-video keys are rotated once and cached per clip, then reused by all denoising steps and both CFG branches. This removes repeated RoPE computation without changing the attention mask, precision, or sampling schedule. The cache uses the existing key storage and is released after the clip; it is not an approximate timestep-skipping cache. Clips containing only padding that would be removed from the final video are not denoised. The padding and conditioning of every retained clip are unchanged. Text encoding uses the native `TextEncodingStage` and its conditioning cache and residency management. CFG-parallel dispatch shares the native CFG utilities while retaining the single-GPU arithmetic order. Each clip uses the shared denoising loop for step profiling, NVTX ranges, progress, and DiT release. Reference K/V preparation and clip-specific prediction remain model hooks; clip decoding still feeds the next clip's conditioning. This does not enable cache-dit or breakable CUDA graphs for this model. Clip decoding uses the shared `DecodingStage.decode_raw` path for VAE compilation, tiling and OOM diagnostics. Raw decoded pixels are kept for the next clip: no extra conversion to and from `[0, 1]` is introduced. The adapter preserves the existing FP32 VAE execution and per-channel latent scaling. QKV projections, cross-attention, FFN and the output head are inherited from the native Wan implementation. Ulysses exchanges use the shared packed-QKV path; the model-specific reference attention mask and K/V lifecycle are retained. Text and image conditioning projections are reused within each clip and CFG branch, rather than recomputed at every timestep. These projected tensors are released after the clip and never reused across requests. Decoded output is copied to CPU and clip-local intermediates are released; only the decoded overlap frames needed by the next clip remain on the GPU. Geometry caches retain at most four RoPE tables and two attention masks, rather than accumulating GPU tensors for every request size. Evicted entries are recomputed without changing the results; frequently alternating among more geometries can add setup work. Measured on one H200, with a 576×1024 budget, 8 steps, CFG 3, seed 42, `clip_len=37`, text-encoder CPU offload, and MP4 plus raw-frame output, the initial reuse changes reduced warm request latency from 42.30s to 41.34s (two requests per variant). With `clip_len=17`, omitting a fully cropped trailing clip reduced 59.60s to 40.45s (one warm request per variant). Raw RGB frames were bit-identical and peak allocated VRAM did not increase. These are workload-specific results, not general speedup claims; both comparisons used PyTorch 2.13.0+cu130, Transformers 5.12.1, and Diffusers 0.37.0. The baseline was the initial native implementation, not the official Diffusers implementation. ## 3. Model Deployment ### 3.1 Recommended configurations Start with **Auto**, or choose **Memory** for more GPU headroom. Memory usage depends on the input video, pixel-area budget, and `clip_len`; see the [hardware comparison](#5-1-hardware-comparison) for measured configurations. For two B300 GPUs, set **GPUs / node** to **2** and **Memory policy** to **CFG parity**; **Auto** CFG parallelism enables the measured two-GPU recipe. The RTX 4090 entry selects its single-GPU memory recipe. Routing to the native Wan-Animate-2 pipeline is automatic from the checkpoint's `model_index.json` (`_class_name: WanAnimate2Pipeline`); pass `--pipeline-class-name WanAnimate2Pipeline` only for a local mirror that lacks `model_index.json`. ### 3.2 Configuration Tips See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. * `--num-gpus {NUM_GPUS}`: Number of GPUs to use. * `--tp-size {TP_SIZE}`: Tensor parallelism size. * `--ulysses-degree {ULYSSES_DEGREE}`: DeepSpeed-Ulysses-style sequence parallelism (USP). The **ring degree must be 1** for this model; `--ring-degree` greater than 1 is rejected with `NotImplementedError`. Combines with `--tp-size`: Ulysses splits the TP-local attention heads, so `--tp-size` must divide the 40 heads and `--ulysses-degree` must divide the per-shard count (see [section 3.3](#3-3-multi-gpu-and-memory-presets)). * `--enable-cfg-parallel`: Split the guided and unguided branches across GPUs. * `--text-encoder-cpu-offload`: CPU-offload the text encoder to save memory; it is moved to the GPU for the one encode per request. ### 3.3 Multi-GPU and memory presets | Target | Recommended server flags | Notes | | - | - | - | | 1 GPU, ample VRAM | *(no extra flags)* | Automatic placement. Start here on H200 or B300. On H200, `speed` used more memory without improving measured latency. | | 1 GPU, memory priority | `--performance-mode memory` | Layerwise offload for supported components. On the measured H200 workload, 31.7 GiB peak versus 68.9 GiB with auto, at nearly the same latency and identical RGB output. | | 1x RTX 4090, 24 GB | `--performance-mode memory` with `PYTORCH_ALLOC_CONF=expandable_segments:True` | Short-video recipe with little spare VRAM. The default allocator OOMed on the measured workload; offload also requires sufficient host RAM. | | 2+ GPUs, long sequences | `--num-gpus N --ulysses-degree N` | Sequence parallel across ranks (ring degree stays 1). | | 2+ GPUs, large DiT | `--num-gpus N --tp-size N` | Tensor parallel across attention heads. With `--tp-size` smaller than `--num-gpus` (or than half of it with `--enable-cfg-parallel`), the leftover GPUs are auto-assigned to Ulysses sequence parallelism (last row). | | 2 GPUs, split CFG | `--num-gpus 2 --enable-cfg-parallel` | Splits the guided/unguided branches. | | 2x B300, recommended | **CFG parity** in the picker: `--num-gpus 2 --enable-cfg-parallel --dit-layerwise-offload --layerwise-offload-components transformer --encoder-parallel replicate --vae-config.use-parallel-decode false` with `SGLANG_DIFFUSION_VAE_CHANNELS_LAST_3D=1` | 54.03 s, 39.53 GiB peak per GPU, and RGB identical to the single-B300 baseline. Only 0.50 s slower than plain CFG, with substantially less GPU memory. | | 4 GPUs | `--num-gpus 4 --enable-cfg-parallel --ulysses-degree 2` | CFG-parallel across two pairs of Ulysses ranks: the fastest 4-GPU layout; pass the flags explicitly to pin the layout instead of relying on the automatic assignment. Ulysses shards the attention sequence, so 4-GPU outputs match the single-GPU video to about 30 dB mean PSNR rather than bit for bit. On 80 GB cards add `--dit-layerwise-offload --layerwise-offload-components transformer` (about 35 GiB per GPU, same speed). | | FSDP sharding | `--use-fsdp-inference` is **not supported** | Rejected at startup, before any weights load; use `--tp-size` to shard the DiT or `--dit-layerwise-offload` to cut peak memory. | | 4+ GPUs, TP x Ulysses | `--num-gpus 4 --tp-size 2 --ulysses-degree 2` | Tensor parallel across attention heads, Ulysses across the TP-local heads: `--tp-size` must divide the 40 heads and `--ulysses-degree` the per-shard count (`40 / tp`). The memory-leaning 4-GPU layout: about 33 GiB peak per GPU (25 GiB torch reserved), the lowest 4-GPU peak after `--tp-size 4`, at about 13 percent slower than CFG-parallel x tensor parallel and 27 percent slower than the default CFG-parallel x Ulysses layout; use the default for speed and this layout for memory. CFG-parallel can be added on top with twice the GPUs (`--num-gpus 8 --enable-cfg-parallel --tp-size 2 --ulysses-degree 2`). | ## 4. Model Invocation ### 4.1 Generate offline with a config file The offline path passes both inputs (and the sampling parameters) through a `--config` YAML: ```yaml wan_animate_2_run.yaml theme={null} prompt: "a person dancing" image_path: /path/to/reference.png video_path: /path/to/reference_video.mp4 clip_len: 37 height: 800 width: 640 num_inference_steps: 40 guidance_scale: 3.0 fps: 16 seed: 42 ``` ```bash Command theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-Animate-2-14B-Diffusers \ --config wan_animate_2_run.yaml \ --num-gpus 1 --text-encoder-cpu-offload --pin-cpu-memory \ --save-output --output-path outputs --output-file-name wan_animate_2.mp4 ``` Any value in the YAML can also be passed as a CLI flag (for example `--image-path`, `--video-path`, `--num-inference-steps`, `--guidance-scale`, `--seed`). ### 4.2 Serve and request over HTTP `POST /v1/videos` creates an asynchronous job. Send the **reference image** as the `input_reference` part and the **reference video** as `video_path`, then poll the job and download the finished MP4. `size` is the pixel-area budget described in [section 2.2](#2-2-defaults); the output dimensions follow the reference image's aspect ratio. ```bash Command theme={null} job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \ --form-string "prompt=a person dancing" \ --form-string "video_path=/path/to/reference_video.mp4" \ --form-string "clip_len=37" \ --form-string "size=640x800" \ --form-string "num_inference_steps=40" \ --form-string "guidance_scale=3.0" \ --form-string "fps=16" \ --form-string "seed=42" \ --form-string "enable_audio=false" \ --form "input_reference=@/path/to/reference.png;type=image/png" \ | python -c 'import json, sys; print(json.load(sys.stdin)["id"])') while true; do status=$(curl -sS "http://127.0.0.1:30010/v1/videos/${job_id}" \ | python -c 'import json, sys; print(json.load(sys.stdin)["status"])') [ "$status" = "completed" ] && break [ "$status" = "failed" ] && exit 1 sleep 1 done curl -sS -L "http://127.0.0.1:30010/v1/videos/${job_id}/content" \ -o wan_animate_2.mp4 ``` These examples produce silent video; use `enable_audio=true` to retain reference audio. With the OpenAI Python client, pass the image as `input_reference` and the server-readable `video_path` and `clip_len` through `extra_body`: ```python Python theme={null} from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://127.0.0.1:30010/v1") video = client.videos.create( model="Wan-AI/Wan2.2-Animate-2-14B-Diffusers", prompt="a person dancing", input_reference=open("/path/to/reference.png", "rb"), extra_body={ "video_path": "/path/to/reference_video.mp4", "clip_len": 37, "guidance_scale": 3.0, "enable_audio": False, # silent output; the reference video's audio is kept by default }, ) ``` For more API usage and request examples, see the [SGLang Diffusion OpenAI API](/docs/sglang-diffusion/api/openai_api) reference. ### 4.3 Reducing memory Use memory mode to stream supported components layer by layer from host RAM: ```bash Command theme={null} sglang serve \ --model-path Wan-AI/Wan2.2-Animate-2-14B-Diffusers \ --performance-mode memory \ --port 30010 ``` For DiT-only offload, choose **DiT layerwise offload** in the picker instead. On RTX 4090, prefix the command with `PYTORCH_ALLOC_CONF=expandable_segments:True` as shown in its hardware preset; memory mode alone OOMed in the tested workload. Lowering `clip_len` also reduces memory, but shortens each clip's temporal context. ## 5. Benchmark ### 5.1 Hardware comparison Native inference at [SGLang `23d0a949a525`](https://github.com/sgl-project/sglang/commit/23d0a949a525006fa309c476f6f824c0644215af), with PyTorch 2.14.1+cu130, Transformers 5.19.0, and Diffusers 0.37.0. The [checkpoint revision](https://huggingface.co/Wan-AI/Wan2.2-Animate-2-14B-Diffusers/tree/7d48412d7b903ff3a89f4f5a960d99e1899605a1) and [reference inputs](https://github.com/sgl-project/ci-data-diffusion/tree/31d8ad69f2434727aad518f0678fbd57823856fe/diffusion-ci/inputs/wan_animate_2) are pinned. The workload uses the [CI prompt](https://github.com/sgl-project/sglang/blob/23d0a949a525006fa309c476f6f824c0644215af/python/sglang/multimodal_gen/test/server/testcase_configs.py#L900-L904) and produces 24 frames at 576x1024: `clip_len=37`, 40 steps, CFG 3, seed 42, and 16 FPS, without audio. E2E is the median of three warm `DiffGenerator` requests, including MP4 saving and raw-frame return; loading and the first request are excluded. Runs use `--warmup-mode off` and `SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1`. The same prompt is repeated with the default conditioning cache enabled. GPU memory is the maximum per-GPU device usage sampled at 10 Hz across loading and all four requests, including allocator reservation. It is **not** a minimum VRAM requirement; offload also uses host RAM. | Hardware | Server flags | Warm E2E (s) | Peak GPU memory (GiB) | | - | - | -: | -: | | 1x RTX 4090 | `--performance-mode memory` with `PYTORCH_ALLOC_CONF=expandable_segments:True` | 634.70 | 23.78 | | 1x H200 | Default (`auto`) | 192.27 | 68.88 | | 1x H200 | `--performance-mode memory` | 193.08 | 31.69 | | 1x H200 | `--dit-layerwise-offload --layerwise-offload-components transformer` | 193.28 | 36.18 | | 1x H200 | `--performance-mode speed` | 192.58 | 90.65 | | 1x B300 | Default (`auto`) | 101.99 | 67.67 | | 1x B300 | `--performance-mode memory` | 104.07 | 30.71 | | 2x B300 | `--num-gpus 2 --enable-cfg-parallel` | 53.52 | 64.93 | | 2x B300 | **CFG parity** preset | 54.03 | 39.53 | | 2x B300 | `--num-gpus 2 --ulysses-degree 2` | 56.17 | 58.42 | | 2x B300 | `--num-gpus 2 --tp-size 2` | 62.97 | 38.88 | RTX 4090 fits this short input only with the expandable allocator; both default `auto` and `memory` without it OOMed. All four successful runs produced identical RGB. This leaves little VRAM headroom and does not validate longer videos. The test host had 1 TiB of RAM; smaller desktop RAM budgets were not validated. On this H200 workload, `memory` reduced the GPU peak by 54% with less than 0.5% additional latency; all 16 raw RGB outputs were identical. Start with `auto` when memory is plentiful, or choose `memory` for headroom. `speed` offered no measured latency benefit here. On B300, **CFG parity** is the recommended two-GPU recipe: 1.89x faster than one GPU with byte-identical RGB on this workload. Plain CFG is slightly faster but uses more memory and changes the output. Memory mode on one B300 also preserves RGB while reducing the GPU peak by 55%, with about 2% additional latency. Comparison against the single-B300 baseline, using all 24 raw RGB frames from the first warm request. SSIM is the per-frame mean; PSNR is computed from the global RGB mean squared error. This checks numerical consistency on one input, not perceptual quality across prompts or parity with the official implementation. | Recipe | Byte-identical | Mean / worst SSIM | Global / worst-frame PSNR (dB) | | - | - | - | - | | Memory; CFG parity | Yes | 1.0000 / 1.0000 | Infinite | | CFG | No | 0.9857 / 0.9574 | 35.36 / 27.65 | | Ulysses | No | 0.9895 / 0.9602 | 37.92 / 27.77 | | TP | No | 0.9896 / 0.9808 | 37.82 / 33.34 | ### 5.2 GB200 reference benchmark This older benchmark uses a longer video and different settings; its numbers are not directly comparable with the hardware comparison above. Test Environment: * Hardware: NVIDIA GB200 GPU (1x, 2x and 4x; 189 GB per GPU) * Model: Wan-AI/Wan2.2-Animate-2-14B-Diffusers * sglang diffusion version: 0.5.22 * Workload: `size` 640x800 (pixel-area budget) with a 16:9 reference image, so the frame is 960x528; a 237-frame reference video at `fps` 30 (7 clips of `clip_len` 37); 40 steps, CFG 3.0, seed 123; `--text-encoder-cpu-offload --warmup-mode request`. E2E is the request time after the warmup request, so it excludes the one-off flex-attention compile. `bench_serving` cannot pass `video_path`, so the benchmark runs offline with `sglang generate --config` (the YAML from [section 4.1](#4-1-generate-offline-with-a-config-file)) and reads timings from `--perf-dump-path`. `SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1` synchronizes the device so the per-stage timings are accurate. **Benchmark Command** (1 GPU shown; add the flags from the table for the other rows): ```shell Command theme={null} SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1 sglang generate \ --model-path Wan-AI/Wan2.2-Animate-2-14B-Diffusers \ --config wan_animate_2_run.yaml \ --num-gpus 1 --text-encoder-cpu-offload --warmup-mode request \ --save-output --perf-dump-path wan_animate_2_perf.json ``` **Result** (peak memory per GPU from `nvidia-smi`): | Config | GPUs | Extra flags | E2E (s) | Peak per GPU (GiB) | | - | -: | - | -: | -: | | 1 GPU, DiT resident | 1 | none | 993 | 69.5 | | 1 GPU, layerwise DiT offload | 1 | `--dit-layerwise-offload --layerwise-offload-components transformer` | 996 | 40.2 | | 2 GPUs, CFG-parallel | 2 | `--enable-cfg-parallel` | 513 | 67.3 | | 2 GPUs, CFG-parallel, replicated encoders, serial VAE decode | 2 | `--enable-cfg-parallel --encoder-parallel replicate --vae-config.use-parallel-decode false` | 515 | 71.0 | | 2 GPUs, CFG-parallel, layerwise DiT offload, bit-identical to 1 GPU | 2 | `--enable-cfg-parallel --dit-layerwise-offload --layerwise-offload-components transformer --encoder-parallel replicate --vae-config.use-parallel-decode false` with `SGLANG_DIFFUSION_VAE_CHANNELS_LAST_3D=1` | 512 | 43.0 | | 2 GPUs, Ulysses | 2 | `--ulysses-degree 2` | 547 | 58.1 | | 2 GPUs, tensor parallel | 2 | `--tp-size 2` | 613 | 36.6 | | 4 GPUs, framework default (CFG2 x SP2, KV-gather exchange) | 4 | none | 283 | 58.4 | | 4 GPUs, CFG-parallel x Ulysses | 4 | `--enable-cfg-parallel --ulysses-degree 2` | 283 | 58.4 | | 4 GPUs, CFG-parallel x Ulysses, layerwise DiT offload | 4 | `--enable-cfg-parallel --ulysses-degree 2 --dit-layerwise-offload --layerwise-offload-components transformer --encoder-parallel replicate --vae-config.use-parallel-decode false` | 291 | 35.2 | | 4 GPUs, Ulysses | 4 | `--ulysses-degree 4` | 317 | 49.0 | | 4 GPUs, CFG-parallel x tensor parallel | 4 | `--enable-cfg-parallel --tp-size 2` | 318 | 37.5 | | 4 GPUs, tensor parallel x Ulysses | 4 | `--tp-size 2 --ulysses-degree 2` | 361 | 33.5 | | 4 GPUs, tensor parallel | 4 | `--tp-size 4` | 379 | 27.3 | Layerwise DiT offload cuts the 1-GPU peak by 29 GiB at the same end-to-end time. The fastest 4-GPU layout is 3.5x faster than one GPU. The framework default and the explicit CFG-parallel x Ulysses layout run at the same speed and produce the same output; the explicit flags pin the layout. Tensor parallel x Ulysses is the memory-leaning 4-GPU layout: the lowest 4-GPU peak after `--tp-size 4`, about 13 percent slower than CFG-parallel x tensor parallel and 27 percent slower than the default. Adding `--encoder-parallel replicate --vae-config.use-parallel-decode false` to the CFG-parallel x Ulysses row costs about 1 percent (287 s, 62.5 GiB) and does not make the output bit-identical to a single GPU: every Ulysses layout matches the single-GPU video to about 30 dB mean PSNR. The layerwise rows are the configurations for 80 GB GPUs. With the replicated encoders, serial VAE decode and the channels-last VAE layout (`SGLANG_DIFFUSION_VAE_CHANNELS_LAST_3D=1`) the 2-GPU CFG-parallel output is bit-identical to a single GPU. ### 5.3 Native regression coverage A separate regression used two B300 GPUs, CFG-parallel with replicated encoders and serial VAE decode, a 576x1024 area budget, 40 steps, `clip_len=37`, CFG=3, and seed 42. The driving input was looped to 64 frames to exercise two clips; this is a bounded regression workload, not a long-video quality benchmark. Both revisions (`4ee69b4cf9ea` and `04725c1c2900`) used PyTorch 2.14.1+cu130, Transformers 5.19.0, Diffusers 0.37.0, and FlashAttention 4.0.0b34 on the same node. Cold and warm raw RGB outputs were byte-identical to the pre-optimization native implementation, both with resident DiT weights and with layerwise DiT offload. The single warm samples measured 112.29 s before and 112.40 s after optimization; offload measured 113.46 s. This does not establish a significant latency change. Rank 0's request-stage peak allocated memory was 52.24, 52.23, and 23.93 GiB, respectively; these are not loading peaks or whole-device physical memory peaks. An H200 HTTP smoke test also covered image upload, asynchronous job polling and download, request warmup, repeated requests, URL video input, and reference-audio passthrough. Repeated decoded RGB outputs were identical. These checks preserve the existing native output, not bitwise parity with the newer Diffusers modular pipeline. The native sampler follows the original Wan sigma grid starting at exactly 1.0 and retains its BF16 conditioning conversion; the modular pipeline's sampling and preprocessing differ. Official-reference accuracy must be validated separately with pinned versions and aligned settings. ## 6. Troubleshooting * **OOM**: lower `clip_len` (for example `37` instead of `65`/`81`) or add `--dit-layerwise-offload --layerwise-offload-components transformer`. * **`--ring-degree` greater than 1** is rejected at startup with `NotImplementedError`. Use `--ulysses-degree` for sequence parallelism and keep the ring degree at 1. * **`--ulysses-degree` that does not divide the TP-local attention heads** (40 / `--tp-size`; with TP 1, degrees 3, 6, 16, ... are rejected) fails at startup with `ValueError`; with TP 1 use 2, 4, 5, 8, 10, 20 or 40. The same check covers leftover GPUs auto-assigned to sequence parallelism (for example `--num-gpus 8 --tp-size 2` runs TP 2 x Ulysses 4). * **FSDP** (`--use-fsdp-inference`) is not supported for this model and is rejected at startup; use `--tp-size` or DiT layerwise offload for memory. * **`--attention-backend`** (or `--component-attention-backends transformer=...`) is rejected at startup with `ValueError`: the in-context self-attention runs on torch `flex_attention` only. The text encoder still accepts `--component-attention-backends text_encoder=`. * **`enable_teacache` / `enable_spectrum`** request fields are rejected with `ValueError`: every block runs at every step, so the cache heuristics do not apply. * **`SGLANG_DIFFUSION_ENABLE_MXFP8_ATTENTION`** is rejected at startup with `ValueError`: this model does not apply the offline Q/K rotation. * **Silent output**: the reference video's audio track is kept only when it can be decoded; otherwise the output is silent and the server logs one warning per process naming the cause. A reference video without an audio track also gives a silent output (one info line). Pass `enable_audio=false` to skip audio extraction altogether. * **Reference video not found**: `video_path` must be a path the server process can read. Pass it as `--video-path` or through `--config` offline, or in the request body online. # Wan2.1 Source: https://docs.sglang.io/cookbook/diffusion/Wan/Wan2.1 Deploy Wan2.1 video generation models with SGLang - community contribution guide for Wan Video's diffusion models. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [Wan2.1](https://github.com/Wan-Video/Wan2.1) is a broad open video family covering text-to-video and image-to-video across 1.3B and 14B checkpoints. Its practical strengths are motion-rich generation, temporal consistency, and readable Chinese/English text, with 480p and 720p variants for different quality and memory targets. Choose the 1.3B T2V model for consumer-GPU experiments and the 14B models when quality matters more than footprint. Wan2.1 is a dense DiT family; for timestep-specialized MoE capacity or the unified 5B TI2V path, use Wan2.2 instead. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The Wan2.1 series offers models in multiple sizes and resolutions. SGLang supports Wan2.1 deployment on NVIDIA B200, B300, H200, H100, and AMD MI300X, MI325X, MI355X GPUs and Ascend A2/A3 Series NPUs. The recommended launch configurations vary by hardware, model size, and memory headroom. Use the [command builder](#1-quick-start) above. ### 3.2 Configuration Tips Current supported optimization options are listed in the [SGLang diffusion support matrix](../../../docs/sglang-diffusion/attention_backends#platform-support-matrix). * `--vae-path`: Path to a custom VAE model or HuggingFace model ID. If not specified, the VAE will be loaded from the main model path. * `--num-gpus {NUM_GPUS}`: Number of GPUs to use. * `--tp-size {TP_SIZE}`: Tensor parallelism size (for the encoder/DiT; keep (\leq 1) if relying heavily on CPU offload). * `--sp-degree {SP_SIZE}`: Sequence parallelism degree. * `--ulysses-degree {ULYSSES_DEGREE}`: Degree of DeepSpeed-Ulysses-style SP in USP. * `--ring-degree {RING_DEGREE}`: Degree of ring attention-style SP in USP. * `--text-encoder-cpu-offload`, `--dit-cpu-offload`, `--vae-cpu-offload`: Use CPU offload to reduce peak GPU memory when needed. ### 3.3 24 GB NVIDIA GPUs The 14B DiTs do not fit on a 24 GB card in BF16 (the I2V-14B one is 30.5 GiB), so their weights have to stream from host memory layer by layer. On CUDA, `--performance-mode memory` streams the DiT, text encoder, image encoder and VAE: ```bash theme={null} # 1 GPU sglang serve --model-path Wan-AI/Wan2.1-I2V-14B-480P-Diffusers --performance-mode memory # 8 GPUs: CFG parallel x Ulysses 4 sglang serve --model-path Wan-AI/Wan2.1-I2V-14B-480P-Diffusers --num-gpus 8 \ --enable-cfg-parallel --ulysses-degree 4 --performance-mode memory ``` Each GPU's worker keeps its own copy of the streamed weights in host memory, about 60 GiB, so 8 GPUs need close to 500 GiB of RAM. Measured with SGLang main `c7be3e9` (2026-09-29) on RTX 4090 24 GB (PCIe, no P2P between GPUs), for Wan2.1-I2V-14B-480P image-to-video at 480x832, 81 frames, 40 steps, BF16: | GPUs | Time per video | Time per step | Peak GPU memory | Host memory | | - | -: | -: | -: | -: | | 1 | 840 s | 20.7 s | 12.5 GiB | 59 GiB | | 8 | 198 s | 4.8 s | 11.9 GiB | 483 GiB | Component CPU offload (`--dit-cpu-offload`) does not help here: it moves the whole DiT onto the GPU around each use. ## 4. Model Invocation ### 4.1 Basic Usage For more API usage and request examples, please refer to: [SGLang Diffusion OpenAI API](../../../docs/sglang-diffusion/api/openai_api) #### 4.1.1 Launch a server and then send requests ```bash Command theme={null} sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers --port 30000 curl http://127.0.0.1:30000/v1/images/generations \ -o >(jq -r '.data[0].b64_json' | base64 --decode > example.png) \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "Wan-AI/Wan2.1-T2V-14B-Diffusers", "prompt": "A cute baby sea otter", "n": 1, "size": "1024x1024", "response_format": "b64_json" }' ``` #### 4.1.2 Generate a video without launching a server ```bash Command theme={null} SERVER_ARGS=( --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers --text-encoder-cpu-offload --pin-cpu-memory --num-gpus 4 --ulysses-degree=2 --enable-cfg-parallel ) SAMPLING_ARGS=( --prompt "A curious raccoon" --save-output --output-path outputs --output-file-name "A curious raccoon.mp4" ) sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}" ``` ### 4.2 Advanced Usage #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve significant inference speedups with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](../../../docs/sglang-diffusion/cache_dit). **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers ``` **Advanced Usage** Combined Configuration Example: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers ``` #### 4.2.2 GPU Optimization * `--dit-cpu-offload`: Keep DiT weights on the CPU and move them onto the GPU whole around each use. This takes the DiT out of FSDP, which shards only resident components. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. * `--image-encoder-cpu-offload`: Use CPU offload for image encoder inference. * `--vae-cpu-offload`: Use CPU offload for VAE. * `--pin-cpu-memory`: Pin memory for CPU offload. Use as a workaround if you see "CUDA error: invalid argument". #### 4.2.3 Supported LoRA Registry SGLang supports applying Wan2.1 LoRA adapters on top of base models:
origin model supported LoRA
[Wan-AI/Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B) [NIVEDAN/wan2.1-lora](https://huggingface.co/NIVEDAN/wan2.1-lora)
[Wan-AI/Wan2.1-I2V-14B-720P](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) [valiantcat/Wan2.1-Fight-LoRA](https://huggingface.co/valiantcat/Wan2.1-Fight-LoRA)
**Example**: ```bash Command theme={null} sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers --port 30000 \ --lora-path NIVEDAN/wan2.1-lora ``` ## 5. Reference Benchmark The following benchmark is a point-in-time reference for one model, hardware platform, SGLang image, and parameter set. It is not a complete hardware support matrix. Test Environment: * Hardware: AMD MI300X GPU (1x) * Model: Wan-AI/Wan2.1-T2V-14B-Diffusers * SGLang Docker Image Version: 0.5.9 ### 5.1 How to Run Benchmarks with SGLang You can use the built-in SGLang diffusion benchmark script to evaluate Wan2.1 performance on your hardware. #### 5.1.1 Generate a single video **Server Command**: ```bash Command theme={null} sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers ``` **Benchmark Command**: ```bash Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-video Model: Wan-AI/Wan2.1-T2V-14B-Diffusers Dataset: vbench -------------------------------------------------- Benchmark duration (s): 1958.41 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.00 Latency Mean (s): 1958.4059 Latency Median (s): 1958.4059 Latency P99 (s): 1958.4059 -------------------------------------------------- Peak Memory Max (MB): 59662.00 Peak Memory Mean (MB): 59662.00 Peak Memory Median (MB): 59662.00 ============================================================ ``` **Server Command**: ```bash Command theme={null} #One A3 Series card has 2 npu chips. Benchmark was done with two A3 Series cards sglang serve \ --model-path /models/Wan-AI/Wan2.1-T2V-14B-Diffusers/ \ --tp-size 2 \ --sp-degree 2 \ --num-gpus 4 \ --attention-backend laser_attn ``` **Benchmark Command**: ```bash Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench \ --task text-to-video \ --num-prompts 1 \ --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-video Model: Wan-AI/Wan2.1-T2V-14B-Diffusers/ Dataset: vbench -------------------------------------------------- Benchmark duration (s): 1282.90 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 Completed outputs: 1 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.00 Output throughput (outputs/s): 0.00 Latency Mean (s): 1282.90 Latency Median (s): 1282.90 Latency P90 (s): 1282.90 Latency P95 (s): 1282.90 Latency P99 (s): 1282.90 -------------------------------------------------- Peak Memory Max (MB): 31938.00 Peak Memory Mean (MB): 31938.00 Peak Memory Median (MB): 31938.00 ============================================================ ``` #### 5.1.2 Generate videos with Cache-DiT acceleration **Server Command**: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers ``` **Benchmark Command**: ```bash Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-video Model: Wan-AI/Wan2.1-T2V-14B-Diffusers Dataset: vbench -------------------------------------------------- Benchmark duration (s): 556.99 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.00 Latency Mean (s): 556.9885 Latency Median (s): 556.9885 Latency P99 (s): 556.9885 -------------------------------------------------- Peak Memory Max (MB): 69306.00 Peak Memory Mean (MB): 69306.00 Peak Memory Median (MB): 69306.00 ============================================================ ``` **Server Command**: ```bash Command theme={null} #One A3 Series card has 2 npu chips. Benchmark was done with two A3 Series cards SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ SGLANG_CACHE_DIT_ENABLED=true sglang serve \ --model-path /models/Wan-AI/Wan2.1-T2V-14B-Diffusers/ \ --tp-size 2 \ --sp-degree 2 \ --num-gpus 4 \ --attention-backend laser_attn ``` **Benchmark Command**: ```bash Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench \ --task text-to-video \ --num-prompts 1 \ --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-video Model: Wan-AI/Wan2.1-T2V-14B-Diffusers/ Dataset: vbench -------------------------------------------------- Benchmark duration (s): 413.88 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 Completed outputs: 1 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.00 Output throughput (outputs/s): 0.00 Latency Mean (s): 413.88 Latency Median (s): 413.88 Latency P90 (s): 413.88 Latency P95 (s): 413.88 Latency P99 (s): 413.88 -------------------------------------------------- Peak Memory Max (MB): 32782.00 Peak Memory Mean (MB): 32782.00 Peak Memory Median (MB): 32782.00 ============================================================ ``` ## 6. Run in ComfyUI # Wan2.2 Source: https://docs.sglang.io/cookbook/diffusion/Wan/Wan2.2 Deploy Wan2.2 video generation models with SGLang - MoE architecture, cinematic aesthetics, and efficient 720P@24fps generation. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [Wan2.2](https://github.com/Wan-Video/Wan2.2) extends the Wan family with timestep-specialized Mixture-of-Experts models and a unified 5B TI2V checkpoint. It targets cinematic composition, lighting, and complex motion while supporting text-to-video, image-to-video, and hybrid text/image conditioning. Choose the A14B MoE checkpoints for maximum T2V or I2V capacity and the 5B TI2V model for a smaller unified 720p-at-24-fps path. MoE reduces active compute relative to total capacity but does not remove the memory cost of loading expert weights, so hardware selection still matters. The Wan2.2 TI2V 5B checkpoint currently has known quality issues when it is used for image-to-video generation. Use `Wan-AI/Wan2.2-I2V-A14B-Diffusers` when I2V quality is the priority. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The Wan2.2 series offers models in various sizes, architectures and input types, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size. Use the [command builder](#1-quick-start) above. ### 3.2 Configuration Tips See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. * `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. * `--num-gpus {NUM_GPUS}`: Number of GPUs to use * `--tp-size {TP_SIZE}`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) * `--sp-degree {SP_SIZE}`: Sequence parallelism size (typically should match the number of GPUs) * `--ulysses-degree {ULYSSES_DEGREE}`: The degree of DeepSpeed-Ulysses-style SP in USP * `--ring-degree {RING_DEGREE}`: The degree of ring attention-style SP in USP ## 4. Model Invocation ### 4.1 Basic Usage For more API usage and request examples, please refer to: [SGLang Diffusion OpenAI API](/docs/sglang-diffusion/api/openai_api) #### 4.1.1 Launch a server and then send requests ```shell Command theme={null} sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers --port 3000 curl http://127.0.0.1:3000/v1/images/generations \ -o >(jq -r '.data[0].b64_json' | base64 --decode > example.png) \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "Wan-AI/Wan2.2-T2V-A14B-Diffusers", "prompt": "A cute baby sea otter", "n": 1, "size": "1024x1024", "response_format": "b64_json" }' ``` #### 4.1.2 Generate a video without launching a server ```shell Command theme={null} SERVER_ARGS=( --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers --text-encoder-cpu-offload --pin-cpu-memory --num-gpus 4 --ulysses-degree=2 --enable-cfg-parallel ) SAMPLING_ARGS=( --prompt "A curious raccoon" --save-output --output-path outputs --output-file-name "A curious raccoon.mp4" ) sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}" ``` ### 4.2 Advanced Usage #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit). **Basic Usage** ```shell Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers ``` **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example: ```shell Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers ``` #### 4.2.2 GPU Optimization * `--dit-cpu-offload`: Keep DiT weights on the CPU and move them onto the GPU whole around each use. This takes the DiT out of FSDP, which shards only resident components. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. Enable if run out of memory with FSDP. * `--image-encoder-cpu-offload`: Use CPU offload for image encoder inference. Enable if run out of memory with FSDP. * `--vae-cpu-offload`: Use CPU offload for VAE. Enable if run out of memory. * `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". #### 4.2.3 Supported LoRA Registry
origin model supported LoRA
[Wan-AI/Wan2.2-I2V-A14B-Diffusers](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B-Diffusers) [lightx2v/Wan2.2-Distill-Loras](https://huggingface.co/lightx2v/Wan2.2-Distill-Loras)
[Wan-AI/Wan2.2-T2V-A14B-Diffusers](https://huggingface.co/Wan-AI/Wan2.2-T2V-A14B-Diffusers) [Cseti/wan2.2-14B-Arcane\_Jinx-lora-v1](https://huggingface.co/Cseti/wan2.2-14B-Arcane_Jinx-lora-v1)
**Example**: ```shell Command theme={null} sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers --port 3000 \ --lora-path Cseti/wan2.2-14B-Arcane_Jinx-lora-v1 ``` ## 5. Benchmark Test Environment: * Hardware: NVIDIA B200 GPU (1x) * Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers * sglang diffusion version: 0.5.6.post2 ### 5.1 Speedup Benchmark ### 5.1.1 Generate a video **Server Command**: ```shell Command theme={null} sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers ``` **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers Dataset: vbench Task: text-to-video -------------------------------------------------- Benchmark duration (s): 630.43 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.00 Latency Mean (s): 630.4277 Latency Median (s): 630.4277 Latency P99 (s): 630.4277 -------------------------------------------------- Peak Memory Max (MB): 62627.41 Peak Memory Mean (MB): 62627.41 Peak Memory Median (MB): 62627.41 ============================================================ ``` **Server Command**: ```shell Command theme={null} #One A3 Series card has 2 npu chips. Using four A3 Series cards in benchmarking sglang serve \ --model-path /models/Wan-AI/Wan2.2-T2V-A14B-Diffusers/ \ --tp-size 2 \ --sp-degree 4 \ --num-gpus 8 \ --attention-backend laser_attn ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench \ --task text-to-video \ --num-prompts 1 \ --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-video Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers/ Dataset: vbench -------------------------------------------------- Benchmark duration (s): 214.50 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 Completed outputs: 1 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.00 Output throughput (outputs/s): 0.00 Latency Mean (s): 214.50 Latency Median (s): 214.50 Latency P90 (s): 214.50 Latency P95 (s): 214.50 Latency P99 (s): 214.50 -------------------------------------------------- Peak Memory Max (MB): 46692.00 Peak Memory Mean (MB): 46692.00 Peak Memory Median (MB): 46692.00 ------------------------------------------------------------ ``` #### 5.1.2 Generate videos with high concurrency **Server Command**: ```shell Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers ``` **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-video --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers Dataset: vbench Task: text-to-video -------------------------------------------------- Benchmark duration (s): 5163.21 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 -------------------------------------------------- Request throughput (req/s): 0.00 Latency Mean (s): 2739.7695 Latency Median (s): 2742.0673 Latency P99 (s): 5121.6331 -------------------------------------------------- Peak Memory Max (MB): 72523.56 Peak Memory Mean (MB): 70253.34 Peak Memory Median (MB): 70824.46 ============================================================ ``` **Server Command**: ```shell Command theme={null} #One A3 Series card has 2 npu chips. Using four A3 Series cards in benchmarking SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ SGLANG_CACHE_DIT_ENABLED=true sglang serve \ --model-path /models/Wan-AI/Wan2.2-T2V-A14B-Diffusers/ \ --tp-size 2 \ --sp-degree 4 \ --num-gpus 8 \ --attention-backend laser_attn ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench \ --task text-to-video \ --num-prompts 20 \ --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-video Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers/ Dataset: vbench -------------------------------------------------- Benchmark duration (s): 4384.65 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 Completed outputs: 20 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.00 Output throughput (outputs/s): 0.00 Latency Mean (s): 2304.17 Latency Median (s): 2297.69 Latency P90 (s): 3972.32 Latency P95 (s): 4178.99 Latency P99 (s): 4343.52 -------------------------------------------------- Peak Memory Max (MB): 46692.00 Peak Memory Mean (MB): 46691.90 Peak Memory Median (MB): 46692.00 ------------------------------------------------------------ ``` ## 6. Run in ComfyUI # Z-Image-Turbo Source: https://docs.sglang.io/cookbook/diffusion/Z-Image/Z-Image-Turbo Deploy Z-Image-Turbo with SGLang - community contribution guide for Z-Image's fast image generation model. ## 1. Quick start Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform. ## 2. Model capabilities [Z-Image-Turbo](https://huggingface.co/Tongyi-MAI/Z-Image-Turbo) is a distilled 6B single-stream DiT for fast text-to-image generation. It reaches its intended operating point in 8 function evaluations and is particularly strong at photorealistic scenes, prompt adherence, and English/Chinese text rendering. Choose it when latency and a relatively small deployment footprint matter more than the editability or maximum capacity of larger image models. It is a generation-only checkpoint; use Qwen-Image-Edit or FLUX.2 when the request includes source images or identity-preserving edits. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration Z-Image-Turbo is optimized for high-quality image generation with only 8 inference steps. The recommended launch configurations vary by hardware. Use the [command builder](#1-quick-start) above. ### 3.2 Configuration Tips See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. * `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. * `--num-gpus`: Number of GPUs to use * `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) * `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs) * `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP * `--ring-degree`: The degree of ring attention-style SP in USP **AMD ROCm Notes**: Requires SGLang >= v0.5.8. ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api). ### 4.1 Generate an Image ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1") response = client.images.generate( model="Tongyi-MAI/Z-Image-Turbo", prompt="A logo With Bold Large text: SGL Diffusion", n=1, response_format="b64_json", ) # Save the generated image image_bytes = base64.b64decode(response.data[0].b64_json) with open("output.png", "wb") as f: f.write(image_bytes) ``` ### 4.2 Advanced Usage Z-Image uses `positive + guidance_scale * (positive - negative)` guidance, unlike the standard CFG convention. Any positive `guidance_scale` enables CFG; omitted negative prompts use an empty string. Z-Image-Turbo defaults to `guidance_scale=0` and does not run the negative branch. Enabling CFG adds a second DiT branch, so expect additional inference cost; it is not automatically recommended for Turbo. #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit). **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Tongyi-MAI/Z-Image-Turbo ``` **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Tongyi-MAI/Z-Image-Turbo ``` #### 4.2.2 CPU Offload * `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. * `--vae-cpu-offload`: Use CPU offload for VAE. * `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". #### 4.2.3 Known LoRA examples Use `--lora-path` at startup or the [LoRA management API](/docs/sglang-diffusion/api/openai_api#lora-management) to load an adapter. Known Z-Image-Turbo examples include: * [`tarn59/pixel_art_style_lora_z_image_turbo`](https://huggingface.co/tarn59/pixel_art_style_lora_z_image_turbo) * [`wcde/Z-Image-Turbo-DeJPEG-Lora`](https://huggingface.co/wcde/Z-Image-Turbo-DeJPEG-Lora) ## 5. Benchmark Test Environment: * Hardware: AMD Instinct MI300X GPU (1x) * Model: Tongyi-MAI/Z-Image-Turbo * Docker Image: lmsysorg/sglang:v0.5.8-rocm700-mi30x * sglang diffusion version: 0.5.8 ### 5.1 Speedup Benchmark #### 5.1.1 Generate an image **Server Command**: ```shell Command theme={null} sglang serve --model-path Tongyi-MAI/Z-Image-Turbo \ --ulysses-degree=1 --ring-degree=1 --port 30000 ``` **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Tongyi-MAI/Z-Image-Turbo Dataset: vbench -------------------------------------------------- Benchmark duration (s): 1.84 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.54 Latency Mean (s): 1.8435 Latency Median (s): 1.8435 Latency P99 (s): 1.8435 -------------------------------------------------- Peak Memory Max (MB): 30689.20 Peak Memory Mean (MB): 30689.20 Peak Memory Median (MB): 30689.20 ============================================================ ``` **Server Command**: ```shell Command theme={null} #One A3 Series card has 2 npu chips sglang serve --model-path Tongyi-MAI/Z-Image-Turbo --tp-size 2 --sp-degree 1 --num-gpus 2 ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Tongyi-MAI/Z-Image-Turbo Dataset: vbench -------------------------------------------------- Benchmark duration (s): 2.43 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 Completed outputs: 1 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.41 Output throughput (outputs/s): 0.41 Latency Mean (s): 2.43 Latency Median (s): 2.43 Latency P90 (s): 2.43 Latency P95 (s): 2.43 Latency P99 (s): 2.43 -------------------------------------------------- Peak Memory Max (MB): 11052.00 Peak Memory Mean (MB): 11052.00 Peak Memory Median (MB): 11052.00 ------------------------------------------------------------ ``` #### 5.1.2 Generate images with high concurrency **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Tongyi-MAI/Z-Image-Turbo Dataset: vbench -------------------------------------------------- Benchmark duration (s): 35.32 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 -------------------------------------------------- Request throughput (req/s): 0.57 Latency Mean (s): 18.5672 Latency Median (s): 18.5573 Latency P99 (s): 34.9880 -------------------------------------------------- Peak Memory Max (MB): 30689.26 Peak Memory Mean (MB): 30689.21 Peak Memory Median (MB): 30689.21 ============================================================ ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: /models/Tongyi-MAI/Z-Image-Turbo/Z-Image-Turbo Dataset: vbench -------------------------------------------------- Benchmark duration (s): 49.08 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 Completed outputs: 20 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.41 Output throughput (outputs/s): 0.41 Latency Mean (s): 25.78 Latency Median (s): 25.77 Latency P90 (s): 44.42 Latency P95 (s): 46.75 Latency P99 (s): 48.61 -------------------------------------------------- Peak Memory Max (MB): 11054.00 Peak Memory Mean (MB): 11054.00 Peak Memory Median (MB): 11054.00 ------------------------------------------------------------ ``` ## 6. Run in ComfyUI # Ming-Image Source: https://docs.sglang.io/cookbook/diffusion/inclusionAI/Ming-Image Deploy Ming-Image Design and Design-Layer with SGLang Diffusion for text-to-image, RGBA image editing, and ordered transparent layer decomposition. ## 1. Quick start Use a nightly Docker image containing this integration, or install from source, following the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). Run the commands below inside that environment. Choose **Design** for generation or editing and **Design-Layer** for decomposition. The two checkpoints require separate servers. Start with one H200 on Linux/CUDA; the picker marks unverified hardware and feature combinations explicitly. Requests return base64 PNGs, preserving all four RGBA channels. For decomposition, each returned image is one ordered layer, not an independent generation. The internal composite frame is omitted from the response. Save all entries in `data`, rather than only the first image. `n` selects independent generations (or sets of layers), processed sequentially. ## 2. Model capabilities [Ming-Image Design](https://huggingface.co/inclusionAI/Ming-Image-0.1-Design) generates images from text and edits a single reference image. It targets design assets, layouts, and transparent RGBA output. Use the prompt prefix `RGBA, 4-channel, transparent background` to request transparency; RGBA output alone does not guarantee a transparent background. [Design-Layer](https://huggingface.co/inclusionAI/Ming-Image-0.1-Design-Layer) decomposes one reference into ordered transparent layers. Supply `num_layers`, or specify a count with `into N layers` or `Number of layers: N` in the prompt; the prompt count takes precedence. Defaults are 12 steps with guidance 1 for Design and guidance 2 for Design-Layer. The model uses zero unconditional embeddings and does not support custom negative prompts. Both checkpoints use the native Bailing multimodal MoE encoder, query connector, joint DiT, and four-channel Qwen-style VAE. No remote modeling code is executed. The original checkpoint directory layout is accepted without conversion. ## 3. Offline generation ```bash Design theme={null} sglang generate --model-path inclusionAI/Ming-Image-0.1-Design \ --prompt "A minimalist exhibition poster with a red geometric sculpture" \ --height 1024 --width 1024 --save-output ``` ```bash Layer decomposition theme={null} sglang generate --model-path inclusionAI/Ming-Image-0.1-Design-Layer \ --image-path /path/to/input.png --num-layers 4 --save-output ``` Design defaults to 2048 by 2048; 1024 by 1024 reduces latency and activation memory. Decomposition defaults to the official 1024-resolution aspect buckets; use `--height 512 --width 512` for the 512 buckets. Reference-image requests select the closest official bucket and restore the input aspect ratio in the returned PNGs. Multiple reference images are not supported. For a renamed local checkpoint directory, also provide `--model-id inclusionAI/Ming-Image-0.1-Design` or the corresponding Layer ID. ## 4. Runtime features DiT parallelism reuses the native TP and sequence-parallel attention layers. The DiT has 30 attention heads: TP multiplied by Ulysses must divide 30. Encoder folding has its own head constraints; two-way folding is compatible with the 16-head language tower, 16-head vision tower, and 12-head connector. Do not infer encoder compatibility from the DiT's head count. Use the standard component residency controls for `transformer`, `text_encoder`, and `vae`. Layerwise offload trades host memory and transfer time for lower VRAM. The default eager path does not use approximate caching or quantized attention. Cache-DiT and SageAttention require workload-specific quality validation. VAE tiling is disabled by default to match the official inference path. Enable `--vae-tiling true` when decode memory is constrained; tiling can change pixels. Breakable CUDA graphs preserve Ming's checkpoint-specific padding and RoPE offsets. They capture exact prompt shapes; an uncaptured shape runs eagerly. Requests with different prompts are not merged into a dynamic batch. A fixed seed does not imply bit-identical output across attention backends, GPU types, or parallel topologies. Full-checkpoint validation covers generation, editing, and layer decomposition on H200, including repeated HTTP requests before and after warmup. Tested resolutions include 2048-square generation and four-layer 1024-square decomposition, plus repeated 512-square HTTP generation and editing with one or two sequential generations per request. The validated runtime scope is: * One H200: DiT and encoder layerwise offload, opt-in VAE tiling, Cache-DiT, SageAttention, breakable CUDA graph replay, and dynamic LoRA loading/removal with a synthetic adapter. * Two H200s: DiT TP2, Ulysses2, Ring2 with FlashAttention, CFG parallelism for Design-Layer, encoder folding, and spatial VAE decoding. These are functional checks, not quality-equivalence claims for lossy optimizations. Other GPU families and quantized checkpoints remain unverified. See [performance optimization](/docs/sglang-diffusion/performance-optimization) for shared runtime controls and the [official repository](https://github.com/inclusionAI/Ming-Image) for model examples. ## 5. Run in ComfyUI Use the generic image node for Design generation. The ComfyUI integration has not been validated for ordered multi-layer output; use the HTTP API above for Design-Layer decomposition. # Overview Source: https://docs.sglang.io/cookbook/diffusion/intro Practical guides for deploying and using diffusion models with SGLang. Choose a recipe by output modality. The sidebar stays organized by model family, while this overview separates image, video, and realtime/world workloads. ## Image Models Image models generate one image request as a bounded denoising job, usually with bidirectional attention over the whole latent sequence. ## Video Models Video models denoise a bounded latent video sequence for each request. Use these recipes for offline text-to-video, image-to-video, and video generation serving. ## Realtime / World Models Realtime models keep a session alive and generate chunk by chunk with causal state, control signals, and cached video history. Use the sidebar group for LingBot World family variants. The overview links the newer LingBot World 2.0 recipe directly. # SGLang Cookbook Source: https://docs.sglang.io/cookbook/intro A community-maintained repository of practical guides and recipes for deploying and using SGLang in production environments. Our mission is simple: answer the question **"How do I use SGLang (and related models) on hardware Y for task Z?"** with clear, actionable solutions. ## Guides ## Benchmarks # SpecBundle Usage Source: https://docs.sglang.io/cookbook/specbundle/specbundle_usage specbundle logo ## About SpecBundle Speculative decoding, especially EAGLE3, offer strong theoretical guarantees alongside consistent empirical improvements in token acceptance rate and end-to-end inference speed. However, despite these advances, adoption of speculative decoding—especially EAGLE3—remains limited in the open-source ecosystem, due primarily to three key factors. 1. Lack of production-ready training infrastructure: Existing speculative decoding toolchains are largely research prototypes, offering limited system-level optimization and inadequate support for diverse architectures and large-scale models. 2. Scarcity of high-quality draft models: Effective speculative decoding depends on strong draft models, yet publicly available EAGLE3-compatible checkpoints are extremely limited, primarily originating from the original authors. 3. Insufficient training scale of existing drafts: Most available draft models are trained on small or curated datasets and fail to generalize to the large, diverse corpora used in modern LLM training, resulting in low token acceptance rates and diminished practical speedups. **SpecBundle** is a direct response to these limitations. Jointly driven by the open-source community and industry partners including **Ant Group**, **Meituan**, **Nex-AGI** and **EigenAI**, **SpecBundle** represents the **first open initiative** aimed at democratizing speculative decoding by providing high-performance, production-grade EAGLE3 draft model weights for mainstream open-source LLMs. This initiative also serves to verify the robustness of the [**SpecForge**](https://github.com/sgl-project/SpecForge) framework through multiple scales and architectures. ## Installation ```bash Command theme={null} git clone https://github.com/sgl-project/SpecForge.git ``` ## Usage ### Launch SGLang Server with SpecBundle models You can use the following command to launch the SGLang server with SpecBundle models. Please add `--tp`, `--ep` and `--mem-fraction-static` arguments when you encounter memory issues. ```bash Command theme={null} python3 -m sglang.launch_server \ --model \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 ``` For example: ```bash Command theme={null} SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python3 -m sglang.launch_server \ --model Qwen/Qwen3-30B-A3B-Instruct-2507 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path lmsys/SGLang-EAGLE3-Qwen3-30B-A3B-Instruct-2507-SpecForge-Nex \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp 4 ``` ### Use SpecBundle to compare the performance of Speculative Decoding draft models We provide a benchmark suite to evaluate the performance of SpecBundle draft models [here](https://github.com/sgl-project/SpecForge/tree/main/benchmarks). #### Example: 1. Launch a SGLang Server ```bash Command theme={null} SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python3 -m sglang.launch_server \ --model Qwen/Qwen3-30B-A3B-Instruct-2507 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path lmsys/SGLang-EAGLE3-Qwen3-30B-A3B-Instruct-2507-SpecForge-Nex \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp 4 ``` 2. Use the benchmark suite to evaluate the performance of SpecBundle draft models `bench_eagle3.py` can help you launch a SGLang server process and a Benchmarking process concurrently. In this way, you don't have to launch the SGLang server manually, this script will manually handle the SGLang launch under different speculative decoding configurations. Some important arguments are: * `--model-path`: the path to the target model. * `--speculative-draft-model-path`: the path to the draft model. * `--port`: the port to launch the SGLang server. * `--trust-remote-code`: trust the remote code. * `--mem-fraction-static`: the memory fraction for the static memory. * `--tp-size`: the tensor parallelism size. * `--attention-backend`: the attention backend. * `--config-list`: the list of speculative decoding configuration to test, the format is `,,,`. * `--benchmark-list`: the list of benchmarks to test, the format is `::`. ```bash Command theme={null} cd SpecForge/benchmarks python bench_eagle3.py \ --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 \ --port 30000 \ --config-list 1,3,1,4 \ --benchmark-list mtbench:5 gsm8k:100 \ --skip-launch-server ``` **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate test command for your model and benchmark. It will generate a json file, content is listed below: ```json Config theme={null} { "mtbench": [ { "batch_size": 1, "steps": null, "topk": null, "num_draft_tokens": null, "metrics": [ { "latency": 12.232808108034078, "output_throughput": 319.71399906382845, "accept_length": 2.170366259711432, "accuracy": null, "num_questions": 5, "num_valid_predictions": 0, "categorical_performance": null } ], "num_samples": 5 } ], "gsm8k": [ { "batch_size": 1, "steps": null, "topk": null, "num_draft_tokens": null, "metrics": [ { "latency": 37.42077191895805, "output_throughput": 373.6160234823207, "accept_length": 2.643410852713178, "accuracy": 0.96, "num_questions": 100, "num_valid_predictions": 100, "categorical_performance": null } ], "num_samples": 100 } ] } ``` ## Performance Scores We evaluate the performance of SpecBundle draft models on various benchmarks, please visit the [Performance Dashboard](https://docs.sglang.io/SpecForge/SpecBundle/index.html) for more details. # Supported Models Source: https://docs.sglang.io/cookbook/specbundle/supported_models ## [Released Models](https://huggingface.co/collections/lmsys/specbundle) We list the models released by the SpecForge and several industrial partners below. These models are released as part of the SpecBundle models, which are trained on large-scale multi-domain datasets and deliver exceptional performance on various benchmarks. > We also include some of the models previously trained by the SpecForge team but not technically part of the SpecBundle release. > We mark models trained on ShareGPT+Ultrachat datasets with a **\*** mark and models trained on Perfect-Blend datasets but released before SpecBundle with **+** mark. ### Llama Series
Target Model EAGLE3 Draft Model
meta-llama/Llama-3.1-8B-Instruct [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Llama-3.1-8B-Instruct-SpecForge)
meta-llama/Llama-3.3-70B-Instruct [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Llama-3.3-70B-Instruct-SpecForge)
meta-llama/Llama-4-Scout-17B-16E-Instruct [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Llama-4-Scout-17B-16E-Instruct-SpecForge)
meta-llama/Llama-4-Maverick-17B-128E-Instruct [🤗 Hugging Face \*](https://huggingface.co/lmsys/sglang-EAGLE3-Llama-4-Maverick-17B-128E-Instruct-v1)
### Qwen Series
Target Model EAGLE3 Draft Model
Qwen/Qwen3-30B-A3B-Instruct-2507 [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-30B-A3B-Instruct-2507-SpecForge-Nex)
Qwen/Qwen3-235B-A22B-Instruct-2507 [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan)
Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-Next-80B-A3B-Instruct-FP8-perfect-blend-regenerated)
### Qwen Coder Series
Target Model EAGLE3 Draft Model
Qwen/Qwen3-Coder-30B-A3B-Instruct [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-Coder-30B-A3B-Instruct-SpecForge)
Qwen/Qwen3-Coder-480B-A35B-Instruct [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-Coder-480B-A35B-Instruct-SpecForge-EigenAI)
### Ling Series
Target Model EAGLE3 Draft Model
inclusionAI/Ling-flash-2.0 [🤗 Hugging Face](https://huggingface.co/AQ-MedAI/Ling-Flash-2.0-eagle3)
### Kimi Series
Target Model EAGLE3 Draft Model
moonshotai/Kimi-K2-Instruct [🤗 Hugging Face](https://huggingface.co/AQ-MedAI/Kimi-K2-Instruct-eagle3)
### GPT-OSS Series
Target Model EAGLE3 Draft Model
openai/gpt-oss-20b [🤗 Hugging Face +](https://huggingface.co/zhuyksir/EAGLE3-gpt-oss-20b-bf16)
openai/gpt-oss-120b [🤗 Hugging Face +](https://huggingface.co/lmsys/EAGLE3-gpt-oss-120b-bf16)
### Nex Series
Target Model EAGLE3 Draft Model
nex-agi/Qwen3-30B-A3B-Nex-N1 [🤗 Hugging Face](https://huggingface.co/nex-agi/SGLANG-EAGLE3-Qwen3-30B-A3B-Nex-N1)
nex-agi/Qwen3-32B-Nex-N1 [🤗 Hugging Face](https://huggingface.co/nex-agi/SGLANG-EAGLE3-Qwen3-32B-Nex-N1)
# FLUX 3 Action Source: https://docs.sglang.io/cookbook/vla/FLUX/FLUX-3-Action ## 1. Model Introduction FLUX 3 Action is a robot policy from Black Forest Labs built on a FLUX 3 video diffusion transformer. From the current camera frames, the robot state and a language instruction, it denoises a chunk of future video latents **jointly** with a chunk of continuous actions; only the actions are returned. The model has three parts: * **DiT** (`JointSingleSeq`, about 6.6B parameters). Each stream (text, `video`, `video_cond`, the action and state streams) runs through its own 5 mode blocks. The text and all streams then share 28 joint single-stream blocks with per-stream modulation and a 4-axis `(t, h, w, l)` RoPE. * **Text encoder**: Qwen3-VL-4B. Eight hidden layers are stacked into a 20480-dim context. * **Video VAE**: a Swin3D neighborhood-attention VAE (NATTEN), 96 latent channels, 32x spatial compression. The sampler is Cosmos UniPC (order 2). Classifier-free guidance is applied per stream (DROID: `4.0` on video, `1.0` on actions). SGLang runs it in the native `multimodal_gen` runtime. Work that does not depend on the noised streams is computed once: * The text context, once per caption, across requests. * The observation and state streams, once per request. * The mode blocks of the noised streams, once per step, shared by the conditional and unconditional passes. Supported checkpoints (`black-forest-labs/flux-3-action-droid`; three 360x640 cameras `wrist`, `left`, `right`; state and action dim 8; 32-action chunks): | `--model-variant` | Recipe | Weights | Latency on 1x RTX 5090 | Latency on 1x H200 | | - | - | - | -: | -: | | (default) / `base` | 4 steps, guidance 4.0 (video) / 1.0 (action) | BF16 | 1.14 s | 0.41 s | | `fp8r` | same | FP8 rowwise | 0.93 s | 0.47 s | | `gd` | guidance-distilled: 4 steps, no CFG | BF16 | 0.63 s | 0.24 s | | `gd-fp8r` | same | FP8 rowwise | 0.52 s | 0.27 s | | `sd` | step-distilled: 1 step | BF16 | 0.18 s | 0.09 s | | `sd-fp8r` | same | FP8 rowwise | 0.16 s | 0.11 s | Latency is the median server-side time per request (`server_timing.infer_ms`) over 50 sequential requests after 10 warmup requests, on one GPU with the default serve command, eager mode (no `torch.compile`). Requests go through the OpenPI WebSocket with msgpack numpy images, and the caption is cached. Measure it with `python -m sglang.multimodal_gen.benchmarks.bench_flux3_action --url ws://127.0.0.1:30000` against a running server. On H200 the FP8r packages are slower than BF16; they save memory. LayerNorm + modulation, SwiGLU and the gated residual run on bit-exact fused kernels; QK RMSNorm + RoPE runs on a fused kernel at bf16 rounding level (set `SGLANG_ENABLE_FUSED_QKNORM_ROPE=0` to use the eager path). FP8r packages load their native E4M3 weights (one scale per output row) and run `torch._scaled_mm` with per-token activation scales. This needs an SM89+ GPU. The frozen text encoder and video VAE are downloaded from the pinned revision of [`black-forest-labs/flux-3-action-base`](https://huggingface.co/black-forest-labs/flux-3-action-base) that the policy config references. References: * [FLUX Action](https://github.com/black-forest-labs/flux-action) * [FLUX 3 Action collection](https://huggingface.co/collections/black-forest-labs/flux-3-action) ## 2. Installation ```bash Command theme={null} git clone https://github.com/sgl-project/sglang.git cd sglang pip install -e "python[diffusion]" ``` The video VAE runs its neighborhood attention on [NATTEN](https://natten.org) when it is installed. NATTEN is not a dependency of `sglang[diffusion]`; without it the VAE uses a compiled FlexAttention fallback with the same windows (actions stay at the bf16 rounding level). The fallback compiles on the first request (about 3 s on H200) and is then no slower than NATTEN for this single-frame encode (24 ms vs. 36 ms per request on H200). To use NATTEN, pick the wheel that matches your torch and CUDA versions from [whl.natten.org](https://whl.natten.org/). ## 3. Model Deployment Serve the DROID policy: ```bash Command theme={null} sglang serve black-forest-labs/flux-3-action-droid \ --model-type diffusion \ --host 127.0.0.1 \ --port 30000 ``` Serve a distilled variant: ```bash Command theme={null} sglang serve black-forest-labs/flux-3-action-droid \ --model-type diffusion \ --model-variant gd \ --port 30000 ``` A local policy export (a directory containing `manifest.json`, `config.native.json` and `model.safetensors`) is detected automatically when passed as the model path. Pass `--revision` to pin a Hub revision. At startup the policy config and weights are checked against the SHA-256 hashes in `manifest.json`. Peak GPU memory and per-request latency on one RTX 5090. The resident rows use the same median protocol as the table above. The layerwise-offload rows are from the earlier single-run measurement. | Flags | Peak memory | Latency | | - | -: | -: | | (none) | 23.3 GiB | 1.14 s | | `--dit-layerwise-offload` | 14.5 GB | 2.51 s | | `--layerwise-offload-components all` | 9.8 GB | 2.56 s | | `--model-variant fp8r` | 17.9 GiB | 0.93 s | | `--model-variant fp8r --dit-layerwise-offload` | 13.2 GB | 1.29 s | Layerwise offload streams the DiT blocks (and, with `all`, the text encoder and VAE blocks) from host memory, so use it only when the resident configuration does not fit. ### 3.1 Multi-GPU Three layouts split the DiT across GPUs. They compose as `--num-gpus` = TP size x SP degree x (2 with CFG parallel): | Flags | What is split | Actions vs 1 GPU | | - | - | - | | `--num-gpus 2 --enable-cfg-parallel` | The conditional and unconditional passes | Bit-identical | | `--num-gpus 2 --tp-size 2` | DiT weights (heads and MLP channels), one all-reduce per block | bf16 rounding level | | `--num-gpus 2 --sp-degree 2` | The joint sequence and the video mode blocks (K/V gather; add `--ulysses-degree 2` for Ulysses) | bf16 rounding level | CFG parallel only helps recipes that use guidance (`base`, `fp8r`). The distilled `gd` and `sd` variants run one pass per step, so both GPUs compute the same pass. TP and SP accept the FP8r packages too. Ring attention is not supported. ```bash Command theme={null} sglang serve black-forest-labs/flux-3-action-droid \ --model-type diffusion \ --num-gpus 2 \ --enable-cfg-parallel \ --port 30000 ``` ### 3.2 Action Request Schema | Field | Type | Description | | - | - | - | | `input.task` | string | Language instruction. | | `input.observation.images` | object | Camera name -> HWC RGB image: a base64 PNG or JPEG (`{"b64_json": ...}`), or a pixel array of uint8 or floats in `[0, 1]`. DROID: `wrist`, `left`, `right` (or the LeRobot names `wrist_image_left`, `exterior_image_1_left`, `exterior_image_2_left`). Alternatively send `composite`: the 540x640 image with the wrist camera on top and the two exterior cameras at half resolution below. | | `input.observation.state` | array | Robot state in dataset units. DROID: 7 joint positions (rad) followed by the gripper position. | | `parameters.num_inference_steps` | integer, optional | Defaults to the checkpoint recipe. | | `parameters.guidance_scale` / `guidance_scale_action` | number, optional | Guidance scale on the video / action stream. Defaults to the checkpoint recipe. | | `parameters.seed` | integer, optional | Noise seed. Defaults to the package's `inference_seed` (`0` for DROID), so a repeated observation returns the same actions. | | `runtime.prefix_cache` | boolean, optional | `false` bypasses the per-caption text context cache for this request. Defaults to `true`. | | `runtime.output_format` | `"list"` or `"numpy"`, optional | Use `"numpy"` with msgpack clients. | The response returns absolute commands of shape `[32, 8]` in the dataset's conventions. ## 4. API Usage ### 4.1 Generic Action HTTP API ```python Example theme={null} import base64 import io import numpy as np import requests from PIL import Image def png(image: np.ndarray) -> dict: buffer = io.BytesIO() Image.fromarray(image).save(buffer, format="PNG") return {"b64_json": base64.b64encode(buffer.getvalue()).decode()} image = np.zeros((360, 640, 3), dtype=np.uint8) payload = { "input": { "task": "put the marker in the cup", "observation": { "images": {"wrist": png(image), "left": png(image), "right": png(image)}, "state": np.zeros(8, dtype=np.float32).tolist(), }, }, } response = requests.post("http://127.0.0.1:30000/v1/actions/generations", json=payload) action = response.json()["data"][0]["action"] print(action["shape"]) # [32, 8] ``` PNG is lossless. Pixel lists (`image.tolist()`) are accepted too, but three 360x640 cameras become about 11 MB of JSON, and encoding and parsing it adds about 0.3 s per request on top of the base64 PNG payload. For the lowest overhead, use the OpenPI WebSocket below with msgpack numpy arrays. `GET /v1/actions/metadata` reports the camera keys, action shape and sampler defaults of the served policy. ### 4.2 OpenPI-Compatible WebSocket `/openpi/policy` takes one observation per message, with camera images as `observation.images.` (`wrist`, `left`, `right`, or the LeRobot names `wrist_image_left`, `exterior_image_1_left`, `exterior_image_2_left`), the state as `observation.state` and the instruction as `task` or `prompt`. The response carries the `[32, 8]` chunk as `actions`. Use the msgpack helpers from the [Pi0.5 page](/cookbook/vla/OpenPI/Pi0.5) to pack numpy arrays. ## 5. Accuracy With the same observation and seed, SGLang matches the FLUX Action reference implementation to the bf16 rounding level: * The DiT agrees to 1e-6 (relative) in fp32. * The VAE latents and text contexts are bit-identical. * Across the full 4-step, CFG 4.0 sampling loop, actions differ by at most 0.015 rad (mean 0.003 rad). The reference's own eager and prepared paths differ from each other by 0.015 rad. * FP8r packages differ from the reference FP8r path by at most 0.025 rad. The reference FP8r path itself differs from BF16 by 0.055 rad. # Pi0.5 Source: https://docs.sglang.io/cookbook/vla/OpenPI/Pi0.5
dVLA OpenPI / LeRobot flow matching action robot edge
## 1. Model Introduction Pi0.5 is an OpenPI / LeRobot diffusion Vision-Language-Action (dVLA) policy. It consumes camera images, a language instruction, and robot state, then returns a continuous action chunk for robot control. SGLang serves Pi0.5 through the native `multimodal_gen` runtime. The implementation uses a SigLIP/PaliGemma prefix encoder and a Gemma action expert: the prefix is encoded once, then the action expert runs the flow-matching denoising loop. This is not a token decode workload, so the Pi0.5 path does not use the LLM sampler, logits processor, token streaming, paged decode KV cache, or a separate SRT serving engine. The SigLIP vision tower, PaliGemma language stack, and action expert are all SGLang-native modules. Transformers is used for checkpoint configuration and tokenization, not for the runtime neural network. The prefix encoder covers both stages of observation encoding: SigLIP turns resized camera pixels into continuous patch embeddings, then the PaliGemma transformer jointly encodes those patches with tokenized task/state inputs and produces per-layer prefix K/V. At flow timestep `t`, the action expert projects the noisy continuous action chunk `x_t` into action embeddings. Its queries attend to both the fixed prefix K/V and the current action K/V, while the timestep follows a separate sinusoidal-MLP path and conditions every action-expert layer through AdaRMSNorm gates. Supported public checkpoints: | Checkpoint | Cameras | State Dim | Output Action Dim | Action Horizon | Denoise Steps | | - | - | -: | -: | -: | -: | | `lerobot/pi05_base` | `base_0_rgb`, `left_wrist_0_rgb`, `right_wrist_0_rgb` | 32 | 32 | 50 | 10 | | `lerobot/pi05_libero_base` | `image`, `image2`, one empty camera | 8 | 7 | 50 | 10 | References: * [OpenPI](https://github.com/Physical-Intelligence/openpi) * [lerobot/pi05\_base](https://huggingface.co/lerobot/pi05_base) * [LeRobot Pi0.5 docs](https://huggingface.co/docs/lerobot/en/pi05) ## 2. Installation Install SGLang with the diffusion extra. Pi0.5 lives in `multimodal_gen`, and the extra includes the runtime dependencies used by the policy server. ```bash Command theme={null} git clone https://github.com/sgl-project/sglang.git cd sglang pip install -e "python[diffusion]" ``` For general environment setup, see the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). ## 3. Model Deployment Serve the base Pi0.5 policy: ```bash Command theme={null} sglang serve lerobot/pi05_base \ --model-type diffusion \ --host 127.0.0.1 \ --port 30000 ``` Serve the LIBERO checkpoint: ```bash Command theme={null} sglang serve lerobot/pi05_libero_base \ --model-type diffusion \ --host 127.0.0.1 \ --port 30000 ``` These registered LeRobot checkpoints dispatch to the native `multimodal_gen` Pi0.5 pipeline automatically. The `--pipeline` / `--pipeline-class-name` flag is only an advanced override for local or private checkpoints that cannot be resolved from the model registry. Pi0.5 serving does not start a second SRT LLM serving engine. ### 3.1 Action Request Schema | Field | Type | Description | | - | - | - | | `model` | string, optional | Served model name. When omitted, the server uses the currently loaded policy. | | `input.task` or `input.prompt` | string | Language instruction for the policy. | | `input.observation.images` | object | Map from camera name to RGB image. Images can be nested here or sent as OpenPI observation keys such as `observation.images.base_0_rgb` through the websocket adapter. | | `input.observation.state` | array or tensor object | Robot state vector. Use the same normalization convention as the OpenPI / LeRobot checkpoint. | | `input.observation.noise` | array or tensor object, optional | Initial action noise with shape `[action_horizon, action_dim]`. Use this for deterministic debugging. | | `parameters.num_inference_steps` | integer, optional | Flow-matching denoise steps. Defaults to `10`. | | `parameters.action_horizon` | integer, optional | Output action horizon. Defaults to the checkpoint config. | | `parameters.action_dim` | integer, optional | Internal padded action dimension. Defaults to the checkpoint config. | | `runtime.return_timing` | boolean, optional | Return stage timing fields. Defaults to `true`. | | `runtime.prefix_cache` | boolean or `"auto"`, optional | Enable exact full-prefix lookup for this request when the server has `enable_global_prefix_cache=true`. Defaults to `"auto"`. | | `runtime.cuda_graph` | boolean or `"auto"`, optional | Enable the available prefix and action-denoise CUDA graph paths for this request. Defaults to `"auto"`. | | `runtime.output_format` | `"list"` or `"numpy"`, optional | Use `"list"` for JSON compatibility. Use `"numpy"` with msgpack or Python clients to avoid Python-list materialization. Defaults to `"list"`. | | `runtime.response_format` | `"envelope"` or `"raw"`, optional | HTTP-only response shape. `"envelope"` returns the generic action envelope. `"raw"` returns the policy payload directly. Defaults to `"envelope"`. | ## 4. API Usage ### 4.1 Generic Action HTTP API Use `/v1/actions/generations` for direct policy calls and debugging. Use `/v1/actions/metadata` to discover the camera keys, state size, action shape, defaults, and websocket capabilities of the currently served policy. ```python Example theme={null} import numpy as np import requests image = np.zeros((224, 224, 3), dtype=np.uint8) payload = { "model": "lerobot/pi05_base", "input": { "task": "pick up the block", "observation": { "images": { "base_0_rgb": image.tolist(), "left_wrist_0_rgb": image.tolist(), "right_wrist_0_rgb": image.tolist(), }, "state": np.zeros(32, dtype=np.float32).tolist(), }, }, "runtime": { "return_timing": True, "prefix_cache": "auto", "cuda_graph": "auto", }, } response = requests.post( "http://127.0.0.1:30000/v1/actions/generations", json=payload, timeout=60, ) response.raise_for_status() data = response.json() actions = data["data"][0]["action"]["values"] print(len(actions), len(actions[0])) print(data.get("timings")) ``` The same `/v1/actions/generations` endpoint also accepts `Content-Type: application/msgpack` and can return msgpack when `Accept: application/msgpack` is set. For msgpack clients, send numpy arrays directly using the `pack_numpy_payload` helper from the websocket example below. Msgpack requests default to numpy action output on the server side; set `runtime.output_format` to `"list"` only when a client explicitly needs nested Python lists. For the lowest-overhead generic HTTP path, use msgpack with the raw policy response: ```python Example theme={null} payload["runtime"] = { "return_timing": True, "prefix_cache": False, "cuda_graph": "auto", "response_format": "raw", } response = requests.post( "http://127.0.0.1:30000/v1/actions/generations", data=packb(payload), headers={ "Content-Type": "application/msgpack", "Accept": "application/msgpack", }, timeout=60, ) result = unpackb(response.content) actions = result["actions"] ``` For `lerobot/pi05_libero_base`, use the LIBERO camera names and an 8-dimensional state vector: ```python Example theme={null} payload = { "input": { "task": "pick up the object", "observation": { "images": { "image": image.tolist(), "image2": image.tolist(), }, "state": np.zeros(8, dtype=np.float32).tolist(), }, }, } ``` ### 4.2 Generic Realtime WebSocket `/v1/actions/realtime` is the generic msgpack websocket path. It sends `action.metadata` on connect and returns the same `action.generation` envelope as the HTTP API for each request. ### 4.3 OpenPI-Compatible WebSocket Robot clients can use the OpenPI-compatible msgpack websocket endpoint at `/openpi/policy`. The server sends metadata immediately after connection, then each client message should contain one observation. ```python Example theme={null} import asyncio import msgpack import numpy as np import websockets def pack_array(obj): if isinstance(obj, np.ndarray): return { b"__ndarray__": True, b"data": obj.tobytes(), b"dtype": obj.dtype.str, b"shape": obj.shape, } if isinstance(obj, np.generic): return { b"__npgeneric__": True, b"data": obj.item(), b"dtype": obj.dtype.str, } return obj def unpack_array(obj): ndarray_marker = obj.get("__ndarray__") or obj.get(b"__ndarray__") npgeneric_marker = obj.get("__npgeneric__") or obj.get(b"__npgeneric__") data = obj.get("data", obj.get(b"data")) dtype = obj.get("dtype", obj.get(b"dtype")) shape = obj.get("shape", obj.get(b"shape")) if ndarray_marker: return np.ndarray( buffer=data, dtype=np.dtype(dtype), shape=shape, ) if npgeneric_marker: return np.dtype(dtype).type(data) return obj def packb(payload): return msgpack.packb(payload, default=pack_array, use_bin_type=True) def unpackb(payload): return msgpack.unpackb(payload, object_hook=unpack_array, raw=False) async def main(): image = np.zeros((224, 224, 3), dtype=np.uint8) observation = { "task": "pick up the block", "observation.images.base_0_rgb": image, "observation.images.left_wrist_0_rgb": image, "observation.images.right_wrist_0_rgb": image, "observation.state": np.zeros(32, dtype=np.float32), } async with websockets.connect( "ws://127.0.0.1:30000/openpi/policy", max_size=None, ) as websocket: metadata = unpackb(await websocket.recv()) print(metadata) await websocket.send(packb(observation)) result = unpackb(await websocket.recv()) actions = result["actions"] print(len(actions), len(actions[0])) print(result.get("server_timing")) asyncio.run(main()) ``` ## 5. Configuration Tips * Request-local `PrefixContext` is always reused across all denoise steps in one request. The prefix K/V is not cloned per step. * The optional global prefix cache is a bounded exact-match LRU. It is disabled by default because changing robot frames rarely hit it and enabling it prevents unrelated misses from entering grouped prefix execution. Set `enable_global_prefix_cache=true` for repeated observations, retries, or multiple policy calls over the same camera/state sample; `runtime.prefix_cache` can then disable lookup per request. * Partial-prefix reuse is not supported because Pi0.5 combines image and tokenized task/state inputs under full attention. Changing any input can change every deeper-layer prefix K/V tensor. The exact key hashes resized and normalized pixels before SigLIP, plus effective token IDs, token masks, camera masks, model revision, dtype, and parallel layout. Tensor content hashing reuses SRT's CPU/CUDA implementation; hashing the pre-SigLIP input lets an exact hit skip both the vision encoder and prefix transformer. * CUDA graph capture targets single-request prefix encoding and one action-denoise step. The lossless default keeps exact prompt lengths and one resident prefix graph. Prefix and action graph residency are independently bounded by `prefix_cuda_graph_max_entries` (default `1`) and `action_cuda_graph_max_entries` (default `4`). Without prompt buckets, unseen signatures run eagerly after the cache reaches capacity, avoiding capture churn. The denoise signature includes whether prefix attention is full or masked, plus batch size, prefix length, action horizon, action dim, dtype, and parallel layout. Mask-aware action graphs rebuild position IDs from the current request mask inside the captured graph. With action SP enabled, the denoise bucket uses the local action shard length and rank-specific position offset. * `prompt_token_buckets` optionally right-pads prompt tokens to fixed sizes such as `[32, 64, 128, 200]`, allowing nearby prompt lengths to reuse both graphs. In this mode the bounded graph caches use LRU replacement and reset evicted CUDA graphs; prompts beyond the largest configured bucket stay exact-length and eager. This is opt-in because padding changes GPU reduction shapes: an H200 sweep over lengths around all four boundaries was structurally correct and bounded to four prefix/action graphs, but non-boundary prompts differed from the exact-length path by up to `0.09589` in normalized action space over five denoise steps. Validate closed-loop policy quality before enabling it; empty buckets preserve the numerically lossless path. An opt-in four-bucket serving config is: ```json File theme={null} { "prompt_token_buckets": [32, 64, 128, 200], "prefix_cuda_graph_max_entries": 4, "action_cuda_graph_max_entries": 4 } ``` * Cache-DiT is not used in the default Pi0.5 path. The current robot policy target is numerically lossless inference, while Cache-DiT-style reuse is an image/video DiT approximation that needs separate policy-quality validation before it can be recommended for action control. * Do not use CFG parallelism to split the 10 Euler steps. Use it only for independent branches such as multiple candidate actions or future conditional/unconditional branches. * Prefix TP uses native SGLang parallel linear layers for the PaliGemma language prefix model when model parallel TP is initialized and the VLA split broadcast group is not active. The action expert does not share that TP layout. The v1 split prefix/action path instead uses the SP group: prefix root computes/broadcasts `PrefixContext`, while action ranks run the SP action path. * The split path uses the SP group as the action group: prefix root computes/broadcasts `PrefixContext`, all action ranks broadcast the initial action noise once, shard the action horizon, and run the action expert through Ulysses attention when the prefix is full-attention, ring degree is one, heads are divisible by SP size, and the horizon is evenly shardable. Otherwise it falls back to action-root execution. * `lerobot/pi05_libero_base` returns 7 action dimensions even though the internal padded action tensor uses 32 dimensions. ## 6. VRAM Tuning For current public Pi0.5 checkpoints, a stable 16GB discrete GPU target is a reasonable v1 deployment bar for robot workstations. In practice, leave headroom for the driver, camera middleware, robot process, and allocator fragmentation. Jetson/Orin unified-memory devices need extra caution because system RAM and GPU memory share the same pool. OpenPI inference is mixed precision, not full fp32: most weights and compute run in `bf16`, selected stability-sensitive weights stay in `fp32`, and returned actions are `float32`. SGLang mirrors that policy by default. The validated `pi05_aloha` OpenPI PyTorch checkpoint keeps `119,720,608` parameters in fp32; SGLang reports the same fp32 stability set, plus `3,233,713,264` bf16 runtime parameters after skipping unused LM heads for continuous action inference. The fp32 set includes SigLIP patch/position embeddings, Gemma layer norms/final norms, and the action/time projection heads. Keep `materialize_dtype` at `bf16` unless you are debugging numerical parity. Current H100 pressure validation shows that the bf16 model path fits a 16GB-free budget without layerwise offload. The Pi0.5 path batches all camera frames for a grouped request into one SigLIP forward before splitting the embeddings back by camera, which is important for multi-camera robot workloads. The latency numbers below use the Python grouped API with global prefix cache disabled and CUDA graph enabled on unconstrained H100; rerun HTTP/OpenPI websocket on your target server before using the policy in a closed-loop robot. | Mode | Command Shape | Steady VRAM Snapshot | Notes | | - | - | - | - | | Single GPU bf16 | `--num-gpus 1`, no offload | fit with `16383 MiB` free before load in H100 pressure test | Recommended first path for 16GB-class discrete GPUs. | | ALOHA bf16 grouped | Python API, batch size 1 / 2 / 4 / 8 | unconstrained H100 | `52.4 ms` single; `65.1 ms / 2`; `91.9 ms / 4`; `147.4 ms / 8`. | | ALOHA 16GB-free pressure | Python API, no offload | same 16GB-free pressure | Fit was validated after the bf16 correction; rerun latency on the target because older pressure latency was collected before the final OpenPI precision fix. | | Global prefix cache disabled | `enable_global_prefix_cache=false` or request `runtime.prefix_cache=false` | avoids cache growth across changing frames | Recommended for tight edge budgets unless repeated exact frames are common. | | Offload fallback | config in 6.2 | target-dependent | Use only if the default bf16 path OOMs on the target device. | Use these knobs first because they do not change action numerics for a fixed input/noise: ### 6.1 16GB Edge Config Use this single-GPU config first for 16GB-class robot workstations. It keeps parameters resident on GPU, keeps CUDA graph enabled, and disables global prefix cache growth. ```json File theme={null} { "materialize_dtype": "bf16", "enable_global_prefix_cache": false, "prefix_cache_max_entries": 0, "enable_prefix_cuda_graph": true, "prefix_cuda_graph_max_entries": 1, "enable_action_cuda_graph": true, "action_cuda_graph_max_entries": 1 } ``` Start a single-GPU server with the override: ```bash Command theme={null} sglang serve lerobot/pi05_base \ --model-type diffusion \ --pipeline-config-path pi05_edge_16gb.json \ --num-gpus 1 \ --warmup-mode off \ --host 127.0.0.1 \ --port 30000 ``` Treat the H100 pressure result as a memory-budget validation, not a Jetson latency guarantee. Jetson/Orin devices use shared system memory and have much lower effective memory bandwidth than H100, so measure closed-loop control latency on the target device before deciding the action chunk cadence. ### 6.2 Offload Fallback Use offload only when the default bf16 config still does not fit on the target. These modes are numerically lossless for fixed inputs/noise, but they move weights between CPU and GPU and can hurt latency substantially. For a moderate fallback, offload cache growth and selected stage-resident modules first: ```json File theme={null} { "materialize_dtype": "bf16", "enable_global_prefix_cache": false, "prefix_cache_max_entries": 0, "enable_prefix_cuda_graph": false, "enable_action_cuda_graph": false, "offload_prefix_image_encoder_after_embed": true, "offload_prefix_token_embedding": true, "offload_prefix_language_layer_count_after_prefix": 2, "offload_action_expert_after_denoise": true, "empty_cache_after_prefix": true } ``` If that still does not fit, full prefix layerwise CPU offload keeps every PaliGemma language layer on CPU and moves one layer at a time to GPU during prefix compute: ```json File theme={null} { "materialize_dtype": "bf16", "enable_global_prefix_cache": false, "prefix_cache_max_entries": 0, "enable_prefix_cuda_graph": false, "enable_action_cuda_graph": false, "offload_prefix_image_encoder": true, "offload_prefix_token_embedding": true, "offload_prefix_language_layers": true, "offload_prefix_language_layers_empty_cache": true, "empty_cache_after_prefix": true } ``` Offload validation should be repeated on the target hardware after any dtype or loader change. Earlier fp32-runtime offload numbers are not comparable to the current bf16 path. ### 6.3 Per-Request Controls For HTTP calls, disable cache or both CUDA graph paths without restarting the server: ```python Example theme={null} payload = { "input": { "task": "pick up the block", "observation": { "images": images, "state": state, }, }, "runtime": { "prefix_cache": False, "cuda_graph": False, }, } ``` For OpenPI websocket clients, include equivalent `enable_prefix_cache` and `enable_cuda_graph` fields in each raw msgpack observation if you need per-request compatibility controls. ### 6.4 Deployment Choices * Keep batch size to one control stream unless grouped robot streams are explicitly validated. More concurrent observations increase activation and PrefixContext residency. * Keep `materialize_dtype` at the default `bf16`. `fp32` is useful only for debugging numerical issues and will increase memory and latency. * Use split prefix/action only when you have multiple GPUs and have validated the robot control latency on that topology. On two GPUs with `--sp-degree 2 --ulysses-degree 2`, the action horizon can be sequence-sharded while the prefix root still computes and broadcasts `PrefixContext`. A single 16GB-class GPU should try the default bf16 config first. * Reducing `num_inference_steps` lowers latency but is not a pure memory fix and can change policy behavior. Validate closed-loop task success before using fewer than the checkpoint default. * CPU offload is a compatibility fallback, not the preferred 16GB path. Quantization and deeper prefix/vision sharding are the next steps for sub-16GB devices. ### 6.5 Loader And Run:ai Model Streamer Run:ai Model Streamer can improve cold-start and checkpoint loading by reading safetensors concurrently and streaming tensors toward GPU memory. It is useful for local SSD, object storage, and cloud deployments where startup time is dominated by model file IO. The `python[diffusion]` extra includes `runai_model_streamer`. If the package is installed, SGLang enables it by default through `SGLANG_USE_RUNAI_MODEL_STREAMER=true`. Set it to `false` to force the plain safetensors loader: ```bash Command theme={null} SGLANG_USE_RUNAI_MODEL_STREAMER=false \ sglang serve lerobot/pi05_base \ --model-type diffusion \ --host 127.0.0.1 \ --port 30000 ``` Pi0.5 uses the direct SSD-to-GPU Run:ai path when all load targets for the current process are GPU-resident. This check is rank-local: distributed action ranks can stream their action-expert subset directly to GPU, while any rank with CPU/offloaded target tensors stays on the CPU safetensors fallback. The single-GPU validation streamed `13.5 GiB` of safetensors directly to `cuda:0` in about `1.5 s`, then returned `[50, 32]` actions over both HTTP and OpenPI websocket. For mixed CPU offload and low-VRAM 16GB-class modes, Pi0.5 still uses the header-filtered safe loader for ranks whose target tensors are not fully CUDA-resident. Direct GPU streaming can increase the exact VRAM pressure the low-memory path is trying to avoid. When debugging distributed startup, compare with `SGLANG_USE_RUNAI_MODEL_STREAMER=false` to separate streamer behavior from model execution behavior. Run:ai Model Streamer does not reduce steady inference VRAM after parameters, caches, activations, CUDA contexts, and graph buffers are resident. For Pi0.5 low-VRAM work, prioritize component placement, prefix cache size, CUDA graph residency, and CPU/offload first. Model Streamer is a cold-start optimization after the steady-memory budget is correct. ## 7. OpenPI Comparison Benchmark Use `bench_pi05_openpi.py` when you need a side-by-side latency and action-difference report against the OpenPI policy implementation. Start the SGLang Pi0.5 server first for HTTP or websocket modes, then run the benchmark from the SGLang repository root: ```bash Command theme={null} python python/sglang/multimodal_gen/benchmarks/bench_pi05_openpi.py \ --profile aloha \ --sglang-url http://127.0.0.1:30000 \ --openpi-checkpoint gs://openpi-assets/checkpoints/pi05_base \ --num-inference-steps 10 \ --batch-size 4 \ --repeats 20 \ --warmup 3 \ --deterministic-noise \ --output pi05_openpi_compare.json ``` The benchmark reports: * SGLang HTTP latency for `/v1/actions/generations`, including stage timings when the server returns them. * SGLang msgpack HTTP latency when `--sglang-api http_msgpack` is set. This keeps the generic HTTP endpoint but avoids JSON image-array overhead. Use `--sglang-http-response-format raw` to benchmark the compact raw policy response. * SGLang OpenPI-compatible websocket latency when `--sglang-api openpi_ws` is set. This uses persistent msgpack websocket connections and is closer to the robot client path than JSON-over-HTTP. * SGLang Python in-process latency when `--sglang-api python` is set. This loads the native Pi0.5 pipeline in the benchmark process and avoids HTTP, websocket, scheduler, and serialization overhead. Use `--sglang-python-batch-mode grouped` to exercise the conservative native grouped-batch path. The Python path also reports actual SGLang module parameter dtype counts and example parameter names. * OpenPI single-request latency through `Policy.infer`. * Batch latency for grouped robot streams. SGLang uses concurrent HTTP requests in HTTP mode and persistent multi-connection msgpack calls in websocket mode. The Python backend can use true grouped model execution for fresh-prefix requests. OpenPI defaults to its internal direct model batch path because the public `Policy.infer` API is single-observation. * Action difference in normalized model space from identical OpenPI-transformed model inputs and noise. The check requires `--deterministic-noise` and fails when either `--action-max-abs-diff` or `--action-mean-abs-diff` is exceeded. This mode isolates model parity from robot-specific normalization and action postprocessing. The LIBERO policy returns only 10 actions after policy postprocessing, but its flow-matching model still generates a 50-step chunk; the benchmark compares that model output with SGLang before OpenPI unnormalization and horizon slicing. For one-sided 16GB-class checks, run each backend separately under the same VRAM pressure. The SGLang Python path accepts the same pipeline config override as serving: ```bash Command theme={null} python python/sglang/multimodal_gen/benchmarks/bench_pi05_openpi.py \ --profile aloha \ --sglang-api python \ --sglang-python-batch-mode grouped \ --skip-openpi \ --sglang-pipeline-config-path pi05_edge_16gb.json \ --num-inference-steps 10 \ --batch-size 4 \ --repeats 5 \ --warmup 2 \ --deterministic-noise \ --disable-prefix-cache \ --disable-cuda-graph ``` Use `--skip-sglang --openpi-pytorch-compile-mode none` to measure an OpenPI eager baseline in the same environment. For a PyTorch-native OpenPI baseline, point `--openpi-checkpoint` at a checkpoint directory containing `model.safetensors` and the OpenPI `assets/` norm-stat tree. The `keep` mode preserves OpenPI's checkpoint default compile setting. The grouped Python path currently requires compatible fresh-prefix requests without split prefix/action workers or effective prefix-cache hits; other cases fall back to per-request execution. ## 8. Validation Notes The following checks were run on H100 GPUs with the native SGLang Pi0.5 path: | Check | Result | | - | - | | `lerobot/pi05_base` direct end-to-end | Prefix length `968`, output shape `[1, 50, 32]`, peak allocated memory `12.817 GiB`. | | Official OpenPI parity | Against OpenPI PyTorch revision `15a9616`, with the same LeRobot checkpoint revision, observation, and noise: first-step velocity max/mean absolute difference `0.02677` / `0.00344`; production 10-step normalized action `0.00813` / `0.00092`. | | Action denoise CUDA graph | Eager 10-step denoise `125.4 ms`; steady graph replay `50.8 ms`; max output difference `0`. | | Prefix CUDA graph | On H200 with action graph already enabled, ALOHA batch=1 p50 improved from `48.04 ms` to `42.98 ms` (`1.118x`). Two observations at 5 and 10 steps were bit-exact with graph disabled. One prefix shape bucket added about `48.5 MiB`; batch=4 showed no benefit and stays eager. | | Exact full-prefix cache | First prefix pass about `203 ms`; exact cache hit prefix stage about `0.2 ms`. | | `lerobot/pi05_libero_base` direct end-to-end | Image keys `image`, `image2`, `empty_camera_0`; state dim `8`; output action dim `7`; output tensor shape `[1, 50, 32]`. | | Python grouped execution | ALOHA batch=4 grouped path measured `91.9 ms / 4` on current mixed precision: prefix `18.4 ms`, action denoise `61.2 ms`, preprocess about `2.4 ms` per request. Sequential Python loop batch=4 measured `211.9 ms / 4`. | | `sglang serve` HTTP | `/v1/actions/generations` returned action shape `[50, 32]`. JSON-over-HTTP remains compatible but image-array serialization dominates. Msgpack HTTP with prefix cache disabled measured `57.2 ms` single and `221.9 ms / 4` for the envelope response, and `56.4 ms` single and `219.7 ms / 4` for `runtime.response_format="raw"`. | | OpenPI websocket | `/openpi/policy` returned action shape `[50, 32]`; with persistent msgpack connections and prefix cache disabled, ALOHA measured about `77.0 ms` single and `162.0 ms / 4`. | | 2-GPU prefix/action split | Earlier split validation returned action shape `[50, 32]` and matched single-GPU HTTP with max absolute difference `0`. After the true action-SP change, rerun this check with `--num-gpus 2 --sp-degree 2 --ulysses-degree 2` and verify both action ranks enter denoise kernels. | | OpenPI/SGLang precision | Official OpenPI JAX inference restores the public GCS checkpoint as bf16 with selected fp32 stability compute and returns float32 actions. The converted OpenPI PyTorch `pi05_aloha` checkpoint keeps `119,720,608` fp32 stability params; SGLang reports the same fp32 set and `3,233,713,264` bf16 runtime params after skipping unused LM heads. | | Native attention dtype | Checkpoint source tensors may be fp32, but SGLang finalizes PiGemma and SigLIP compute dtype before native attention backend selection; backend logs showed `Using fa attention backend` for the PiGemma path in the prior run. | | 16GB-free Python pressure | With an H100 artificially constrained to `16381 MiB` free before model load, single-GPU bf16 no-offload Python grouped path completed without OOM. Re-run latency after precision or loader changes before using pressure numbers for deployment sizing. | | Low-VRAM switches | Disabling prefix cache prevents cache growth across changing robot frames. Prefix graph residency defaults to `1`, action graph residency defaults to `4`, and setting either limit to `0` disables that runner. One validated ALOHA prefix graph used about `48.5 MiB`; action graph residency is now bounded separately. Keep `prompt_token_buckets` empty for the lossless path. | | Offload fallback | CPU/offload modes are retained as numerically lossless compatibility fallbacks, but earlier fp32-runtime offload latency numbers are stale after the bf16 dtype correction and should be revalidated before deployment decisions. | | Run:ai direct loader | Single-GPU serve streamed `13.5 GiB` safetensors to `cuda:0` in about `1.5 s` and returned `[50, 32]` actions. Distributed direct streaming is now rank-local and should be revalidated on the target split topology; offload ranks with CPU targets still use the safe loader. | | OpenPI comparison status | Official OpenPI GCS `pi05_base` is a JAX checkpoint; converted PyTorch eager was validated without `torch.compile`. On 80GB H100, ALOHA OpenPI PyTorch eager was about `125-130 ms` single and about `164 ms / 4` in the direct-model batch path. Current SGLang Python grouped measured `52.4 ms` single and `91.9 ms / 4`; JAX OpenPI was `53.0 ms` single and `59.5 ms / 2` in a short check. | Run a full HTTP or websocket smoke test in your robot deployment environment before using the policy in a closed-loop controller. # Overview Source: https://docs.sglang.io/cookbook/vla/intro Practical guides for deploying and using Vision-Language-Action policies with SGLang. Vision-Language-Action policies map camera observations, language instructions, and robot state into continuous action chunks. They share some runtime machinery with diffusion pipelines, but the user-facing workload is robot action inference rather than image or video generation. This section keeps VLA policies separate from the diffusion model cookbook so robot deployments can document their own input schemas, policy endpoints, cache behavior, and control-loop performance targets. ## OpenPI ## FLUX # Adaptive Speculative Decoding Source: https://docs.sglang.io/docs/advanced_features/adaptive_speculative_decoding Adaptive speculative decoding lets SGLang adjust `speculative_num_steps/speculative_num_draft_tokens` at runtime instead of keeping a single fixed value for the whole server lifetime. It is designed for workloads whose accept length changes over time, where one static step count is rarely optimal. ## Current support * Only `--speculative-algorithm EAGLE` or `EAGLE3` * Only `--speculative-eagle-topk 1` * If either condition is not met, SGLang falls back to static speculative settings ## Why adaptive steps help `speculative_num_steps` controls how many draft-model autoregressive steps run in each speculative round. In practice, the best value depends on the current workload. * If `num_steps` is too small, the draft model could have produced more accepted tokens, but the round stops too early. * If `num_steps` is too large, the draft model produces many candidate tokens that the target model rejects, so extra draft work is wasted. * At **high batch sizes**, the cost of each wasted draft step is multiplied across all sequences in the batch, so the optimal step count is often lower than at low batch sizes. Real traffic often moves between high-acceptance and low-acceptance phases, and batch sizes vary continuously. Adaptive mode follows both signals at runtime instead of hard-coding a single global `num_steps`. ## Design overview The adaptive mechanism has three pieces: * `AdaptiveSpeculativeParams`: the EMA-based policy * `SpecRuntimeState`: the per-tier runtime state bundle * `AdaptiveController`: the coordinator that queries the policy for the current batch size and activates the matching runtime state ### Per-batch-size independent tracking The controller maintains **independent EMA trackers for each batch size range**, so observations at small BS don't pollute the large BS signal. Each BS range can have its own candidate steps, hysteresis thresholds, and ceiling coefficient. BS ranges are defined as lower bounds in the config file (e.g., keys `"1"` and `"8"` mean BS 1–7 uses one slot, BS 8+ uses another). `SpecRuntimeState` objects are shared across BS ranges with the same step count — each state owns CUDA graphs captured for the reachable padded batch sizes of that step. ```mermaid theme={null} --- title: "SpecRuntimeState — speculative_num_steps / speculative_num_draft_tokens" --- graph LR subgraph SR[" "] direction LR subgraph D["Draft stage"] direction TB d1[attn_backend] d2[cuda_graph] end subgraph V["Verify stage"] direction TB v1[attn_backend] v2[cuda_graph] end subgraph E["Extend stage"] direction TB e1[attn_backend] e2[cuda_graph] end end ``` This matters because `CudaGraphRunner` is shape-dependent. Each candidate tier owns its own graph and backend state, so runtime switching is a reference swap, not an online graph recapture. ## Runtime flow The adaptive update happens in two places: 1. **Pre-draft**: query the optimal step for the current batch size and activate if different 2. **Post-verify**: update the matching BS slot's EMA with observed accept lengths ```mermaid theme={null} --- title: "EAGLEWorker.forward_batch_generation() — decode path" --- flowchart TD Z["⓪ activate_step_by_batch(batch_size)
query optimal step for current BS range, activate if different"] A["① draft(batch)
draft model multi-step generation with current tier"] B["② verify(batch, spec_info)
target model tree verification → produces num_correct_drafts_per_req"] C["③ forward_draft_extend_after_decode(batch)
draft model KV-cache catch-up"] D["④ adaptive_controller.on_verify_complete(num_correct_drafts_per_req, batch_size)
update EMA for matching BS slot, apply warmup / interval / hysteresis gates
if tier changed, select a pre-built state from pool"] E["worker.apply_runtime_state(state)"] Z --> A --> B --> C --> D --> E ``` > Tier switch happens after the current round completes. Backends and CUDA graphs are never swapped mid-round. ## How the policy decides After each verify pass, SGLang reads the accepted draft length per request, computes the batch average, smooths it with an exponential moving average (EMA), and switches among the candidate tiers for the matching BS slot. The decision logic is intentionally conservative: * `warmup_batches` skips the first few batches * `update_interval` avoids switching every batch * `down_hysteresis` and `up_hysteresis` reduce oscillation * `ceiling_coeff` — an optional EMA ceiling rule can cap `num_steps` proportionally to observed draft quality, preventing over-speculation at high BS Conceptually, the policy probes one step beyond the observed acceptance: ```text theme={null} target_steps ≈ clamp(round(ema_accept_len) + 1, min(candidate_steps), max(candidate_steps)) ``` So if recent requests consistently accept more drafted tokens, the policy tends to move up. If they start rejecting earlier, it tends to move down. ## Usage `--speculative-adaptive-config` is optional, but the speculative setup still needs to be valid for adaptive mode. ```bash theme={null} python3 -m sglang.launch_server \ --model meta-llama/Llama-2-7b-chat-hf \ --speculative-algorithm EAGLE \ --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \ --speculative-eagle-topk 1 \ --speculative-num-steps 3 \ --speculative-num-draft-tokens 4 \ --speculative-adaptive ``` If you want to override the defaults, add `--speculative-adaptive-config /path/to/adaptive_spec.json`. Example config: ```json theme={null} { "ema_alpha": 0.2, "warmup_batches": 10, "update_interval": 5, "1": {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 0}, "8": {"candidate_steps": [1], "up_hysteresis": 0.0, "down_hysteresis": 0.0, "ceiling_coeff": 0} } ``` Non-integer keys (`ema_alpha`, `warmup_batches`, `update_interval`) are global overrides applied to every BS slot. Integer keys (`"1"`, `"8"`) define per-BS slots. ## Config file reference The config file is optional. When provided, each integer BS-slot key must specify `candidate_steps`; all other keys fall back to defaults. ### Per-BS slot parameters
Key Default Meaning
candidate\_steps required Candidate speculative\_num\_steps tiers for this BS range. Must be a non-empty list of positive ints; a slot that omits it raises a config error
down\_hysteresis -0.25 Extra margin before moving to a smaller step
up\_hysteresis 0.0 Extra margin before moving to a larger step
ceiling\_coeff 0 (disabled) EMA ceiling coefficient; set > 0 to cap steps proportionally to draft quality
### Global parameters
Key Default Meaning
ema\_alpha 0.2 EMA smoothing factor for accepted draft length
update\_interval 5 Recompute interval, in verify batches, after warmup
warmup\_batches 10 Number of verify batches to observe before switching
## Monitoring You can inspect the active tier and acceptance metric via `/server_info`: ```bash theme={null} curl -s http://127.0.0.1:30000/server_info | jq '.internal_states[0] | {speculative_num_steps, avg_spec_accept_length}' ``` * `speculative_num_steps` is the current active tier * `avg_spec_accept_length` helps explain whether the server is likely to move up or down ## Tuning tips * Start with the built-in default (conservative) — it is safe for all draft model qualities * For strong draft models, use the aggressive config with ceiling rule * Use fewer candidate steps if you want lower startup GPU memory overhead * Increase `ema_alpha` to react faster, or lower it for more stability * Increase `warmup_batches` or `update_interval` if tier switching is too noisy * At high batch sizes, narrower ladders (e.g., `[1, 2]` or `[1]`) often outperform wide ones * If your workload is already stable and one static setting is well tuned, adaptive mode may not help much ## Recommended configs The built-in default is conservative — safe for all draft models but may under-speculate for strong ones. Save one of these as a JSON file and pass via `--speculative-adaptive-config`. ### Conservative (default) — for weak draft models This is the built-in default: BS 8–31 allows `[1, 3]`, and BS≥32 locks to `step=1` to avoid wasted compute. Best for models like MiniMax-M2.5, DSV4. ```json theme={null} { "1": {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 0}, "8": {"candidate_steps": [1, 3], "up_hysteresis": 0.0, "down_hysteresis": 0.0, "ceiling_coeff": 0}, "32": {"candidate_steps": [1], "up_hysteresis": 0.0, "down_hysteresis": 0.0, "ceiling_coeff": 0} } ``` ### Aggressive — for strong or high-variance draft models Uses wider ladders with ceiling rule to cap speculation at high BS. Best for models like GLM-4.7-FP8. ```json theme={null} { "1": {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 0}, "8": {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 3.0}, "64": {"candidate_steps": [1, 3], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 1.67}, "128": {"candidate_steps": [1, 3], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 1.2} } ``` ### Custom per-model config For the best performance, benchmark your specific model across batch sizes with different static `num_steps` values, then build a per-BS config that matches each range's optimal step. A well-tuned per-model config might outperform the generic presets above. # Attention Backend Source: https://docs.sglang.io/docs/advanced_features/attention_backend SGLang supports a large variety of attention backends. Each of them has different pros and cons. You can test them according to your needs. Selecting an optimal attention backend is crucial for maximizing your performance. Different backends excel in various scenarios, so choose based on your model, hardware, and use case. Not all backends are supported on all platforms and model architectures. If you don't specify `--attention-backend`, SGLang makes a best effort to automatically select the most performant backend based on your hardware and model architecture. ## Support Matrix The support matrix is split into two parts: MHA (standard attention) and MLA (multi-head latent attention). For an explanation of the key differences between MHA and MLA, please see the [SGLang documentation on DeepSeek MLA](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-4-mla-optimizations) and the original [DeepSeek MLA paper](https://arxiv.org/pdf/2405.04434). ### MHA Backends
**Backend** **Page Size > 1 (native)** **FP8 KV Cache** **FP4 KV Cache** **Spec topk=1** **Spec topk>1** **Sliding Window** **MultiModal**
**FlashInfer** ✅ ✅ ❌ ✅ ✅ ✅ ❌
**FA3 (FlashAttention 3)** ✅ ✅ ❌ ✅ ✅ ✅ ✅
**FA4 (FlashAttention 4)** 128 ❌ ✅ ✅ ✅ ✅ ✅
**Triton** ❌ ✅ ✅ ✅ ✅ ✅ ✅
**Torch Native (SDPA)** ❌ ✅ ✅ ❌ ❌ ❌ ✅
**FlexAttention (PyTorch)** ❌ ❌ ✅ ❌ ❌ ❌ ❌
**TRTLLM MHA** 16, 32 or 64 ✅ ✅ ✅ ❌ ✅ ❌
**Dual Chunk FlashAttention** ✅ ❌ ❌ ❌ ❌ ❌ ❌
**HPC-Ops** 64 ✅ ❌ ❌ ❌ ❌ ❌
**AITER (ROCm)** ✅ ✅ ❌ ✅ ✅ ✅ ✅
**Wave (ROCm)** ✅ ❌ ❌ ❌ ❌ ❌ ❌
**Ascend (NPU)** ✅ ❌ ❌ ✅ ❌ ✅ ✅
**Intel XPU** ✅ ❌ ❌ ❌ ❌ ✅ ❌
**Intel AMX (CPU)** ❌ ❌ ❌ ❌ ❌ ❌ ❌
### MLA Backends
**Backend** **Native Page Sizes** **FP8 KV Cache** **FP4 KV Cache** **Chunked Prefix Cache** **Spec topk=1** **Spec topk>1**
**FlashInfer MLA** 1 ❌ ✅ ✅ ✅ ❌
**FlashMLA** 64 ✅ ✅ ✅ ✅ ❌
**Cutlass MLA** 128 ✅ ✅ ✅ ✅ ❌
**TRTLLM MLA (Blackwell)** 32 or 64 ✅ ✅ ✅ ✅ ❌
**CuteDSL MLA (Blackwell)** 32 or 64 ✅ ❌ ✅ ✅ ❌
**TokenSpeed MLA (Blackwell)** 32 or 64 ✅ (required) ❌ ✅ ✅ ❌
**FA3 (FlashAttention 3)** n/a ❌ ❌ ✅ ✅ ⚠️ (page\_size=1 only)
**Triton** n/a ❌ ❌ ❌ ✅ ⚠️ (page\_size=1 only)
**FA4** 1 ❌ ✅ ✅ ❌ ❌
**Ascend MLA (NPU)** 128 ❌ ❌ ❌ ❌ ❌
Multimodal attention is selected by `--mm-attention-backend`. The "MultiModal" column indicates whether a corresponding multimodal implementation exists for that backend family. * DSA is specifically designed for [DeepSeek V3.2](https://lmsys.org/blog/2025-09-29-deepseek-V32/). See the [DSA Attention Backend](#dsa-attention-backend) section and [DeepSeek V3.2 deployment guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2) for details. For the KV4 FA4 scenario, FA4 requires using a different --decode-attention-backend to run. Except for trtllm\_mha being incompatible with FA4, all other decode backends behave as shown in the table. Speculative decoding topk: `topk` is the number of draft tokens sampled per step from the draft model. `topk = 1` follows classic EAGLE; `topk > 1` explores multiple branches and requires backend support in both draft and verification paths. Page size controls how many tokens are grouped into a KV cache block. For the prefix cache to take effect, the number of tokens must fill at least one complete page. For example, if your prompt is only 32 tokens and `page_size = 64`, it won't fill a complete page and cannot be matched in the prefix cache (pages cannot be padded). With 65 tokens and `page_size = 64`, only the first page of 64 tokens will be cached and matched; the remaining 1 token is discarded. Use `page_size = 1` for maximum prefix reuse (token-level matching). Note that higher page sizes generally improve attention kernel performance, so prefer `page_size > 1` when prefix cache reuse is not critical. Many backends that do not natively operate on pages can emulate `page_size > 1` at the wrapper layer by expanding page tables to per-token indices. The "Page Size > 1 (native)" column indicates true in-kernel paging. Some backends require fixed native page sizes and cannot be reduced/emulated differently: TRTLLM MHA (16/32/64), TRTLLM MLA (32/64), CuteDSL MLA (32/64), FlashMLA (64), Cutlass MLA (128), Ascend (128), HPC-Ops (64). MLA page-size constraints: * FlashInfer MLA: page\_size = 1. * FlashMLA: page\_size = 64. * Cutlass MLA: page\_size = 128. * TRTLLM MLA: page\_size ∈ \{32, 64}. * CuteDSL MLA: page\_size ∈ \{32, 64} (decode-only; prefill falls back to `trtllm_mla` when unset). * TokenSpeed MLA: page\_size ∈ \{32, 64} (Blackwell SM100/SM12x only; requires `--kv-cache-dtype fp8_e4m3`). ### GDN Attention Backends GDN (Gated Delta Network) is a linear attention mechanism with O(n) complexity, used in hybrid models that alternate GDN linear attention layers with standard full attention layers. GDN is **not** selected via `--attention-backend`; it is automatically activated when the model architecture requires it (e.g., Qwen 3.5, Qwen 3 Next, Jet Nemotron, Jet VLM). The GDN linear attention layers have their own kernel backends, selected via `--linear-attn-backend` (default: `triton`). You can override the kernel per phase with `--linear-attn-decode-backend` and `--linear-attn-prefill-backend`. On SM100/SM103 with CUDA 13+, SGLang automatically selects FlashInfer for GDN prefill when the per-phase override is unset, the base linear-attention backend is Triton, recurrent state is BF16, key/value head dimensions are 128, dynamic chunking and page-major KV layout are disabled, and `--chunked-prefill-size` is between 1 and 8192. Radix caching may be disabled or use `no_buffer`, `extra_buffer`, or `extra_buffer_lazy`; the extra-buffer paths use state checkpoints.
Backend Decode Prefill / Extend Spec Decoding (Target Verify)
Triton (CUDA) ✅ ✅ ✅
Triton (AMD/ROCm) ✅ ✅ ✅
Triton (NPU) ✅ ✅ ❌
Triton (CPU) ✅ ✅ ❌
CuTe DSL (CUDA only) ✅ ❌ ❌
FlashInfer (CUDA, SM90/SM100/SM103) ✅ ✅ ✅ linear chain; tree falls back to Triton
GDN models are hybrid: the full-attention layers still require a standard `--attention-backend`. Platform constraints for the full-attention backend on hybrid GDN models: * **Blackwell SM120 (e.g., RTX PRO 6000 Blackwell)**: `triton` or `flashinfer` for prefill/full attention; `trtllm_mha` is supported for `--decode-attention-backend` only. * **Other Blackwell variants (including SM100 B200/GB200)**: `triton`, `trtllm_mha`, or `fa4` only. * **NPU (Ascend)**: `ascend` only. * **AMD (ROCm)**: `triton` recommended. * **Other CUDA (Hopper, Ampere, etc.)**: auto-selection works; no special constraints. ### DSA Attention Backend DSA (DeepSeek Sparse Attention) is a native sparse attention mechanism used by [DeepSeek V3.2](https://lmsys.org/blog/2025-09-29-deepseek-V32/). It is activated automatically when the model architecture requires it and is selected via `--attention-backend dsa` (deprecated alias: `nsa`). Internally, the DSA backend dispatches to different sub-backends for prefill and decode phases. You can override these with `--dsa-prefill-backend` and `--dsa-decode-backend`:
Sub-backend Prefill Decode Notes
flashmla\_sparse ✅ ✅ Default prefill on Hopper and Blackwell (BF16)
flashmla\_sparse\_q8 ✅ ❌ Native FP8 (q8×kv8) sparse prefill on Hopper (SM90); requires --kv-cache-dtype fp8\_e4m3
triton\_sparse\_mla ✅ ❌ Fused Triton sparse prefill that works on the post-TP head count instead of padding heads to 64/128; opt-in. CUDA SM90+ (tiles tuned on SM90 and SM120; other architectures run an untuned tile and log a warning at startup), at most 32 query heads per rank, not with index\_kpool. --dsa-triton-union 2|4 lets adjacent query tokens share one gathered index set (exact; num\_q\_heads \* union must be a power of two in \[16, 32])
flashmla\_kv ✅ ✅ Default for FP8 on Hopper (prefill + decode)
flashmla\_auto ✅ ❌ Picks flashmla\_sparse or flashmla\_kv by KV cache dtype
fa3 ✅ ✅ Default decode on Hopper (BF16)
trtllm ✅ ✅ Default decode on Blackwell (BF16); default for FP8 on Blackwell (prefill + decode)
tilelang ✅ ✅ Default on AMD (ROCm)
aiter ✅ ✅ AMD-specific kernel library (requires aiter package)
#### Gemma 4 on `aiter` (ROCm) Gemma 4's hybrid attention mixes sliding-window layers (`head_dim` 256) with full-attention layers (`head_dim` 512) and shares KV across layers. The `aiter` backend supports this and is allow-listed for Gemma 4, so you can serve it with `--attention-backend aiter` on MI300X/MI350X. Under the hood `aiter` routes the `head_dim`-512 prefill through the Triton `unified_attention` kernel (the CK batch-prefill path caps `head_dim` at 256), honors each layer's softmax scale, and resolves the sliding-window KV pool for speculative-decode draft workers. Non–sliding-window models (`head_dim` ≤ 256) are unaffected and keep using the CK path. For Gemma 4 on `aiter`, use bf16 KV cache (`--kv-cache-dtype auto`). Do **not** use `--kv-cache-dtype fp8_e4m3`: batched decode currently returns incorrect output under fp8 KV on this path (a pre-existing aiter fp8-decode issue, tracked separately). bf16 KV is correct at every batch size. For deployment examples, see the [DeepSeek V3.2 deployment guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2). ### Hybrid attention (different backends for prefill vs decode) (Experimental) Hybrid attention is an experimental feature. You can mix-and-match attention backends for prefill and decode. This is useful when one backend excels at prefill and another excels at decode. For the implementation details, please see `python/sglang/srt/layers/attention/hybrid_attn_backend.py`. ```bash Command theme={null} # Example: Prefill with FA4, Decode with TRTLLM MLA (Blackwell) python3 -m sglang.launch_server \ --model-path nvidia/DeepSeek-R1-FP4 \ --tp 8 \ --attention-backend trtllm_mla \ --moe-runner-backend flashinfer_trtllm \ --quantization modelopt_fp4 \ --prefill-attention-backend fa4 ``` #### Speculative decoding with hybrid attention Hybrid attention also works with speculative decoding. The backend used for draft decoding and target verification depends on `--speculative-attention-mode`: * `--speculative-attention-mode decode` (recommended): draft/verify use the decode backend. * `--speculative-attention-mode prefill` (default): draft/verify use the prefill backend. Constraints when combining hybrid attention with speculative decoding: * If any attention backend is `trtllm_mha`, speculative decoding supports only `--speculative-eagle-topk 1`. * For paged MHA backends with `--page-size > 1` and `--speculative-eagle-topk > 1`, only `flashinfer` is supported. * CUDA Graph: the decode backend is always captured; the prefill backend is captured only when `--speculative-attention-mode prefill`. If you set only one of `--prefill-attention-backend` or `--decode-attention-backend`, the unspecified phase inherits `--attention-backend`. If both are specified and differ, SGLang automatically enables a hybrid wrapper to dispatch to the chosen backend per phase. ## Attention Backend Selection Guide (CUDA) If the `--attention-backend` argument is not specified, SGLang automatically selects the best backend based on the hardware (CUDA) and model architecture. ### Automatic Selection Logic **1. MHA Models (e.g., Llama, Qwen)** * **Hopper (e.g., H100, H200)**: Defaults to `fa3` if using CUDA 12.3+ and the model configuration is supported. * **Blackwell (e.g., B200)**: Defaults to `trtllm_mha`, unless using speculative decoding with `topk > 1`. * **Other Architectures (Ampere, Ada, etc.)**: Defaults to `flashinfer` if available; otherwise falls back to `triton`. **2. MLA Models (e.g., DeepSeek V3)** * **Hopper**: Defaults to `fa3` (requires CUDA 12.3+). * **Blackwell**: Defaults to `flashinfer`; `trtllm_mla` is auto-selected for DeepSeek V3 models specifically. * **Other Architectures**: Defaults to `triton`. ## User Guide ### Launch Command for Different Attention Backends * FlashInfer (Default for Non-Hopper Machines, e.g., A100, A40) ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend flashinfer python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-V3 \ --attention-backend flashinfer \ --trust-remote-code ``` * FlashAttention 3 (Default for Hopper Machines, e.g., H100, H200, H20) ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend fa3 python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-V3 \ --trust-remote-code \ --attention-backend fa3 ``` * Triton ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend triton python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-V3 \ --attention-backend triton \ --trust-remote-code ``` * FlashMLA ```bash Command theme={null} python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-R1 \ --attention-backend flashmla \ --trust-remote-code python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-R1 \ --attention-backend flashmla \ --kv-cache-dtype fp8_e4m3 \ --trust-remote-code ``` * TRTLLM MLA (Optimized for Blackwell Architecture, e.g., B200) ```bash Command theme={null} python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-R1 \ --attention-backend trtllm_mla \ --trust-remote-code ``` * TRTLLM MLA with FP8 KV Cache (Higher concurrency, lower memory footprint) ```bash Command theme={null} python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-R1 \ --attention-backend trtllm_mla \ --kv-cache-dtype fp8_e4m3 \ --trust-remote-code ``` * TRTLLM MHA (Optimized for Blackwell Architecture, e.g., B200) ```bash Command theme={null} python3 -m sglang.launch_server \ --tp 4 \ --model Qwen/Qwen3.5-35B-A3B-FP8 \ --attention-backend trtllm_mha \ --trust-remote-code ``` * TRTLLM MHA (XQA backend) (Optimized for SM90 and SM120, e.g., H20, H200, 5090) Note that TRTLLM XQA backend only works well for pagesize 64. ```bash Command theme={null} python3 -m sglang.launch_server \ --tp 4 \ --model Qwen/Qwen3.5-35B-A3B-FP8 \ --decode-attention-backend trtllm_mha \ --trust-remote-code ``` * HPC-Ops (MHA kernels from [HPC-Ops](https://github.com/Tencent/hpc-ops) by the Tencent Hunyuan AI Infra team; Hopper (SM90) only, requires installing the `hpc` package from source, page size 64, bf16 or fp8\_e4m3 KV cache, head\_dim 128, q/kv head group 4 or 8) ```bash Command theme={null} python3 -m sglang.launch_server \ --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \ --attention-backend hpc_ops \ --page-size 64 \ --trust-remote-code # FP8 models should also set --kv-cache-dtype fp8_e4m3 to run the FP8 attention # kernels. This enables the fused QKNorm+RoPE+FP8-quant+StoreKV op, which is # currently wired for Hunyuan V3 and requires per-rank (q_heads, kv_heads) of # (64, 8) or (8, 1), e.g. --tp 1 or --tp 8 for Hy3. python3 -m sglang.launch_server \ --tp 8 \ --model tencent/Hy3-FP8 \ --attention-backend hpc_ops \ --kv-cache-dtype fp8_e4m3 \ --page-size 64 \ --trust-remote-code ``` * FlashAttention 4 (MHA & MLA) ```bash Command theme={null} # FA4 for both prefill and decode on SM90/SM100 python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \ --attention-backend fa4 \ --page-size 128 \ --trust-remote-code python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-R1 \ --prefill-attention-backend fa4 \ --trust-remote-code ``` * Ascend ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend ascend ``` * Intel XPU ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend intel_xpu ``` * Wave ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend wave ``` * FlexAttention ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend flex_attention ``` * Torch Native ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend torch_native ``` ## Steps to add a new attention backend To add a new attention backend, you can learn from the existing backends (`python/sglang/srt/layers/attention/triton_backend.py`, `python/sglang/srt/layers/attention/flashattention_backend.py`) and follow the steps below. Linear attention kernel backends (GDN, KDA) follow a different pattern. They implement `LinearAttnKernelBase` in `python/sglang/srt/layers/attention/linear/kernels/` and are dispatched by `GDNKernelDispatcher` / `KDAKernelDispatcher` rather than registered via `@register_attention_backend`. 1. Run without cuda graph. Support the two forward functions * forward\_extend * Will be used for prefill, prefill with KV cache, and target verification * It will be called once per layer * forward\_decode * Will be used for normal decode, and draft decode * It will be called once per layer * init\_forward\_metadata * Initialize the class and common metadata shared by all layers * Call the plan function for optimizations like split\_kv * It will be called once per forward 2. Run with cuda graph. It has two phases (capture and replay) and you need to implement three functions * init\_cuda\_graph\_state * It will be called once during life time * Create all common shared buffers * init\_forward\_metadata\_capture\_cuda\_graph * It will be called before capturing a cuda graph * It is similar to init\_forward\_metadata but write the medatada to some pre-defined buffers * init\_forward\_metadata\_replay\_cuda\_graph * It will be called before replaying a cuda graph * This function is in the critical path and needs to be fast # Breakable CUDA Graph Source: https://docs.sglang.io/docs/advanced_features/breakable_cuda_graph ## Motivation Standard CUDA graphs capture an entire forward pass as a single, opaque graph. This is great for performance, but creates two problems: 1. **Debugging is hard.** When something goes wrong inside a captured graph (wrong outputs, numerical mismatches, crashes), there is no way to step through the operations or insert print statements because the graph replays as a monolithic unit. 2. **Some ops are incompatible.** Certain operations — dynamic control flow, host-device synchronization, JIT compilation, or ops that change behavior across iterations — cannot be captured into a CUDA graph at all. Today, the only workaround is to disable CUDA graphs entirely, which sacrifices the kernel launch overhead savings for the rest of the model. **Breakable CUDA Graph** solves both problems by allowing graph breaks to be inserted at specific points. The computation is split into multiple captured graph segments with eager (non-graph) execution in between. This preserves most of the CUDA graph performance benefit while allowing targeted operations to run outside the graph. ## Usage ### Debug Mode: Run Everything Eagerly The simplest use case is debugging. The `--debug-cuda-graph` flag wraps the entire decode forward pass in a graph break, so every operation runs eagerly while still going through the full CUDA graph capture/replay code path. This lets you debug CUDA graph issues without changing model code. ```bash theme={null} python -m sglang.launch_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --debug-cuda-graph ``` This mode is intended for debugging only — it eliminates the performance benefit of CUDA graphs since every op runs eagerly. ### Selective Graph Breaks in Model Code For production use, you can mark specific functions as "non-graphable" using the `@eager_on_graph` decorator. During CUDA graph capture, these functions run eagerly between captured graph segments. Outside of capture, they behave normally. ```python theme={null} from sglang.srt.model_executor.runner_backend_utils.breakable_cuda_graph import eager_on_graph @eager_on_graph def my_dynamic_op(x): # This op is incompatible with CUDA graph capture return some_dynamic_operation(x) ``` You can also insert a bare graph break (no computation) using the `break_graph()` helper: ```python theme={null} from sglang.srt.model_executor.runner_backend_utils.breakable_cuda_graph import break_graph def forward(self, x): x = self.layer1(x) break_graph() # force a segment split here x = self.layer2(x) return x ``` To enable breakable CUDA graph at the environment level (without debug mode), set the environment variable: ```bash theme={null} export SGLANG_USE_BREAKABLE_CUDA_GRAPH=1 python -m sglang.launch_server \ --model meta-llama/Llama-3.1-8B-Instruct ``` ### Server Args
Argument Default Description
--debug-cuda-graph False Enable debug/eager mode. Wraps the entire forward pass in a graph break so every op runs eagerly through the capture/replay path.
SGLANG\_USE\_BREAKABLE\_CUDA\_GRAPH 0 Environment variable. Enables breakable CUDA graph without debug mode. Required for @eager\_on\_graph decorators to take effect.
## How It Works ### Capture Breakable CUDA graph extends PyTorch's `torch.cuda.CUDAGraph` by splitting a single capture into multiple segments separated by graph breaks. During capture, the flow is: ``` Begin capture (segment 1) ... graphable ops ... @eager_on_graph function encountered: 1. End current capture segment 2. Run the function eagerly (allocates output tensors) 3. Record the function for later replay 4. Begin new capture segment ... more graphable ops ... End capture (segment N) ``` Each segment is independently instantiated as a CUDA graph executable. The non-graph functions and their argument references are stored for replay. ### Replay During replay: ``` For each segment i: 1. Launch CUDA graph segment i 2. Run the recorded non-graph function i eagerly Launch final CUDA graph segment ``` The non-graph functions are re-invoked with the same tensor references as capture time. Since these references point to the CUDA graph's static input/output buffers, they see updated values on each replay. ### Output Writeback When a non-graph function produces output during replay, the result must be written back into the same tensor buffers that downstream graph segments reference. The mechanism handles: * **Plain tensors**: In-place `copy_()` into the original buffer. * **Structured outputs** (dataclasses, objects with tensor attributes): Tensor fields are copied in-place; non-tensor fields are replaced. * **Dicts of tensors**: Tensor values are copied in-place; non-tensor values are replaced. ### Stream Fork/Join Tracking Some models fork work onto secondary CUDA streams (e.g., for overlapped computation). Breakable CUDA graph hooks `torch.cuda.Stream.wait_stream` to track which streams are forked from the capture stream. When a graph break occurs, all forked streams are automatically joined back before ending the capture segment, and re-forked after beginning the next segment. ## Compatibility * **CUDA and ROCm/HIP.** Breakable CUDA graph runs on both NVIDIA and AMD GPUs. Other platforms (NPU, CPU, MPS, XPU) are unsupported; there `--debug-cuda-graph` is automatically disabled with a warning. * **Requires `cuda-python` on NVIDIA.** Stream-capture-status queries use the CUDA runtime via `cuda.bindings` (`pip install cuda-python`); the portable `torch.cuda.is_current_stream_capturing()` has proven unreliable on CUDA. On ROCm/HIP — where `cuda-python` is unavailable — the portable `torch.cuda` API (which maps to the HIP runtime) is used instead. * **Not compatible with memory saver mode.** Cannot be used together with `SGLANG_MEMORY_SAVER_CUDA_GRAPH`. ## Performance When no graph breaks are inserted, breakable CUDA graph has minimal overhead compared to standard CUDA graph — the capture/replay path is nearly identical. Each graph break adds: * One `cudaGraphLaunch` call (to replay the segment before the break) * One eager Python function call * One `cudaStreamBeginCapture` / `cudaStreamEndCapture` pair during capture For typical use cases with a small number of graph breaks, the overhead is negligible compared to the saved kernel launch overhead from the captured segments. ## Code Reference
File Description
python/sglang/srt/model\_executor/runner\_backend\_utils/breakable\_cuda\_graph/breakable\_cuda\_graph.py Core implementation: eager\_on\_graph, BreakableCUDAGraph, BreakableCUDAGraphCapture
python/sglang/srt/model\_executor/runner\_backend\_utils/breakable\_cuda\_graph/cuda\_utils.py CUDA runtime binding utilities (NVIDIA stream-capture queries)
python/sglang/srt/model\_executor/runner\_backend/breakable\_cuda\_graph\_backend.py Integration with CUDA graph runner backends
python/sglang/srt/server\_args.py --debug-cuda-graph flag and environment variable handling
python/sglang/srt/environ.py SGLANG\_USE\_BREAKABLE\_CUDA\_GRAPH environment variable definition
# Checkpoint Engine Integration Source: https://docs.sglang.io/docs/advanced_features/checkpoint_engine The SGLang checkpoint engine integration provides an efficient way to load model weights using a distributed checkpoint loading system. This feature significantly reduces model loading time, especially for large models and multi-node setups, by parallelizing the weight loading process across multiple processes and nodes. ## Overview The checkpoint engine integration allows SGLang to: * Load model weights in parallel using multiple processes * Distribute weight loading across multiple nodes to increase effective disk bandwidth * Overlap weight loading with other initialization tasks like CUDA graph capture * Support both single-node and multi-node deployments ## Installation First, install the checkpoint engine package: ```bash Command theme={null} pip install 'checkpoint-engine[p2p]' ``` ## Architecture The system consists of two main components: 1. **SGLang Server**: Runs with `--checkpoint-engine-wait-weights-before-ready` flag to wait for weights before becoming ready 2. **Checkpoint Engine Workers**: Separate processes (managed by torchrun) that load and distribute model weights The checkpoint engine uses a parameter server architecture with support for: * **Broadcast mode**: Weights are broadcast from loading processes to inference processes * **P2P mode**: Direct peer-to-peer weight transfer between processes * **All mode**: Combination of both broadcast and P2P methods ## Usage Examples ### Single Node Setup **Terminal 1 - Launch SGLang Server:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ --checkpoint-engine-wait-weights-before-ready ``` **Terminal 2 - Run Checkpoint Engine:** Using sglang entrypoint: ```bash Command theme={null} python -m sglang.srt.checkpoint_engine.update \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 8 ``` Using torchrun directly: ```bash Command theme={null} torchrun --nproc-per-node 8 \ examples/checkpoint_engine/update.py \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 8 ``` ### Multi-Node Setup (2 Nodes) **Node 0:** Launch SGLang server: ```bash Command theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ --checkpoint-engine-wait-weights-before-ready \ --host [IP] ``` Run checkpoint engine: Using sglang entrypoint (recommended): ```bash Command theme={null} python -m sglang.srt.checkpoint_engine.update \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 8 ``` Using torchrun directly: ```bash Command theme={null} torchrun --nproc-per-node 8 \ --nnodes 2 \ --node-rank 0 \ --master-addr [IP] \ --master-port 29500 \ examples/checkpoint_engine/update.py \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 8 ``` **Node 1:** Launch SGLang server: ```bash Command theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ --checkpoint-engine-wait-weights-before-ready \ --host [IP] ``` Run checkpoint engine: Using sglang entrypoint (recommended): ```bash Command theme={null} python -m sglang.srt.checkpoint_engine.update \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 8 ``` Using torchrun directly: ```bash Command theme={null} torchrun --nproc-per-node 8 \ --nnodes 2 \ --node-rank 1 \ --master-addr [IP] \ --master-port 29500 \ examples/checkpoint_engine/update.py \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 8 ``` ### Multi-Node Setup with Tensor Parallelism (TP=16) **Node 0:** Launch SGLang server: ```bash Command theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ --checkpoint-engine-wait-weights-before-ready \ --host [IP] \ --dist-init-addr [IP]:9120 \ --nnodes 2 \ --node-rank 0 ``` Run checkpoint engine: Using sglang entrypoint (recommended): ```bash Command theme={null} python -m sglang.srt.checkpoint_engine.update \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 16 ``` Using torchrun directly: ```bash Command theme={null} torchrun --nproc-per-node 8 \ --nnodes 2 \ --node-rank 0 \ --master-addr [IP] \ --master-port 29500 \ examples/checkpoint_engine/update.py \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 16 ``` **Node 1:** Launch SGLang server: ```bash Command theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ --checkpoint-engine-wait-weights-before-ready \ --host [IP] \ --dist-init-addr [IP]:9120 \ --nnodes 2 \ --node-rank 1 ``` Run checkpoint engine: Using sglang entrypoint (recommended): ```bash Command theme={null} python -m sglang.srt.checkpoint_engine.update \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 16 ``` Using torchrun directly: ```bash Command theme={null} torchrun --nproc-per-node 8 \ --nnodes 2 \ --node-rank 1 \ --master-addr [IP] \ --master-port 29500 \ examples/checkpoint_engine/update.py \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 16 ``` ## Configuration Options ### SGLang Server Options * `--load-format dummy`: Use dummy format for initial loading (allows overlapping with other tasks) * `--checkpoint-engine-wait-weights-before-ready`: Wait for checkpoint engine to provide weights before becoming ready * `--host`: Host address for multi-node setups * `--dist-init-addr`: Distributed initialization address for tensor parallelism ### Checkpoint Engine Options * `--update-method`: Weight update method (`broadcast`, `p2p`, or `all`) * `--checkpoint-path`: Path to model checkpoint directory * `--inference-parallel-size`: Number of inference parallel processes * `--endpoint`: SGLang server endpoint (default: `http://localhost:19730`) * `--checkpoint-name`: Name for the checkpoint (default: `my-checkpoint-iter-0`) * `--save-metas-file`: File to save checkpoint metadata * `--load-metas-file`: File to load checkpoint metadata from * `--uds`: Unix domain socket path for communication * `--weight-version`: Version identifier for weights ## Performance Benefits The checkpoint engine provides significant time savings in two main aspects: 1. **Multi-node Loading**: Each node only loads a portion of weights from disk, effectively increasing disk bandwidth. More participating nodes provide greater acceleration. Preliminary tests show 20-second acceleration when loading DeepSeek-R1 on H20-3e with two nodes. 2. **Single Process Optimization**: Using dummy format allows overlapping disk-to-CPU transfer with CUDA graph capture and other initialization tasks, providing additional time savings. ## Troubleshooting * Ensure checkpoint engine package is installed: `pip install 'checkpoint-engine[p2p]'` * Verify network connectivity between nodes in multi-node setups * Check that the checkpoint path contains valid model files * Monitor logs for connection errors between SGLang server and checkpoint engine * Use `--sleep-time` parameter to add delays if needed for debugging ## References * [Checkpoint Engine Repository](https://github.com/MoonshotAI/checkpoint-engine) # Cuda Graph for Multi-Modal Encoder in SGLang Source: https://docs.sglang.io/docs/advanced_features/cuda_graph_for_multi_modal_encoder ## Motivation In multimodal reasoning services, the visual encoder (ViT / Vision Transformer) typically has a few characteristic traits: Many layers, fragmented operators: Each layer includes LN, QKV projections, attention, MLP, residual connections, etc., resulting in extremely frequent kernel launches. Server-side “small batch / low latency” is common: The batch size is very small (sometimes it looks like 1 after “flattening” the batch), so kernel launch overhead accounts for a large portion of end-to-end latency. Input token count (number of patches) varies frequently: Different image/video resolutions and different batch composition lead to different sequence lengths S — and this is precisely the biggest obstacle for CUDA Graph (unstable shapes). The value of CUDA Graph: It captures a long sequence of GPU kernels with fixed shapes and fixed memory addresses into a graph; later, for the same shapes, it can replay the graph directly, dramatically reducing launch overhead and making GPU scheduling more compact. This led us to seek a CUDA Graph enabled feature for ViT in order to improve ViT performance. ## Design and Restrictions The new CUDA Graph enabled ViT logic is built on ViTCudaGraphRunner. This runner captures the "blocks + merger + deepstack merger (optional)" part of a vision transformer into a CUDA graph and replays it for identical shapes. See the following design consideration and restrictions for more details. ### Dynamic inputs to fit static constraints of CUDA Graph Variable sequence length S is very common in ViT. While CUDA Graph requires fixed shapes. The solution is to build a graph cache by S(e.g., graph\_key = S). The first time create a new S, and then capture a graph; afterwards, replay it. If there are many distinct S values, we need to increase VRAM usage which is graph-private memory pools for many graphs. ### Stable addresses Everything "parameter-like" becomes a static buffer: * block\_input / block\_ws / block\_output * cu\_full\_len / cu\_window\_len and their kk variants * sin\_cos\_ws In this way to solve the underlying requirement: during replay, not allowed to swap tensors, can only modify tensor contents. ### Attention backend arguments Attention backend arguments are fixed inside the graph: TritonAttn expects \[cu\_seqlens, cu\_seqlens\_kk, max\_len] FA3 expects \[cu\_seqlens, max\_len] max\_len is frozen as an int constant. cu\_seqlens is cached into a dict during create\_graph(), and its contents are not updated during subsequent replays. For the same graph\_key = S, you not only require the input shape to match, but also require the segmentation pattern in cu\_seqlens (and window seqlens) to be identical. Otherwise, attention will segment the sequence incorrectly. ### Rotary buffer management The feature reallocates a larger sin\_cos\_ws when seq\_len increases. The max\_content\_len is used to make sure the maximum size of the allocated rotary buffer. ## Command Example You can enable CUDA Graph for ViT by setting env variable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1`, for example: ```shell Command theme={null} SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \ python3 -m sglang.launch_server \ --model Qwen/Qwen3-VL-8B-Instruct ``` To run the ViT graph together with Piecewise CUDA Graph over the decoder's prefill, ask for the prefill backend explicitly. Qwen3-VL is on neither multimodal allowlist, so the default backend resolves prefill to `disabled` for it; naming a backend locks the choice and skips that rule: ```shell Command theme={null} SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \ python3 -m sglang.launch_server \ --model Qwen/Qwen3-VL-8B-Instruct \ --cuda-graph-backend-prefill breakable \ --cuda-graph-max-bs-prefill 4096 \ --cuda-graph-tc-compiler eager ``` ## Known supported models * Qwen2.5-VL ([https://github.com/sgl-project/sglang/pull/14422](https://github.com/sgl-project/sglang/pull/14422)) * Qwen3-VL ([https://github.com/sgl-project/sglang/pull/15320](https://github.com/sgl-project/sglang/pull/15320)) # Decode Context Parallelism Source: https://docs.sglang.io/docs/advanced_features/dcp Decode context parallelism (DCP) stripes a request's MLA KV cache across ranks in an existing tensor-parallel group. With DCP size `c`, rank `r` owns position `p` when `p mod c = r`, so each rank stores and reads roughly `1/c` of the cache. Local attention then sees only part of the context. The MLA path returns a partial output and log-sum-exp (LSE), exchanges both in one packed all-to-all, and merges them exactly. DCP complements TP and [attention data parallelism (DPA)](/docs/advanced_features/dp_dpa_smg_guide) rather than replacing them. The primary path is absorbed MLA decode and static target verification. `dcp_size=1` retains existing non-DCP behavior. On Kimi K3, DCP applies only to MLA layers. Request-indexed KDA state is unchanged, so DCP grows long-context KV capacity but does not raise the KDA concurrency ceiling. ## Enable DCP | Setting | Behavior | | - | - | | `--dcp-size N` | Enables DCP and widens virtual capacity and page size by `N`. Alias: `--decode-context-parallel-size`. | | `--dcp-comm-backend` | Selects the decode partial-output merge (MLA on CUDA/ROCm, MHA on Intel XPU): `ag_rs`, `a2a`, or `fi_a2a`. Unset resolves to `fi_a2a` where the DCP group shares an MNNVL domain on Blackwell, otherwise `a2a` on CUDA/ROCm/XPU. | | `--dcp-replicate-q-proj` | Removes the query all-gather for supported MLA weights. Use `--no-dcp-replicate-q-proj` to disable a model-specific default. | | `--attn-dp-size N` | Composes only when DCP groups nest inside attention TP. | A DCP group must fit inside one attention-TP group and one attention-DP replica: ```text Topology theme={null} attn_tp_size = tp_size / attn_dp_size attn_tp_size % dcp_size == 0 ``` For example, `TP=64, attention DP=4, DCP=16` creates four valid 16-rank attention replicas. `DCP=32` would cross replica boundaries and is invalid. Startup currently checks only `tp_size % dcp_size == 0`, not the stronger condition above. Until that check is fixed, DPA+DCP deployments must enforce containment when choosing the topology. MLA decode on DeepSeek-V3.1 with DCP 8 inside TP 8: ```bash Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-V3.1 \ --trust-remote-code \ --tp-size 8 \ --dcp-size 8 \ --host 0.0.0.0 \ --port 30000 ``` Kimi K3 on 32 GPUs with DPA and DCP 8. The model override enables replicated Q by default; the communication backend resolves generically to `fi_a2a` on MNNVL systems, otherwise `a2a`: ```bash Command theme={null} sglang serve \ --trust-remote-code \ --model-path moonshotai/Kimi-K3 \ --tp-size 32 \ --ep-size 32 \ --attn-dp-size 4 \ --dcp-size 8 \ --host 0.0.0.0 \ --port 30000 ``` See the [Kimi K3 cookbook](/cookbook/autoregressive/Moonshotai/Kimi-K3) for large-scale presets that combine DCP with DPA and expert parallelism. ## Why decode context parallelism? Absorbed MLA shares one latent KV representation across all query heads. TP can split the heads and weights, but not that cache, so every attention-TP rank stores and reads the same context. ```mermaid theme={null} flowchart LR subgraph TP["TP only: every rank stores the full MLA context"] direction TB T0["Rank 0
tokens 0 1 2 3 4 5 6 7"] T1["Rank 1
tokens 0 1 2 3 4 5 6 7"] T2["Rank 2
tokens 0 1 2 3 4 5 6 7"] T3["Rank 3
tokens 0 1 2 3 4 5 6 7"] end TP -- "stripe by position" --> DCP subgraph DCP["DCP4: one interleaved shard per rank"] direction TB D0["Rank 0
tokens 0 4"] D1["Rank 1
tokens 1 5"] D2["Rank 2
tokens 2 6"] D3["Rank 3
tokens 3 7"] end ``` DCP stripes that cache round-robin by logical token position, keeps a shared virtual index on every rank, runs attention against the owner-local shard, and merges partial outputs by LSE into the usual TP-local head layout. ## Architecture ### Virtual KV layout Let each rank have physical token capacity `C` and physical page size `P`. DCP exposes a shared virtual allocator: ```text Layout theme={null} virtual capacity = C * c virtual page size = P * c owner(v) = v mod c physical(v) = floor(v / c) ``` Every rank sees the same request-to-token map. On write, it keeps its positions and compacts them into the local physical pool. One widened virtual page maps to one physical page on each rank, so the layout stays balanced to within one token as a sequence grows. ### MLA decode dataflow ```mermaid theme={null} flowchart LR H["Hidden states"] --> Q["Full DCP-group query heads"] Q --> A0["Rank 0 local attention
partial O0 + LSE0"] Q --> A1["Rank 1 local attention
partial O1 + LSE1"] Q --> AN["Rank c-1 local attention
partial O + LSE"] A0 --> A2A["Packed all-to-all"] A1 --> A2A AN --> A2A A2A --> MERGE["Exact LSE merge"] MERGE --> OUT["TP-local head shard"] ``` For context partition `r`, the kernel returns locally normalized output `o_r` and `lse_r`. The global result is `lse = logsumexp_r(lse_r)` and `o = sum_r(exp(lse_r - lse) * o_r)`. This is exact aside from floating-point reduction order. The merge uses the attention backend's LSE base, either base-e or base-2. ### Query-projection replication Normally each layer all-gathers its absorbed and rotary query components. `--dcp-replicate-q-proj` gathers the query-projection and `w_kc` weights once at startup and computes the full DCP-group query locally. The trade is more weight memory and GEMM work for one fewer collective per layer. Only unquantized BF16/FP16 weights use this path; other layers fall back to the query all-gather. ### Communication backends | Backend | DCP collectives per MLA layer | Notes | | - | - | - | | `ag_rs` | Query AG + LSE AG + FP32 output RS | Fallback for platforms without the a2a path; opt in with `--dcp-comm-backend ag_rs` | | `a2a`, gathered Q | Query AG + packed NCCL A2A | Two collectives | | `a2a`, replicated Q | One packed NCCL A2A | Canonical low-latency path | | `fi_a2a`, replicated Q | One FlashInfer MNNVL A2A | Requires Blackwell and a DCP group inside one MNNVL domain (GB200/GB300 fabric, or a single node) | When `--dcp-comm-backend` is unset, every DCP launch resolves it the same way: `fi_a2a` where the DCP group shares one MNNVL domain on Blackwell, otherwise `a2a` on CUDA/ROCm/XPU, and `ag_rs` only on platforms without the a2a path. Query replication stays model-resolved: Kimi K3 enables it by default on the a2a family, and its decode backend is `cutedsl_mla`. DCP adds a context-independent decode collective while cutting context-dependent KV storage and reads by about `c`. Extend is outside that decode cost model: it may gather cached prefix shards, restore token order, and append new tokens, and that work grows with prefix length. ## Compositions ### DCP × speculative decoding The draft KV cache is replicated: every DCP rank stores the full draft context. DCP only stripes the target MLA KV, so it saves target cache, not draft cache. Target verify and decode use the DCP-aware MLA kernel `cutedsl_mla`. It: 1. Builds a rank-local page table from the cyclic token map (`owner(p) = p mod c`). 2. Runs attention against that shard, passing `cp_world`, `cp_rank`, and the global KV lengths so causality still sees the full sequence. 3. Returns a rank-local `(output, LSE)` that the usual DCP all-to-all merge combines. Draft steps use the same backend against the replicated draft cache, so they skip DCP metadata. Kimi K3 DCP selects `cutedsl_mla` automatically. ### DCP × PD disaggregation A dense prefill page cannot be copied directly into striped decode storage. The PD sender uses the decode rank's DCP metadata to select and pack the right rows. See [PD disaggregation](/docs/advanced_features/pd_disaggregation) for the transfer engines. * Equal DCP sizes require matching DCP ranks and use the existing page path. * DCP1 prefill to DCP decode uses token relayout for MLA and hybrid-MLA pools. * All other DCP-size transitions are rejected. The plan includes the exact token count, so stale rows do not leak out of a partial final page. Mooncake and NIXL use the same plan. KDA state keeps its attention-TP mapping and bypasses DCP filtering. This composition requires Mooncake or NIXL, matching physical page size and KV dtype, and prefill attention CP `1`. Decode radix cache and HiCache are not supported here. ### DCP × HiCache L2 HiCache keeps one widened logical page space across L1 and L2. The controller sees `P * c` slots, while each GPU and host buffer stores only its `P` local rows. See [HiCache](/docs/advanced_features/hicache) for the cache hierarchy. Before H2D or D2H, both index lists keep positions owned by this rank and map `i` to `floor(i / c)`. Transfers cover whole widened pages, and every rank moves its shard independently, with no DCP collective. For Kimi K3 this applies to MLA KV; KDA/Mamba state uses its own request-indexed host pool. The current scope is MLA L1/L2. L3, LMCache, HiSparse, non-MLA KV host pools, speculative decoding, and PD decode are not supported in this combination. ## References * [DCP and Helix Parallelism roadmap](https://github.com/sgl-project/sglang/issues/29736) * [SGLang and Miles Add Day-0 Support for Kimi K3](https://www.lmsys.org/blog/2026-07-27-kimi-k3-day0-support#decode-context-parallelism) * [Original SGLang DCP design](https://docs.google.com/document/d/1pjBSID0palpuI7rx8Ck7Z0BcyzBXAgnkDrf9uQg0uqU/edit?tab=t.0) # Deterministic Inference Source: https://docs.sglang.io/docs/advanced_features/deterministic_inference ## Why Deterministic Inference Matters Deterministic inference ensures consistent LLM outputs across runs, which is critical for: * **Reinforcement Learning**: Ensures consistent logprobs across runs, reducing stochastic noise and making RL training more stable, reproducible, and debuggable. * **Testing & Debugging**: Enables reproducible validation * **Production**: Improves reliability and user experience Even with `temperature=0`, standard LLM inference can produce different outputs due to dynamic batching and varying reduction orders in GPU kernels. ## The Root Cause of Non-Determinism The main source is **varying batch sizes**. Different batch sizes cause GPU kernels to split reduction operations differently, leading to different addition orders. Due to floating-point non-associativity (`(a + b) + c ≠ a + (b + c)`), this produces different results even for identical inputs. ## SGLang's Solution Building on [Thinking Machines Lab's batch-invariant operators](https://github.com/thinking-machines-lab/batch_invariant_ops), SGLang achieves fully deterministic inference while maintaining compatibility with chunked prefill, CUDA graphs, radix cache, and non-greedy sampling. The development roadmap for deterministic inference features can be found in this [issue](https://github.com/sgl-project/sglang/issues/10278). ### Supported Backends Deterministic inference is only supported with the following three attention backends: **FlashInfer**, **FlashAttention 3 (FA3)**, and **Triton**. The following table shows feature compatibility for deterministic inference across different attention backends:
Attention Backend CUDA Graph Chunked Prefill Radix Cache Non-greedy Sampling (Temp > 0)
**FlashInfer** ✅ Yes ✅ Yes ❌ No ✅ Yes
**FlashAttention 3 (FA3)** ✅ Yes ✅ Yes ✅ Yes ✅ Yes
**Triton** ✅ Yes ✅ Yes ✅ Yes ✅ Yes
## Usage ### Basic Usage Enable deterministic inference by adding the `--enable-deterministic-inference` flag: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --attention-backend fa3 \ --enable-deterministic-inference ``` ### Server Arguments
Argument Type/Default Description
`--enable-deterministic-inference` flag; default: disabled Enable deterministic inference with batch-invariant operations
`--attention-backend` string; default: fa3 Choose attention backend (flashinfer, fa3, or triton)
### Example Configurations #### Qwen3-8B ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --attention-backend flashinfer \ --enable-deterministic-inference ``` #### Llama Models ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --attention-backend fa3 \ --enable-deterministic-inference ``` #### Qwen3-30B-A3B (MoE Model) ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-30B-A3B \ --attention-backend fa3 \ --enable-deterministic-inference ``` ### Deterministic Inference with Non-Greedy Sampling (Temperature > 0) SGLang supports deterministic inference even with non-greedy sampling by using sampling seeds. This is particularly useful for reinforcement learning scenarios like GRPO (Group Relative Policy Optimization) where you need multiple diverse but reproducible responses. #### Default Behavior By default, SGLang uses a sampling seed of `42` for reproducible sampling: ```python Example theme={null} import requests response = requests.post( "http://localhost:30000/generate", json={ "text": "Tell me a joke", "sampling_params": { "temperature": 0.8, # Non-greedy sampling "max_new_tokens": 128, }, }, ) print(response.json()) # This will always produce the same response across runs ``` #### Generating Multiple Reproducible Responses To sample different responses from the same prompt while maintaining reproducibility (e.g., for GRPO training), provide different sampling seeds in your requests: ```python Example theme={null} import requests # Prepare a list of sampling seeds for different responses sampling_seeds = [42, 43, 44, 45, 46] responses = [] for seed in sampling_seeds: response = requests.post( "http://localhost:30000/generate", json={ "text": "Tell me a joke", "sampling_params": { "temperature": 0.8, "max_new_tokens": 128, "sampling_seed": seed, # Specify sampling seed }, }, ) responses.append(response.json()) # Each seed will produce a different but reproducible response # Using the same seed will always produce the same response ``` This approach ensures that: * Different seeds produce diverse responses * The same seed always produces the same response across different runs * Results are reproducible for debugging and evaluation ## Verification Run deterministic tests to verify consistent outputs: ```bash Command theme={null} # Single test: same prompt, varying batch sizes python3 -m sglang.test.test_deterministic --test-mode single --n-trials 50 # Prefix test: prompts with different prefix lengths python3 -m sglang.test.test_deterministic --test-mode prefix --n-trials 50 # Radix Cache Consistency mode: test radix cache determinism (cached vs uncached prefill) python3 -m sglang.test.test_deterministic --test-mode radix_cache ``` Expected result: All tests should show `Unique samples: 1` (perfectly deterministic). # DP, DPA and SGLang DP Router Source: https://docs.sglang.io/docs/advanced_features/dp_dpa_smg_guide This guide explains the difference between Data Parallelism (DP) and Data Parallelism Attention (DPA), how to enable each mode correctly, and how to use the SGLang Model Gateway (SMG) for production-grade DP deployments. ## Data Parallelism (DP) **Data Parallelism (DP)** is the most common parallelism strategy that replicates the entire model across multiple GPU sets and processes different batches of requests in parallel. Each GPU set handles independent requests. With dedicated routing strategies, as we will introduce later, with those proper routing algorithms in SGLang Model Gateway, the throughput of your serving system could be multiplied nearly linearly. ### Key characteristics * Each replica has a full copy of the model * Requests are distributed/scattered across replicas * No inter-replica communication during one request's inference (for simple DP) ## Data Parallelism Attention (DPA) **Data Parallelism Attention (DPA)**, also known as DP Attention, is an advanced parallelism strategy. While DPA provides the most significant benefits for **Multi-Head Latent Attention (MLA)** models (such as DeepSeek, MiniMax, Kimi-K2), it also supports **standard attention models** like Qwen. ### The Problem with Tensor Parallelism for MLA Models The most common parallelism strategy for inference is **Tensor Parallelism (TP)**. However, TP might not be the most efficient strategy for certain models. For example, DeepSeek models use MLA and only have **one KV head**. If we use tensor parallelism on 8 GPUs, it will lead to: * **Duplicated KV cache** across all GPUs * **Unwanted memory usage** that limits batch size * **Reduced throughput** due to memory constraints ### How DPA Works DPA addresses these limitations by applying **data parallelism specifically to the attention component**.
DPA + EP Architecture

Each DP replica:

  • Processes different batches independently (can be in different forward modes: prefill, decode, or idle)
  • Maintains its own KV cache (no duplication)
  • Enables significantly larger batch sizes due to memory savings

Communication patterns in DPA + EP:

  • All2All (Dispatch): Routes tokens to expert sub-groups based on gating decisions
  • All2All (Combine): Gathers computed results from experts back to original token positions
### Key benefits of DPA 1. **Significantly reduced KV cache memory**: Each DP replica only stores KV cache for its own batches 2. **Larger batch sizes**: Memory savings enable larger batch sizes 3. **Improved decoding throughput**: Significant throughput gains for MLA-based models 4. **Independent forward modes**: Each DP replica can be in different forward modes (prefill, decode, or idle) and handles its assigned batches independently during attention computation ### DPA with Expert Parallelism for MoE For MoE models like DeepSeek, DPA is **often** paired with Expert Parallelism (EP) for best throughput at scale. However, **DPA does not require EP**: you can enable DPA without EP if your deployment does not need expert sharding. * Distribute 256+ expert weights across GPUs (cannot fit on a single GPU) * Enable efficient all-to-all token routing via DeepEP * Scale to large clusters (up to 5x throughput improvement over vanilla TP) ### Recommended setup for DeepSeek ```bash theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3 \ --tp 8 \ --attn-dp-size 8 \ --ep 8 \ --moe-a2a-backend deepep \ --moe-runner-backend deep_gemm ``` > **Note**: `--attn-dp-size` sets the number of attention data-parallel groups inside the TP group. If `attn_dp_size` is 1 (default), DPA is disabled. For detailed EP configuration (DeepEP, Two-Batch Overlap, EPLB), see [Expert Parallelism](/docs/advanced_features/expert_parallelism). ### Target Models DPA supports the following model architectures: * **MLA (Multi-Head Latent Attention) models** - where DPA provides the most significant benefits: * DeepSeek family (DeepSeek-V2, DeepSeek-V3, DeepSeek-R1) * MiniMax models * Kimi-K2 * Other models using MLA architecture * **Standard attention models** - also supported: * Qwen models (see [PR #6121](https://github.com/sgl-project/sglang/pull/6121)) For models like Llama, with standard GQA, standard DP, or TP is typically recommended. To enable DPA, add `--attn-dp-size N` (N > 1) to your server launch command. ### Activation Logic DPA is enabled explicitly via server arguments (CLI or config) with `--attn-dp-size` (alias `--attention-data-parallel-size`). It creates N attention data-parallel groups inside the TP group, one scheduler each, while the FFN stays tensor parallel: ```bash theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3 \ --tp 8 \ --attn-dp-size 8 ``` **Important**: `--attn-dp-size` must be greater than 1 for DPA to work. When `attn_dp_size == 1` (default), DPA is disabled. The constraint `tp_size % attn_dp_size == 0` must also be satisfied. `--dp-size` only sets the number of data-parallel replicas of the model; combining replicas (`--dp-size` > 1) with DPA is not supported. > **Deprecated**: `--enable-dp-attention` still works: `--dp-size N --enable-dp-attention` resolves to `--attn-dp-size N --dp-size 1` and prints a deprecation warning. Use `--attn-dp-size N` instead. The flag will be removed after 2026-12-31. > > Under DeepSeek context parallelism (DSA, MLA, and DeepSeek-V4 prefill CP), `--dp-size N` alone still runs N attention-DP groups, and logs a deprecation warning; it will be rejected after 2026-12-31. Use `--attn-dp-size N` there too. **Elastic EP scale joiners.** A rank that joins an existing DPA deployment through elastic EP scale-up runs DPA even when it is launched with `--attn-dp-size 1`: its own attention DP group is one rank wide. **DP rank count.** The number of DP ranks (one scheduler each) is `dp_size × attn_dp_size`; one of the two is always 1. * `/server_info` reports `dp_size` (replicas) and `attn_dp_size` separately, so under DPA its `dp_size` is 1. Until the flag is removed, its deprecated `enable_dp_attention` field still reports whether DPA runs; new clients should read `attn_dp_size`. * Wire formats that predate `--attn-dp-size` keep the key `dp_size` for the DP rank count: KV-event descriptors, the PD prefill server info, and load snapshots. ### TP LMHead with all-to-all `dp-lm-head` keeps full weight and computes the local batch, making the gemm inefficient when batchsize per-rank is small. The existing `tp-lm-head` improves the gemm efficiency but introduce heavy allgather communication on full vocabulary. Therefore, all-to-all can be applied to eliminate the heavy allgather while retaining the gemm efficiency under small per-rank batchsize. Each rank will send and receive only the local rows with sharded vocab, which is more lightweight. This optimization is **enabled by default for decode node and pure DPA**. Users can use `--no-enable-tp-lm-head-all-to-all` to opt out, or `--enable-tp-lm-head-all-to-all` for force enabling. ```bash theme={null} python -m sglang.launch_server \ --model-path MODEL_PATH \ --disaggregation-mode decode \ --tp 8 \ --attn-dp-size 8 ``` Prefill / colocated workers will fallback to TP LMHead with allgather, as well as other parallel strategies. More details can be found in [PR #32313](https://github.com/sgl-project/sglang/pull/32313). ### Standard DP for MLA models Note that MLA models, of course, also support DP. Suppose you want to enable standard DP for MLA models. First, launch each MLA model's replica independently. You may launch these replicas one by one with DPA enabled. After launching each MLA model's replica, launch an SMG and connect all the replicas to the SMG. A detailed explanation of SMG is as follows. ## Modern Data Parallelism SGLang Model Gateway (SMG) ### Native DP Mode Native DP (built-in Data Parallelism) in SGLang creates multiple worker processes within a single SGLang instance, under the control of `DataParallelController` with the launching parameter of `dp-size`. ```bash theme={null} # Native DP mode python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --dp-size 4 ``` **Limitations:** * Built-in in-process load balancing only (e.g., `round_robin`, `total_requests`, `total_tokens`) * No cache-aware routing * Limited observability and metrics * No fault tolerance or circuit breakers * Not suitable for production workloads ⚠️ Native DP is **highly not recommended for use right now**. It is only used in some ancient/outdated RL frameworks. You can use SGLang Model Gateway (SMG) to power up your data parallelism in any use case. ### SMG-Based DP (Recommended) Starting from September 2024, SGLang Model Gateway, i.e., SMG, formerly named as SGLang DP Router, was built especially as a production-ready DP routing system with Rust. It starts from DP routing, but later we further expanded its scope to coordinate RL, PD Disaggregation, and other scenarios. This doc only discusses SMG's usage in DP routing. For other usage, please refer to [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway). > To achieve the best production-level routing performance and reduce the overhead to an extreme extent, we use Rust to build SMG, but not Python, since Python is never FAST enough. **We strongly recommend using the SGLang Model Gateway (SMG) for production-grade Data Parallelism.** SMG provides significant advantages over native DP mode. ```bash theme={null} # SMG-based DP mode (Recommended) python -m sglang_router.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --dp-size 4 ``` ⚠️ Note that **SMG and Naive DP share the same launching parameter, `--dp-size`**. But the entrypoint of Naive DP is `python -m sglang.launch_server`, and SMG's entrypoint is `python -m sglang_router.launch_server`. **Advantages of SMG-Based DP:**
Feature Native DP SMG-Based DP
Load Balancing Built-in in-process methods Advanced policies (cache-aware, power-of-two, etc.)
Cache Awareness ❌ No ✅ Yes - significantly higher cache hit rate
Throughput Baseline Significant improvement
Multi-Node Support Limited ✅ Full support
Worker Health Monitoring Basic ✅ Circuit breakers, health checks
Reliability Basic ✅ Retries, rate limiting, queuing
Observability Basic metrics ✅ 40+ Prometheus metrics, OpenTelemetry
Hot Worker Add/Remove ❌ No ✅ Yes
### SMG's Performance The cache-aware routing policy in SMG significantly improves performance for workloads with shared prefixes:
Metric Without Cache-Aware With Cache-Aware SMG
Throughput (token/s) 82,665 158,596 (+92%)
Cache Hit Rate 20% 75% (+275%)
*Benchmark from [SGLang v0.4 blog](https://lmsys.org/blog/2024-12-04-sglang-v0-4/), workload with multiple long prefix groups, 8x A100 80GB GPUs, dp-size=8* ### When to Use Each **Use Native DP when:** * ~~Never use Native/Naive DP~~ * Learning material of DP routing **Use SMG-Based DP when:** * In any case, when you think DP is needed * Production deployments * Multi-node distributed setups * Workloads with shared prefixes (high cache reuse potential) * You need high availability and reliability features * You require detailed observability and metrics * You want to have highly efficient RL rollout systems Note that for RL rollout systems, **there are four crucial reasons that SMG-Based DP is far better than naive DP routing**. Details can be found at [Load Balancing Router in RL](/docs/advanced_features/sglang_for_rl#load-balancing-router). ### Quick Start For SMG **Installation** ```bash theme={null} pip install sglang-router # or pip install "sglang[all]" ``` **Option A: Co-launch Workers and SMG (Simplest)** This is the easiest way to get started - SMG and workers are launched together: ```bash theme={null} python -m sglang_router.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --dp-size 4 \ --host 0.0.0.0 \ --port 30000 ``` **Option B: Separate Launch (Multi-Node)** For distributed deployments across multiple machines: 1. Launch workers on each node ```bash theme={null} # Node 1 python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --port 8000 # Node 2 python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --port 8000 ``` 2. Launch SMG pointing to workers ```bash theme={null} python -m sglang_router.launch_router \ --worker-urls http://node1:8000 http://node2:8000 \ --policy cache_aware \ --host 0.0.0.0 \ --port 30000 ``` **Option C: Dynamic Worker Registration** For elastic deployments where workers can be added/removed dynamically: ```bash theme={null} # Launch SMG first python -m sglang_router.launch_router \ --policy cache_aware \ --host 0.0.0.0 \ --port 30000 # Register workers dynamically curl -X POST http://localhost:30000/workers \ -H "Content-Type: application/json" \ -d '{"url": "http://worker1:8000"}' curl -X POST http://localhost:30000/workers \ -H "Content-Type: application/json" \ -d '{"url": "http://worker2:8000"}' ``` ### Load Balancing Policies SMG supports multiple load balancing policies:
Policy Description Best For
cache\_aware Combines cache locality with load balancing Recommended for most workloads
round\_robin Cycles through workers in order Simple, predictable distribution
random Random worker selection Baseline, testing
power\_of\_two Samples two workers, picks lighter one Low latency requirements
**Cache-Aware Policy (Default, Recommended)** The cache-aware policy provides the best performance for most workloads: ```bash theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 http://worker2:8000 \ --policy cache_aware \ --cache-threshold 0.5 \ --balance-abs-threshold 32 \ --balance-rel-threshold 1.5 \ --eviction-interval-secs 120 \ --max-tree-size 67108864 ``` **How it works:** 1. Maintains an approximate radix tree for each worker based on request history 2. Routes requests to workers with the highest prefix match (cache hit) 3. Falls back to shortest-queue routing when load is imbalanced 4. Automatically evicts old entries to prevent memory overflow ### Best Practices 1. **Start with `cache_aware` policy** - It provides the best balance between cache locality and load distribution for most workloads 2. **Use SMG for production** - Prefer `sglang_router.launch_server` over `sglang.launch_server` for better reliability and observability 3. **Enable health checks** - Configure `--router-health-check-interval-secs` to detect and remove unhealthy workers automatically **Recommended command with best practices applied:** ```bash theme={null} python -m sglang_router.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --dp-size 4 \ --router-policy cache_aware \ --router-health-check-interval-secs 30 \ --router-prometheus-port 10001 \ --host 0.0.0.0 \ --port 30000 ``` For advanced configuration (circuit breakers, retries, Prometheus metrics, K8s integration), see [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway). ### Verifying Traffic Distribution After launching SMG, verify that traffic is being distributed correctly: **1. Check worker status:** ```bash theme={null} curl http://localhost:30000/workers ``` **2. Check load distribution:** ```bash theme={null} curl http://localhost:30000/get_loads ``` **3. Monitor metrics (if Prometheus enabled):** ```bash theme={null} # Key metrics to check smg_router_requests_total{model="..."} smg_worker_requests_active{worker="..."} sglang_cache_hit_rate{source="..."} ``` For detailed metrics and monitoring setup, see [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway). ## Reference
Strategy Use Case Key Benefit
Native DP (--dp-size) Never Easy to understand, not rust based
SMG-Based DP Production (recommended) Cache-aware routing, high availability
DPA (--attn-dp-size N) DeepSeek/MLA models Eliminates KV cache duplication, improved throughput
DPA + EP DeepSeek MoE models Significant throughput improvement vs vanilla TP
**Recommended production setup for DeepSeek:** 1. Enable **DPA** for attention layers (`--attn-dp-size 8`) 2. Enable **EP** for MoE layers (`--ep 8 --moe-a2a-backend deepep`) 3. Use **SMG** with **cache\_aware** policy **Related documentation:** * [Expert Parallelism](./expert_parallelism) - DeepEP, Two-Batch Overlap, EPLB * [SGLang Model Gateway Documentation](./sgl_model_gateway) - SMG configuration & troubleshooting * [Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/) - 96 GPU deployment guide # DP for Multi-Modal Encoder in SGLang Source: https://docs.sglang.io/docs/advanced_features/dp_for_multi_modal_encoder A typical VLM architecture involves two main components: an multi-modal encoder and a text decoder. Most VLMs utilize a Vision Transformer (ViT) as their multi-modal encoder, it is responsible for processing visual data, extracting features (objects, colors, textures, etc.), and transforming them into a format that can be understood by the model. The text deocoder is based on LLM. It processes textual data and generates output based on the encoded visual features. However, since the size of ViT is very small compared to language decoders, there is relatively little gain from TP. On the other hand, TP incurs significant communication overhead because of all-reduce being performed after every layer. Placing the ViT in data parallel while keeping the LLM in tensor parallel consistently lowers TTFT and boosts end-to-end throughput. In this hybrid layout, the vision front-end becomes parallel and lightweight, while scarce interconnect bandwidth and collective ops are reserved for the LLM. Data parallelism replicates the entire model across multiple GPU sets and processes different batches of requests in parallel. ## Command Example You can enable batch-level DP by setting `mm-enable-dp-encoder`, for example: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen2.5-VL-7B-Instruct \ --tp 2 \ --mm-enable-dp-encoder ``` ## Known supported models * Qwen2.5-VL (\<[https://github.com/sgl-project/sglang/pull/13126](https://github.com/sgl-project/sglang/pull/13126)>) * Qwen3-VL (\<[https://github.com/sgl-project/sglang/pull/13724](https://github.com/sgl-project/sglang/pull/13724)>) * InternVL (\<[https://github.com/sgl-project/sglang/pull/13925](https://github.com/sgl-project/sglang/pull/13925)>) * GLM-4.5V & GLM-4.6V (\<[https://github.com/sgl-project/sglang/pull/14097](https://github.com/sgl-project/sglang/pull/14097)>) # EPD Disaggregation Source: https://docs.sglang.io/docs/advanced_features/epd_disaggregation ## Why and What is EPD Disaggregation? In modern Vision-Language Model (VLM) inference, request execution naturally decomposes into three distinct stages: Encoder, Prefill, and Decode. The Encoder stage performs vision preprocessing and ViT-based image encoding, which is highly compute-intensive but only required during request initialization. The Prefill stage processes the full multimodal input sequence to initialize the language model’s Key-Value (KV) cache, while the Decode stage is dominated by memory bandwidth and KV cache access for autoregressive token generation. Existing deployments typically colocate these stages within a unified execution engine, or at best apply Prefill–Decode (PD) disaggregation. However, such designs still tightly couple vision encoding with language prefill, leading to inefficient resource utilization, limited scalability for image-heavy workloads, and suboptimal scheduling under load. To address these challenges, we introduce Encoder–Prefill–Decode (EPD) Disaggregation in SGLang. EPD further separates vision encoding from language processing, enabling independent horizontal scaling of encoder servers, improved load balancing for multimodal requests, and seamless integration with existing PD disaggregation to form a fully decoupled three-tier inference architecture. ### Usage You can launch a language-only model using `--language-only`, or an encoder-only model using `--encoder-only`. When launching a language-only model, you must additionally specify the encoder service endpoints via `--encoder-urls`. We support multiple encoder transfer backends, including zmq\_to\_scheduler, zmq\_to\_tokenizer, and mooncake (the default is zmq\_to\_scheduler). The backend can be selected using `--encoder-transfer-backend`. ### Encoder transfer with Mooncake `--encoder-transfer-backend mooncake` controls **how encoder outputs are transferred** between encoder and language/prefill services. It is an encoder transfer option and can be used independently of the global multimodal embedding cache. Example: ```bash Command theme={null} # encoder python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --encoder-transfer-backend mooncake \ --port 30000 # language-only server python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --language-only \ --encoder-urls http://127.0.0.1:30000 \ --encoder-transfer-backend mooncake \ --port 30002 ``` ### Global multimodal embedding cache with Mooncake SGLang also supports a Mooncake-backed **global multimodal embedding cache** for EPD workloads. When enabled on encoder servers, repeated image inputs can reuse previously computed ViT embeddings across instances instead of running the vision encoder again. This feature is useful when: * the deployment serves repeated or overlapping image inputs, * encoder compute is the bottleneck, and * Mooncake is already available in the cluster. At a high level, the encoder checks whether the image embedding already exists in Mooncake. Cache hits are prefetched from the global store, while misses are encoded normally and inserted into the cache in the background. To enable it: * install and configure Mooncake in the same way as other SGLang Mooncake integrations, * add `--enable-mm-global-cache` on the encoder server. `--enable-mm-global-cache` controls **whether multimodal embeddings are looked up and stored in the global Mooncake cache**. It is separate from `--encoder-transfer-backend`, which only controls encoder output transport. For Mooncake deployment and configuration details, see [HiCache best practices](./hicache_best_practices#deployment-with-mooncake) and the [Mooncake backend README](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/mooncake_store/README.md). Example: ```bash Command theme={null} # Shared Mooncake configuration export MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata" export MOONCAKE_MASTER="127.0.0.1:50051" export MOONCAKE_PROTOCOL="rdma" export MOONCAKE_GLOBAL_SEGMENT_SIZE="4gb" # encoder with global multimodal cache enabled python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --enable-mm-global-cache \ --port 30000 # language-only server python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --language-only \ --encoder-urls http://127.0.0.1:30000 \ --port 30002 ``` Notes: * This cache is for **multimodal encoder embeddings**, not the language model KV cache. * The feature currently uses Mooncake as the shared backing store. * It can be enabled regardless of which `--encoder-transfer-backend` you use. * It is most relevant for EPD or encoder-disaggregated VLM deployments where the same images are likely to appear across requests or instances. #### Qwen VL * EP Disaggregation ```bash Command theme={null} # encoder 0 python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --encoder-transfer-backend zmq_to_scheduler \ --port 30000 # encoder 1 python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --encoder-transfer-backend zmq_to_scheduler \ --port 30001 # language-only server python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --language-only \ --encoder-urls http://127.0.0.1:30000 http://127.0.0.1:30001 \ --encoder-transfer-backend zmq_to_scheduler \ --port 30002 ``` * EPD Disaggregation ```bash Command theme={null} # encoder 0 python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --encoder-transfer-backend zmq_to_scheduler \ --port 30000 # encoder 1 python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --encoder-transfer-backend zmq_to_scheduler \ --port 30001 # prefill 0 python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --disaggregation-mode prefill \ --language-only \ --encoder-urls http://127.0.0.1:30000 http://127.0.0.1:30001 \ --encoder-transfer-backend zmq_to_scheduler \ --port 30002 # decode 0 python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --disaggregation-mode decode \ --port 30003 # router python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://$PREFILL_HOST:30002 \ --decode http://$DECODE_HOST:30003 \ --port 8000 ``` #### gRPC Encoder (EPD) You can run the encoder as a gRPC server while keeping prefill/decode as HTTP. When using gRPC encoders, set `SGLANG_ENCODER_MM_RECEIVER_MODE=grpc` for the prefill process so it uses the gRPC receiver. ```bash Command theme={null} # gRPC encoder python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --grpc-mode \ --encoder-transfer-backend zmq_to_scheduler \ --port 30000 # prefill (HTTP) - tell it to use gRPC receiver SGLANG_ENCODER_MM_RECEIVER_MODE=grpc \ python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --disaggregation-mode prefill \ --language-only \ --encoder-urls grpc://127.0.0.1:30000 \ --encoder-transfer-backend zmq_to_scheduler \ --port 30002 # decode (HTTP) python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --disaggregation-mode decode \ --port 30003 # router python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://$PREFILL_HOST:30002 \ --decode http://$DECODE_HOST:30003 \ --port 8000 ``` # Expert Parallelism Source: https://docs.sglang.io/docs/advanced_features/expert_parallelism Expert Parallelism (EP) in SGLang distributes expert weights across multiple devices in Mixture-of-Experts (MoE) models, addressing memory bottlenecks and enabling efficient scaling for high-performance inference. It is particularly vital for serving large-scale MoE models where tokens are dynamically routed to specialized experts across GPUs. By leveraging optimized all-to-all communication and grouped matrix multiplications (GEMMs), EP reduces latency, boosts throughput, and minimizes idle GPU time. SGLang's EP offers strong extensibility through its modular framework, allowing seamless integration of custom kernels, backends, and optimizations without refactoring core logic, supporting diverse hardware and quantization schemes. ## Supported Backends and Selection Guidance SGLang's EP integrates diverse, highly efficient backends for different use cases, allowing fine-grained control over performance trade-offs. Users specify backends via command-line flags: * `--moe-a2a-backend`: Selects the backend for all-to-all communication. * `--moe-runner-backend`: Selects the backend for MoE computation. ### Backends for All-to-All Communication
Backend Description Use Cases
**`none` (default)** Disables all-to-all for EP. Uses All-Reduce or All-Gather for token dispatch. Hybrid EP and TP setups.
`deepep` DeepEP, a communication library for efficient token shuffling in MoE models. Large-scale EP deployments.
`mooncake` An extension of DeepEP for elastic inference, leveraging RDMA for high-performance data transfers. Elastic EP serving.
nixl NIXL-EP, an elastic EP communication library built on NVIDIA's NIXL framework with native RDMA and NVLink support. Elastic EP serving with fault tolerance and dynamic scaling.
mori MORI-EP, AMD's native all-to-all communication implementation optimized for ROCm. AMD GPU deployments.
`flashinfer` Flashinfer implementation of all-to-all. Large-scale EP deployments.
`ascend_fuseep` Ascend NPU native fused all-to-all communication. Ascend NPU deployments.
`pplx` pplx-kernels, Perplexity's NVSHMEM-based all-to-all dispatch/combine kernels. Low-latency (masked) only; targets FP8 (DeepGEMM) MoE models on Hopper. Requires NVSHMEM 3.2.5, nvshmem4py, cuda-python, and a prebuilt libpplx\_kernels.so (sm\_90a). Low-latency decode EP on Hopper.
DeepEP and Mooncake backends support two modes for token dispatch: `normal` mode (optimized for prefill workloads with high throughput) and `low_latency` mode (optimized for decode workloads with low latency and CUDA Graph compatibility). MORI backend only supports `normal` mode now. NIXL-EP and PPLX currently operate in low-latency mode with CUDA Graph support (PPLX reuses the DeepEP low-latency masked expert-compute path). Users are recommended to set `--deepep-mode auto` to enable automatic dispatch mode switching during runtime. Setting `--deepep-mode normal` or `--deepep-mode low_latency` is useful for debugging or development purposes. Currently, DeepEP, Mooncake, NIXL-EP, `ascend_fuseep`, `pplx` and MORI only support cases where `ep_size = tp_size`. For hybrid EP and TP (i.e., `ep_size < tp_size`), only the `none` backend (All-Reduce or All-Gather-based dispatching) is supported. Note that `pplx` additionally requires DP attention with at least 2 attention DP groups (i.e., `--attn-dp-size` > 1, so `tp_size / attention_tp_size > 1`); otherwise pplx-kernels' AllToAll cannot be constructed. ### Backends for MoE Computation
Backend Description Use Cases
**`auto` (default)** Automatically selects the optimal backend based on model architecture, hardware (e.g., NVIDIA architecture like Ampere, Hopper, Blackwell), quantization scheme (e.g., FP8, FP4), and runtime conditions. General-purpose deployments; ensures compatibility and performance without user intervention.
`triton` Triton-based implementation for grouped GEMMs. To achieve higher performance, it's highly recommended to create tuned configurations. Custom kernel development or scenarios requiring high extensibility with Torch compilation support.
`deep_gemm` DeepGEMM backend optimized for MoE matrix multiplications, supporting contiguous layouts for prefill and masked layouts for decode; often JIT-compiled for performance. Large-scale EP deployments with FP8 block-wise quantization.
`cutlass` CUTLASS-based backend for efficient GEMMs. NVIDIA architectures with CUTLASS support.
`flashinfer_trtllm` FlashInfer integrated with TensorRT-LLM for accelerated MoE computations, supporting FP4 communication operators and high-performance GEMMs. Blackwell with TRT-LLM.
flashinfer\_trtllm\_routed FlashInfer integrated with TensorRT-LLM for accelerated routed MoE computations, consuming SGLang-computed top-k expert assignments and weights. Compatible with flashinfer all-to-all. Blackwell with TRT-LLM.
`flashinfer_cutlass` FlashInfer combined with CUTLASS for high-performance grouped GEMMs in MoE layers, handling FP4/FP8 quantization efficiently. Compatible with flashinfer all-to-all. Blackwell with FP4/FP8 models.
`flashinfer_mxfp4` FlashInfer variant optimized for MXFP4 (mixed FP4) quantization in MoE runners, focusing on memory-efficient low-precision inference. Low-precision models with MXFP4.
`flashinfer_cutedsl` FlashInfer with a custom DSL for flexible and efficient MoE kernel generation, integrated with ModelOpt FP4 quantization. Compatible with flashinfer all-to-all. Low-precision models with NVFP4.
### Examples Launch with DeepEP and DeepGEMM for DeepSeek-V3: ```bash Command theme={null} python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --moe-a2a-backend deepep --moe-runner-backend deep_gemm --tp 8 --ep 8 ``` ## Extensible EP Framework SGLang's EP framework provides modular abstractions for easy integration of custom kernels, backends, and optimizations. It decouples the MoE forward pass into stages (dispatch → pre-permute → core runner → post-permute → combine), enabling seamless extensions without refactoring core logic. ### Framework Overview The framework centers on `FusedMoE` as the unified entry point for a single, extensible structure. Key components include: * **Dispatcher**: Manages dispatch/combine for backends like DeepEP (implements `BaseDispatcher` subclasses). * **MoeRunner**: Orchestrates grouped-GEMM execution via `MoeRunnerCore` implementations (e.g., `TritonRunnerCore`). * **PermuteMethodPool**: Auto-registers layout conversions (e.g., pre/post-permute via `register_pre_permute` and `register_post_permute` for dynamic modes, or `register_fused_func` for static, torch.compile-compatible fused operations). * **TopK Router**: Backend-agnostic expert selection. This design supports multiple backends via `--moe-a2a-backend` and `--moe-runner-backend`, with quantization integrated through a standardized `apply()` method. The computation flow ensures modularity: ```text Output theme={null} [input_hidden_states] | v TopK.forward -> select_experts / triton_kernels.routing / bypass | v [TopKOutput] | v FusedMoE.forward -> Dispatcher.dispatch -> DeepEP / bypass | | | v | [DispatchOutput] | | | v | quant_method.apply -> MoeRunner.forward | | | | | v | | pre-permute + grouped_gemm + post-permute | | | | |-------------- | v | [CombineInput] | | | v | Dispatcher.combine -> DeepEP / bypass | | |--------------------- v [final_hidden_states] ``` For details, see the [MoE Refactor Roadmap](https://github.com/sgl-project/sglang/issues/8715). ### Implementing New Backends To add a new backend: 1. For a new all-to-all dispatcher, implement a `BaseDispatcher` subclass with `dispatch` and `combine` methods. 2. For a new MoE runner backend, define a `MoeRunnerCore` subclass for core operations (e.g., grouped GEMMs). 3. Define new input/output formats for the dispatcher or model runner (e.g., `RunnerInput`, `RunnerOutput`). 4. Register permute/unpermute methods to ensure compatibility: * **Fused Mode** (static, torch.compile-compatible): Use `register_fused_func` for end-to-end operations. * **Permute Mode** (dynamic): Register `register_pre_permute` and `register_post_permute` for flexible layouts. See the [MoE Refactor Implementation PR](https://github.com/sgl-project/sglang/pull/9269) for full changes, including type hints and config expansions. ### Examples For an example implementation, see [moe\_runner/triton.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/layers/moe/moe_runner/triton.py), which demonstrates Triton-based grouped GEMMs with registered fused and permutation functions. ## Computation and Communication Overlap SGLang's EP employs advanced overlap techniques to hide communication latency behind computation, maximizing GPU utilization in MoE layers. ### Two-Batch Overlap (TBO) TBO splits requests into micro-batches, interleaving attention computation with dispatch/combine operations. Yield points in the execution graph allow pausing for overlaps, increasing overall throughput without peak memory spikes: ```python Example theme={null} operations = [ self._forward_attn, YieldOperation(), # Overlap with dispatch of prior micro-batch self._forward_dispatch, self._forward_mlp, YieldOperation(), # Overlap with combine self._forward_combine, ] ``` Users need to specify `--enable-two-batch-overlap` to unlock up to 2x throughput. For details, see the [Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/#two-batch-overlap). ### Single-Batch Overlap (SBO) SGLang introduces a dispatcher-hook system for Single-Batch Overlap (SBO), enabling the overlap of operations within a single batch—such as shared experts computation with communication—while decentralizing logic to enhance modularity. These hooks execute before and after the `dispatch` and `combine` operations without modifying core MoE modules. This design simplifies interfaces, reduces coupling, and improves extensibility. For implementation details and an example of overlapping shared experts with DeepEP's combine operation, refer to [PR #13327](https://github.com/sgl-project/sglang/pull/13327). Users can set `--enable-single-batch-overlap` to enable this feature. ## Workload Balancer SGLang integrates the [Expert Parallelism Load Balancer (EPLB)](https://github.com/deepseek-ai/EPLB) from DeepSeek to address routing imbalances in MoE models. By analyzing expert activation statistics, EPLB computes an optimal expert arrangement, strategically placing or replicating experts to minimize GPU utilization variance, reduce idle cycles, and enhance scalability. To enable EPLB, use the flags `--enable-eplb`. For optimal performance, increase batch sizes to stabilize activation statistics and configure periodic rebalancing (e.g., every 1000 requests) to adapt to evolving workloads. Simulations demonstrate significant improvements in load balancedness (ratio of mean to max computation time), correlating strongly with throughput gains. For more details, refer to the [EPLB Section in the Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/#expert-parallelism-load-balancer) and the [EPLB Repository](https://github.com/deepseek-ai/eplb). ## Ascend NPU Guidance ### Guidance on SGLang configuration in Ascend NPU * `--moe-a2a-backend` only supports `deepep` and `ascend_fuseep` backends, * `deepep`: The mechanism is consistent with the above description. * `ascend_fuseep`: Offer a large fused operator which integrates all operations between dispatch and combine to boost MoE computation. Only used for decode stage in PD Disaggregation Mode. * `--moe-runner-backend` parameter does not need to be configured. * `--deepep-mode`: * In PD mixed mode, please set `--deepep-mode auto`. * In PD Disaggregation Mode, prefill instance sets `--deepep-mode normal`, and decode instance sets `--deepep-mode low_latency`. ### DeepEP Ascend Introduction DeepEP Ascend is the adapted version of the DeepEP communication library for Huawei Ascend NPUs, specifically designed for Mixture-of-Experts (MoE) model Expert Parallelism (EP). It supports the Ant-moving Function (Split the sequence length into rounds for streaming batch transmission) to optimize the buffer size occupied during collective communication in prefill stage, especially for long sequences. Ant-moving Function can be enabled for both the dispatch and combine phases via the following environment variables: * `DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS`: Enable ant-moving function in dispatch stage. Indicates the number of tokens transmitted per round on each rank, default 8192. * `DEEPEP_NORMAL_LONG_SEQ_ROUND`: Enable ant-moving function in dispatch stage. Indicates the number of rounds transmitted on each rank, default 1. * `DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ`: Enable ant-moving function in combine stage, default 0 (means disabled). `DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS * DEEPEP_NORMAL_LONG_SEQ_ROUND` means input sequence length. When the input sequence length exceeds 8192, it is recommended to enable the ant-moving function in both dispatch and combine phase. The environment variable `HCCL_BUFFSIZE` is used to configure the buffer size (MB) actually allocated. Its calculation formula is as follows: ```text Output theme={null} # Enable Ant-moving Function HCCL_BUFFSIZE >= 2 * (102MB + 4MB + DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS * (hidden_size + hidden_size + hidden_size) * topk) + PADDING_BUFFSIZE # Disable Ant-moving Function HCCL_BUFFSIZE >= 2 * (102MB + 4MB + TOTAL_SEQ_LEN * (hidden_size + hidden_size) * topk) + PADDING_BUFFSIZE ``` Wherein the parameters are described as follows: * `hidden_size`: hidden size in model config. * `topk`: The number of selected routing experts. * `TOTAL_SEQ_LEN`: input sequence length. * `PADDING_BUFFSIZE`: A value of 20 or greater is recommended. # Hierarchical KV Caching (HiCache) Source: https://docs.sglang.io/docs/advanced_features/hicache * [Hicache Best Practices](./hicache_best_practices) * [Hicache Design](./hicache_design) * [Hicache Storage Runtime Attach Detach](./hicache_storage_runtime_attach_detach) # SGLang HiCache Best Practices Source: https://docs.sglang.io/docs/advanced_features/hicache_best_practices ## Why HiCache Matters SGLang HiCache extends the traditional RadixAttention with a three-tier hierarchical KV caching system that dramatically improves performance for long-context and multi-turn conversation scenarios. By intelligently managing KV caches across GPU memory, host memory, and external storage backends, HiCache addresses the fundamental capacity bottleneck that limits cache hit rates in conventional systems. L1 and L2 are private to a single inference instance; only L3 can be shared. Host memory cannot be pooled across instances or across hosts, not even for two instances on the same node. Raising `--hicache-ratio` or `--hicache-size` only enlarges the instance's own private L2. Cross-instance reuse is the job of L3, which needs `--hicache-storage-backend` and a backend configured for the scope you want: the `file` backend defaults to the node-local `/tmp/hicache`, whereas `mooncake`, `hf3fs`, `nixl` and `aibrix` reach cluster scope when every instance shares the same namespace. See [Tier Sharing Scope](/docs/advanced_features/hicache_design#tier-sharing-scope). ## Configuration Guidelines ## Core HiCache Parameters ```bash Command theme={null} # Essential HiCache flags --page-size 64 # Page size for cache management --enable-hierarchical-cache # Enable HiCache --hicache-ratio 2 # Host memory ratio (2x GPU memory) --hicache-size 100 # Host memory size in GBs, will override the above ratio --hicache-io-backend kernel # The I/O backend of moving data between CPU and GPU --hicache-write-policy write_through # Cache write policy from GPU to CPU --hicache-storage-backend # Optional storage backend (e.g., hf3fs, mooncake, etc.) ``` Notes: * Besides configuring `--hicache-storage-backend` at startup, SGLang also supports **runtime attach/detach** of the HiCache storage backend (no restart required) via HTTP admin endpoints. See [Runtime Attach/Detach HiCache Storage Backend](./hicache_storage_runtime_attach_detach). ## Key Configurations with Storage Backends Enabled ### Memory Layout Optimization ```bash Command theme={null} # Page-first: Optimized for I/O efficiency with zero-copy (recommended with kernel backend) --hicache-mem-layout page_first # Page-first-direct: Optimized for direct I/O operations (Compatible with fa3 and same zero-copy performance as page_first) --hicache-mem-layout page_first_direct # Layer-first --hicache-mem-layout layer_first ``` **Layout Compatibility:** * `page_first`: Only compatible with `kernel` I/O backend, automatically switches to `layer_first` with `direct` backend * `page_first_direct`: Specifically designed for `direct` I/O backend with optimized memory organization ### Heterogeneous TP Support (GQA/MHA models) HiCache storage supports cross-cluster KV reuse when different deployments use different TP sizes (for example, `tp=4` and `tp=8`) and share the same storage backend namespace. Use `tp_lcm_size` in `--hicache-storage-backend-extra-config`: ```bash Command theme={null} # Example: heterogeneous TP = {4, 8}, so lcm = 8 --hicache-storage-backend-extra-config '{"tp_lcm_size": 8}' ``` Guidelines: * Set `tp_lcm_size` to the least common multiple (LCM) of all TP sizes that will share the same HiCache storage. * For MHA models with Mooncake and `page_head` layout, HiCache will split head shards based on `tp_lcm_size` to make keys reusable across heterogeneous TP deployments. * If all clusters use the same TP size, this option is not needed. ### Prefetch Policies ```bash Command theme={null} # Best-effort: Terminate prefetch when needed --hicache-storage-prefetch-policy best_effort # Wait-complete: Ensure complete prefetch, higher cache reuse --hicache-storage-prefetch-policy wait_complete # Timeout: Balance between completion and best-effort --hicache-storage-prefetch-policy timeout ``` ### Integration with PD Disaggregation HiCache works seamlessly with PD Disaggregation. You can choose between two configurations: 1. **Prefill-only HiCache**: Enable HiCache only on Prefill nodes, allowing KV cache sharing among Prefill instances 2. **Full HiCache with async offloading**: Enable HiCache on Prefill nodes and async KV cache offloading on Decode nodes, allowing Prefill nodes to reuse KV caches from Decode nodes in multi-turn dialogue scenarios ```bash Command theme={null} # Prefill node with HiCache enabled for cross-prefill sharing (ideal for SystemPrompt scenarios) python3 -m sglang.launch_server \ --model-path /xxx/DeepSeek-R1/ \ --tp 8 \ --host 0.0.0.0 \ --port 10000 \ --enable-metrics \ --enable-cache-report \ --mem-fraction-static 0.85 \ --page-size 64 \ --enable-hierarchical-cache \ --hicache-ratio 2 \ --hicache-size 0 \ --hicache-mem-layout page_first_direct \ --hicache-io-backend direct \ --hicache-write-policy write_through \ --hicache-storage-backend hf3fs \ --hicache-storage-prefetch-policy wait_complete \ --disaggregation-ib-device mlx5_0 \ --disaggregation-mode prefill \ --disaggregation-transfer-backend mooncake # Decode node with async offloading enabled for KV cache reuse by Prefill (ideal for multi-turn conversations) python3 -m sglang.launch_server \ --model-path /xxx/DeepSeek-R1/ \ --tp 8 \ --host 0.0.0.0 \ --port 10000 \ --enable-metrics \ --enable-cache-report \ --page-size 64 \ --hicache-ratio 2 \ --hicache-size 0 \ --hicache-mem-layout page_first_direct \ --hicache-io-backend direct \ --hicache-write-policy write_through \ --hicache-storage-backend hf3fs \ --hicache-storage-prefetch-policy wait_complete \ --disaggregation-decode-enable-offload-kvcache \ # Enable async KV cache offloading in decode node --disaggregation-ib-device mlx5_0 \ --disaggregation-mode decode \ --disaggregation-transfer-backend mooncake ``` ### Deployment with HF3FS Here is an example of deploying DeepSeek-R1 with HiCache-HF3FS. For more details, see the [HF3FS Documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/hf3fs/docs/README.md). ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path /xxx/DeepSeek-R1/ \ --log-level info \ --tp 8 \ --host 0.0.0.0 \ --port 10000 \ --enable-metrics \ --enable-cache-report \ --page-size 64 \ --mem-fraction-static 0.85 \ --enable-hierarchical-cache \ --hicache-ratio 2 \ --hicache-size 0 \ --hicache-mem-layout page_first_direct \ --hicache-io-backend direct \ --hicache-write-policy write_through \ --hicache-storage-backend hf3fs \ --hicache-storage-prefetch-policy wait_complete \ ``` ### Deployment with Mooncake Here is an example of deploying Qwen3-235B-A22B-Instruct-2507 with Mooncake. For more details, see the [Mooncake Documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/mooncake_store/README.md). ```bash Command theme={null} # Set Mooncake environment variables export MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata" export MOONCAKE_GLOBAL_SEGMENT_SIZE=816043786240 export MOONCAKE_PROTOCOL="rdma" export MOONCAKE_DEVICE="$DEVICE_LIST" export MOONCAKE_MASTER=127.0.0.1:50051 # Launch SGLang server with Mooncake backend python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --tp 8 \ --page-size 64 \ --enable-hierarchical-cache \ --hicache-ratio 2 \ --hicache-mem-layout page_first_direct \ --hicache-io-backend direct \ --hicache-storage-backend mooncake \ --hicache-write-policy write_through \ --hicache-storage-prefetch-policy timeout ``` ## Custom Storage Backend Integration To integrate a new storage backend: 1. **Implement three core methods:** * `get(key)`: Retrieve value by key * `exists(key)`: Check key existence * `set(key, value)`: Store key-value pair 2. **Register your backend:** Add your storage backend to the HiCache [BackendFactory](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/backend_factory.py#L188) The HiCache controller handles all scheduling and synchronization automatically. ### Dynamic Backend Loading Alternatively, you can use dynamic loading to avoid hard-coding your backend in the repository: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path your-model \ --enable-hierarchical-cache \ --hicache-storage-backend dynamic \ --hicache-storage-backend-extra-config '{"backend_name":"custom_backend_name", "module_path": "your_module_path", "class_name": "YourHiCacheClassName"}' ``` **Configuration Parameters:** * `--hicache-storage-backend`: Set to `dynamic` * `--hicache-storage-backend-extra-config`: JSON configuration with: * `backend_name`: Custom backend identifier * `module_path`: Python module path to your implementation * `class_name`: Your HiCache implementation class name * `interface_v1`: 0 (disable) or 1 (enable) to control usage of batch\_get\_v1 and batch\_set\_v1 methods ## Community and Support * **GitHub Issues**: Report bugs and feature requests * **Slack Channel**: Join community discussions in #sgl-kv-cache-store * **Documentation**: Refer to storage backend-specific guides *** *This document will be continuously updated based on community feedback and new features. Contributions and suggestions are welcome!* # HiCache System Design and Optimization Source: https://docs.sglang.io/docs/advanced_features/hicache_design This document provides a comprehensive overview of SGLang HiCache, covering its system architecture, workflow and key components. It also details configuration parameters, optimization techniques, and integration with various L3 storage backends, serving as a complete reference for users and developers to understand and tune HiCache for efficient LLM inference. ## Why and What is HiCache? In large language model inference, the prefill phase is often time-consuming: input sequences need to be first converted into Key-Value cache (KV cache) for subsequent decoding. When multiple requests share the same prefix, the KV cache for that prefix is identical. By caching and reusing these shared KV caches, redundant computation can be avoided. To address this, SGLang introduced RadixAttention, which leverages idle GPU memory to cache and reuse prefix KV caches, and **HiCache**, which extends this idea to host memory and distributed storage. Inspired by the classic three-level cache design of modern CPUs, HiCache organizes GPU memory as L1, host memory as L2, and distributed storage as L3. This hierarchy enables HiCache to fully exploit the "idle" storage space of GPUs and CPUs, while integrating distributed cache systems such as Mooncake, 3FS, NIXL, and AIBrix KVCache for global KV cache storage and scheduling. As a result, HiCache significantly expands KV cache capacity while maintaining strong read performance—especially in workloads such as multi-QA and long-context inference, where KV cache reuse is frequent. For detailed benchmark results, see [this blog](https://lmsys.org/blog/2025-09-10-sglang-hicache/). ## System Design ### Overall Architecture In many modern CPU architectures, the small but fast L1 and L2 caches are private to each core, enabling rapid access to the hottest data, while the larger L3 cache is shared across all cores to significantly reduce redundancy within the cache. Similarly, in HiCache, the L1 and L2 KV caches are private to each inference instance, whereas the L3 KV cache is shared among all inference instances within the cluster. ### Tier Sharing Scope Which tier is private and which is shared decides where a cache hit can come from, so it is worth stating explicitly: | Tier | Medium | Scope | Shared across instances? | | - | - | - | - | | **L1** | GPU memory (HBM) | One inference instance | No | | **L2** | Host memory (CPU DRAM) | One inference instance, on its own node | **No** | | **L3** | Storage backend (`file`, `mooncake`, `hf3fs`, `nixl`, `aibrix`, or a custom one) | Whatever the backend is configured to span | **Only if the backend is set up for it** | ```mermaid theme={null} flowchart TB subgraph N0["Node A"] subgraph I0["Instance 0"] L1A["L1 - GPU memory
private"] L2A["L2 - host memory
private to this instance"] L1A --- L2A end subgraph I1["Instance 1"] L1B["L1 - GPU memory
private"] L2B["L2 - host memory
private to this instance"] L1B --- L2B end end subgraph N1["Node B"] subgraph I2["Instance 2"] L1C["L1 - GPU memory
private"] L2C["L2 - host memory
private to this instance"] L1C --- L2C end end L3["L3 - storage backend
shared by the instances the backend is configured to span"] L2A --> L3 L2B --> L3 L2C --> L3 ``` **L2 is node-local and instance-private.** It is host memory owned by one inference instance's process, so two instances never read each other's L2, not even two instances on the same node. A KV cache produced by instance 0 becomes visible to instance 1 only after it reaches L3. This is the answer to a common question: HiCache cannot pool the host memory of several machines into one larger L2. Growing `--hicache-ratio` or `--hicache-size` only makes each instance's own private L2 larger. Cross-instance reuse is the job of L3, so it needs `--hicache-storage-backend`. L3 is the tier that *can* be shared, but whether it actually is depends on the backend and how it is configured. The `file` backend writes to `/tmp/hicache` by default (`SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR` overrides it), which is node-local unless that path is a shared mount. Distributed backends such as `mooncake`, `hf3fs`, `nixl` and `aibrix` give cluster-wide scope, but only when every instance is pointed at the same namespace and configuration. ### HiRadixTree: Metadata Organization in HiCache For KV cache data organization, HiCache builds upon the RadixTree structure introduced in RadixAttention and proposes HiRadixTree. In RadixAttention, each node of the RadixTree corresponds to the KV cache of a consecutive span of tokens in GPU memory. A path from the root to a leaf node represents the prefix of a request, and shared prefixes across multiple requests can reuse the same nodes, thereby avoiding redundant storage. HiRadixTree extends this idea: each node corresponds to the KV cache of a span of consecutive tokens and records where that KV cache is stored—whether in local GPU memory, CPU memory, L3 storage, or multiple of these tiers. If stored locally, HiRadixTree maintains precise metadata, including the exact storage address. However, to reduce overhead, HiRadixTree does not store or continuously synchronize metadata for L3 KV cache. Instead, when accessing L3 data, it queries the backend in real time to retrieve the necessary metadata, such as whether the data exists and on which server and location it resides. ### Overall Workflow The workflow of HiCache mainly involves three key operations: **local match**, **prefetch** and **write-back**. When the system receives a new request, it first searches the local L1 and L2 caches for matching KV caches. For parts not found locally, it attempts to prefetch from L3. After prefetching, all required KV caches are loaded into the GPU for computation. Once the prefill computation is complete, the system considers storing the newly generated data into L2 or L3. HiCache Workflow ### Local Match Local matching is the first step in HiCache's workflow, where incoming request tokens are matched against the HiRadixTree to locate cached KV data in local memory tiers (L1 GPU memory and L2 host memory). The matching algorithm traverses the HiRadixTree from the root node, following child nodes that match the token sequence prefix. At each node, the incoming token sequence is compared with the node’s stored token sequence. When `page_size > 1`, matching is performed at the page granularity to optimize memory access patterns. If a match terminates within a node’s stored sequence, the node is automatically split to create an exact boundary, improving the efficiency of future matches. The algorithm returns a continuous prefix of the request, with the first part residing in L1 and the latter part in L2. Since the process only requires traversing the local HiRadixTree and does not involve any actual data copying, local matching is extremely fast. ### Prefetch from L3 Data prefetching is one of HiCache’s core optimization techniques, designed to proactively load KV caches from L3 storage into local L2 memory, thereby reducing access latency during subsequent operations. **Prefetch Trigger Conditions**: After local matching, for the parts not found in L1 or L2, the system queries L3 to retrieve metadata for the next continuous matching KV caches. If the length of hit cache in L3 exceeds a threshold (default: 256 tokens, configurable), a prefetch operation is triggered. **Prefetch Strategies**: HiCache provides three different prefetch termination strategies to address different scenario needs: * **best\_effort**: Terminates immediately when GPU can execute prefill computation, with no waiting time, suitable for scenarios extremely sensitive to latency. * **wait\_complete**: Must wait for all prefetch operations to complete, suitable for scenarios requiring high cache hit rates. * **timeout**: Terminates after specified time or when complete, balancing latency and cache hit rate needs. After prefetching stops, the data already fetched is used together with the local data for the prefill computation. For **timeout** strategy, HiCache introduces three configuration parameters to support fine-grained control over prefetch timeout conditions: * `prefetch_timeout_base`: the base timeout, representing overhead unrelated to the number of tokens (e.g., scheduling and synchronization). Default: `2` seconds. * `prefetch_timeout_per_ki_token`: the incremental timeout per thousand tokens. Default: `0.1` seconds per 1024 tokens. * `prefetch_timeout_max`: the upper bound applied to the linear timeout, preventing very long prompts from waiting unboundedly. Default: `30` seconds. The timeout is computed as: ```python Example theme={null} timeout = min( prefetch_timeout_max, prefetch_timeout_base + prefetch_timeout_per_ki_token * num_token_to_fetch / 1024, ) ``` ### Data Write-back The write-back mechanism is responsible for moving frequently accessed KV caches from L1 to L2 and L3, enabling larger and longer-term storage as well as cache sharing across instances. **Configurable Write-back Policies**: HiCache supports three write-back strategies: * **write\_through**: Every access is immediately written back to the next level. When bandwidth is sufficient, this strategy provides the strongest caching benefit. * **write\_through\_selective**: Data is written back only after the access frequency exceeds a threshold. This strategy backs up only hot data, reducing I/O overhead. * **write\_back**: Data is written back to the next level only when it is evicted from the upper level. This strategy alleviates storage pressure and is suitable for scenarios where storage capacity is limited but memory utilization must be maximized. **Cross-instance Sharing**: When data is written back from L2 to L3, only data not already present in L3 is transferred. KV caches stored in L3 can then be shared across all SGLang instances in the cluster (depending on the L3 backend implementation), significantly improving cache hit rates within the same memory budget. ### Multi-Rank Synchronization During multi-GPU parallel computation, such as tensor parallelism (TP), HiCache must ensure consistent states across different ranks. Therefore, critical computation steps require the use of `all_reduce` for state synchronization. For example, during prefetching, `all_reduce(op=min)` is used to ensure that all ranks obtain the same number of L3 hits, preventing inconsistent judgments about whether the prefetch threshold has been reached. Similarly, after prefetching completes or terminates, `all_reduce(op=min)` is again required to guarantee consensus among ranks on the prefix length of the successfully retrieved KV cache. ### Data Transfer Optimization **Zero-Copy Data Transfers**: Both prefetching and write-back involve substantial data movement. Minimizing the number of data copies can significantly improve system performance. HiCache supports passing memory addresses and sizes directly when transferring data from L2 memory to an L3 backend. **“Batch-Oriented” Data Organization**: The granularity of data reads and writes has a major impact on performance. To address this, HiCache L3 stores and transfers KV cache data at the granularity of **pages** and supports different data layouts beyond the existing `layer first` scheme, including `page first` and `page first direct`. Under the `page first` and `page first direct` layouts, all KV cache data belonging to the same page is placed in contiguous memory, allowing it to be passed as a single object to L3 using zero-copy transfers. HiCache L2 MEM layout However, because GPU KV computation is naturally performed layer by layer, the GPU inherently operates in a `layer first` layout. When transferring `page first` data from L2 to the GPU, data must be transferred at the granularity of one token per layer. The `page first direct` layout mitigates this issue by grouping together all tokens of a given layer within a page, allowing transfers from L2 to GPU to be aggregated at the page-layer level. **CPU-to-GPU Transfer Optimizations**: In HiCache, moving data from CPU memory to GPU is as performance-critical as prefetching data from L3 to L2. HiCache employs several optimizations for this process: * **Compute-Transfer Overlap**: During the prefill phase, when transferring data from CPU to GPU, HiCache overlaps layers by concurrently loading the KV cache of layer N+1 while computing layer N. This effectively hides data transfer latency. * **GPU-assisted I/O Kernels**: On top of `cudaMemcpyAsync`, HiCache implements a set of GPU-assisted I/O kernels specifically optimized for KV cache transfers between CPU and GPU. Compared to the baseline approach, these kernels achieve up to 3x higher transfer speed. **Write-back Optimization for MLA**: For MHA (Multi-Head Attention) models under multi-TP, each rank holds `1/tp_size` of a token’s KV data. In contrast, for MLA (Multi-Layer Attention) models, all ranks hold the complete and identical KV data for each token. HiCache includes a dedicated optimization for MLA: only one rank initiates the write-back operation, ensuring that data is not redundantly stored across ranks. ### Integration with PD-Disaggregation Deployment Mode SGLang supports a PD (Prefill-Decode) disaggregation deployment mode through the Mooncake TransferEngine (for details, see [this doc](./pd_disaggregation)). In the PD-disaggregation deployment mode, HiCache can be enabled on both the prefill nodes and decode nodes to optimize prefill performance. If enabled on decode nodes, the decode output will also be written back to L3. ### Unified Interfaces and Rich L3 Storage Backends HiCache encapsulates all read, write, and query operations on L3 backends within the `class HiCacheStorage(ABC)`, exposing a set of simple and consistent interfaces. This design supports a wide range of L3 storage backends and allows users to select the one that best fits their specific use cases. * **Mooncake**: Mooncake is a high-performance caching system for LLM inference that leverages RDMA and multi-NIC resources to enable zero-copy, ultra-fast data transfers. Try Mooncake [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/mooncake_store). * **DeepSeek 3FS (HF3FS)**: HF3FS is a Kubernetes-native distributed storage solution with operator-based deployment. Try HF3FS [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/hf3fs). * **NIXL**: NIXL provides a unified API for accessing various storage plugins, including but not limited to DeepSeek's 3FS, GPU Direct Storage (GDS) and Amazon S3-compatible object storage. Try NIXL [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/nixl). * **AIBrix KVCache**: AIBrix KVCache is a production-ready KVCache Offloading Framework, which enables efficient memory tiering and low-overhead cross-engine reuse. Try AIBrix KVCache [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/aibrix_kvcache). * **HiCacheFile**: A simple file-based storage backend for demonstration purposes. Specifically, **LMCache**, an efficient KV cache layer for enterprise-scale LLM inference, provides an alternative solution to HiCache. Try LMCache [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/lmcache). ## Related Parameters * **`--enable-hierarchical-cache`**: Enable hierarchical cache functionality. This is required to use HiCache. * **`--hicache-ratio HICACHE_RATIO`**: The ratio of the size of host KV cache memory pool to the size of device pool. For example, a value of 2 means the host memory pool is twice as large as the device memory pool. The value of this parameter must be greater than 1, as the current implementation requires the host memory allocated for the KV cache to be larger than the device memory allocated for the KV cache. * **`--hicache-size HICACHE_SIZE`**: The size of host KV cache memory pool in gigabytes. This parameter overrides `hicache-ratio` if set. For example, `--hicache-size 30` allocates 30GB (1GB = 1e9 bytes) for the host memory pool **for each rank**. If there are 8 ranks, then the total memory size is 240GB. Just like `hicache-ratio`, the value of this parameter must be larger than the size of device memory allocated for KV cache. **Note**: `--hicache-ratio` and `--hicache-size` are two critical parameters. In general, a larger HiCache size leads to a higher cache hit rate, which improves prefill performance. However, the relationship between cache size and hit rate is not linear. Once most reusable KV data—especially hot tokens—are already cached, further increasing the size may yield only marginal performance gains. Users can set these parameters based on their workload characteristics and performance requirements. * **`--page-size PAGE_SIZE`**: The number of tokens per page. This parameter determines the granularity of KV cache storage and retrieval. Larger page sizes reduce metadata overhead and improve I/O efficiency for storage backends, but may lower the cache hit rate when only part of a page matches the stored KV cache. For workloads with long common prefixes, larger pages can improve performance, while workloads with more diverse prefixes may benefit from smaller pages. See [Data Transfer Optimization](#data-transfer-optimization) for how page granularity affects I/O performance. * **`--hicache-storage-prefetch-policy {best_effort,wait_complete,timeout}`**: Controls when prefetching from storage should stop. See [Prefetch from L3](#prefetch-from-l3) for details. * `best_effort`: Prefetch as much as possible without blocking * `wait_complete`: Wait for prefetch to complete before proceeding * `timeout`: Terminates after specified time or when complete (Recommended for production environments, as setting an appropriate timeout helps the system meet required SLOs) * **`--hicache-write-policy {write_back,write_through,write_through_selective}`**: Controls how data is written from faster to slower memory tiers. See [Data Write-back](#data-write-back) for details. * `write_through`: Immediately writes data to all tiers (strongest caching benefits) * `write_through_selective`: Uses hit-count tracking to back up only frequently accessed data * `write_back`: Writes data back to slower tiers only when eviction is needed (reduces I/O load) * **`--hicache-io-backend {direct,kernel}`**: Choose the I/O backend for KV cache transfer between CPU and GPU. See [Data Transfer Optimization](#data-transfer-optimization) for details. * `direct`: Standard CUDA memory copy operations * `kernel`: GPU-assisted I/O kernels (recommended for better performance) * **`--hicache-mem-layout {layer_first,page_first,page_first_direct}`**: Memory layout for the host memory pool. See [Data Transfer Optimization](#data-transfer-optimization) for details. * `layer_first`: Compatible with GPU computation kernels (default for GPU memory) * `page_first`: Optimized for I/O efficiency * `page_first_direct`: Groups all tokens of a given layer within a page, allowing transfers from L2 to GPU to be aggregated at the page-layer level * **`--hicache-storage-backend {file,mooncake,hf3fs,nixl,aibrix,dynamic}`**: Choose the storage backend for the L3 tier. Built-in backends: file, mooncake, hf3fs, nixl, aibrix. For dynamic backend, use --hicache-storage-backend-extra-config to specify: `backend_name` (custom name), `module_path` (Python module path), `class_name` (backend class name). See [Unified Interfaces and Rich L3 Storage Backends](#unified-interfaces-and-rich-l3-storage-backends) for available backends. * **`--enable-lmcache`**: Using LMCache as an alternative hierarchical cache solution. * **`--lmcache-config-file`**: Path to the LMCache YAML configuration file. * **`--hicache-storage-backend-extra-config HICACHE_STORAGE_BACKEND_EXTRA_CONFIG`**: the extra config can be either * a JSON string containing extra configuration for the storage backend, e.g., `--hicache-storage-backend-extra-config '{"prefetch_threshold":512, "prefetch_timeout_base": 0.5, "prefetch_timeout_per_ki_token": 0.25}' `, or * a TOML or JSON or YAML file specifying the extra configuration for the storage backend (to differentiate from the JSON string input, prepend a `@` in front of the file name), e.g., `--hicache-storage-backend-extra-config "@config.toml"` where `config.toml` is the config file containing the complex configurations. This can be useful when the configuration consists of many or complex key-value pairs (for instance, it is preferred to use a config file for NIXL backend as its configurations can be complex). # Runtime Attach/Detach HiCache Storage Backend (No Restart) Source: https://docs.sglang.io/docs/advanced_features/hicache_storage_runtime_attach_detach This document explains how to **dynamically attach/detach the HiCache L3 storage backend at runtime** (e.g., `mooncake` / `hf3fs` / `nixl` / `file` / `aibrix` / `eic`) while **SGLang is already running and serving traffic**, without restarting the process. For safety and consistency, the current implementation **strictly requires** these operations to happen only when the service is **idle**: * **No running requests** * **No waiting/queued requests** If the idle condition is not met, the API will fail fast (HTTP 400) and **will not modify** the current service state. *** ## 1. Background and implementation overview ### 1.1 Architecture / control path The control path is: 1. **HTTP Server** (`python/sglang/srt/entrypoints/http_server.py`) * Exposes `PUT /hicache/storage-backend`, `DELETE /hicache/storage-backend`, `GET /hicache/storage-backend` 2. **TokenizerManager** (`python/sglang/srt/managers/tokenizer_control_mixin.py`) * Sends the request to the Scheduler via `FanOutCommunicator` 3. **Scheduler** (`python/sglang/srt/managers/scheduler.py`) * Performs a **strict idle check** * Calls `tree_cache.attach_storage_backend(...)` / `detach_storage_backend(...)` 4. **UnifiedRadixCache** (`python/sglang/srt/mem_cache/unified_radix_cache.py`) * Parses `hicache_storage_backend_extra_config_json` (supports both backend config and prefetch knobs) * Calls `cache_controller.attach_storage_backend(...)` / `detach_storage_backend(...)` 5. **HiCacheController** (`python/sglang/srt/managers/cache_controller.py`) * Creates/destroys the storage backend instance (via `StorageBackendFactory`) * Starts/stops backend background threads at runtime (prefetch/backup) *** ## 2. Idle-state requirement (strict) The Scheduler uses `is_fully_idle()` which checks: * No running batches (including chunked prefill, overlap, pipeline-parallel, and disaggregation paths) * No waiting requests in any queue (waiting, grammar, disagg bootstrap/prealloc/transfer/inflight) * No DLLM staging requests If the condition is not met, attach/detach returns an error like: * `Reject attach: scheduler is not idle. #queue-req=... #running-req=...` before switching, drain upstream traffic and wait for the server to become idle, then call attach/detach. ### 2.1 DP (data parallel) semantics When `dp_size > 1`, the tokenizer dispatches the request to **all DP scheduler instances** and aggregates their responses: * The final `success` is **true only if all DP ranks return success** * The final `message` concatenates messages from all DP ranks This is intended to prevent “silent partial success”, but it also means you may see: * Overall **failure** even though **some ranks already succeeded** Currently there is **no automatic partial rollback** across DP ranks (see TODO in code). Operationally: * Prefer to keep backend config identical across ranks * If attach fails, immediately call detach (best-effort/idempotent), fix config, then retry attach *** ## 3. How to use (HTTP Admin API) The examples below assume your SGLang HTTP server is at `http://127.0.0.1:30000`. ### 3.1 Query current storage backend status ```bash Command theme={null} curl -s http://127.0.0.1:30000/hicache/storage-backend ``` Example response: ```json Config theme={null} { "hicache_storage_backend": "mooncake", "hicache_storage_backend_extra_config": "{\"master_server_address\":\"127.0.0.1:50051\", ...}" } ``` ### 3.2 Attach (enable) a storage backend ```bash Command theme={null} curl -s -X PUT http://127.0.0.1:30000/hicache/storage-backend \ -H 'Content-Type: application/json' \ -d '{ "hicache_storage_backend": "mooncake" }' ``` ```bash Command theme={null} curl -s -X PUT http://127.0.0.1:30000/hicache/storage-backend \ -H 'Content-Type: application/json' \ -d '{ "hicache_storage_backend": "mooncake", "hicache_storage_backend_extra_config_json": "{\"master_server_address\":\"127.0.0.1:50051\",\"protocol\":\"tcp\",\"global_segment_size\":\"4gb\",\"prefetch_threshold\":256}", "hicache_storage_prefetch_policy": "timeout" }' ``` Notes: * `hicache_storage_backend_extra_config_json` can include both: * **Backend configuration** (e.g., Mooncake master/metadata/protocol, etc.) * **Prefetch configuration** (`prefetch_threshold`, `prefetch_timeout_base`, `prefetch_timeout_per_ki_token`, `prefetch_timeout_max`, `hicache_storage_pass_prefix_keys`) ### 3.3 Detach (disable) the storage backend ```bash Command theme={null} curl -s -X DELETE http://127.0.0.1:30000/hicache/storage-backend ``` Notes: * Detach only makes SGLang **stop using** the L3 storage backend and stops prefetch/backup threads * It **does not automatically delete** data stored in Mooncake/HF3FS (or other remote backends) *** ## 4. Behavior and caveats * **No restart required**: attach/detach switches in-process at runtime * **Must be idle**: otherwise the request is rejected to avoid consistency issues * **Host KV layout constraints still apply**: for example, Mooncake still requires layouts like `page_first/page_first_direct/page_head`; if the server's HiCache host-memory layout does not satisfy the backend requirements, attach will fail with an error * **Observability**: * After attach, `server_args.hicache_storage_backend*` is updated on both the tokenizer and scheduler sides * If metrics are enabled, attach will create a storage metrics collector in `UnifiedRadixCache` on demand # HiSparse: Hierarchical Sparse Attention Source: https://docs.sglang.io/docs/advanced_features/hisparse_guide HiSparse reduces per-request GPU memory consumption during the decode phase by maintaining only a small "hot" KV buffer on GPU while keeping complete KV data in CPU pinned memory. Combined with PD disaggregation, it enables significantly higher decode concurrency. > **Prerequisites**: HiSparse works with models that use **DeepSeek Sparse Attention (DSA)** architectures (e.g., DeepSeek-V3.2, GLM-5.1), **DeepSeek V4**, and **MiniMax M3**. These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires **PD disaggregation mode** and is enabled on the **decode instance** only. ## Why HiSparse? In long-context LLM inference, each decoding request holds a full-length KV cache on GPU, limiting the number of concurrent requests a decode instance can serve. HiSparse addresses this by: * **Reducing GPU memory per request**: Each request occupies only a fixed-size device buffer (e.g., 4KB tokens) instead of the full sequence length. * **On-demand swap-in**: A CUDA kernel dynamically loads the top-k most relevant KV entries from host memory based on attention scores. * **Transparent to prefill**: HiSparse is entirely a decode-side optimization; the prefill instance requires no changes. ## Design Overview ### Decode Workflow Each decode step follows this flow: 1. **Forward decode** — generate the next token 2. **Top-k selection** — select the most relevant token positions via attention scores 3. **Swap-in** — the CUDA kernel loads top-k KV entries from host to device buffer: * *Short sequences* (`seq_len ≤ device_buffer_size`): fast path, all KV already in buffer * *Long sequences*: hit detection → LRU reordering → miss handling (host → device copy) 4. **Decode attention** — compute attention using the top-k device locations 5. **Eager backup** — asynchronously copy the previous token's KV from device to host ### PD Disaggregation Integration (Direct-to-Host) In PD disaggregation mode, the prefill instance transfers KV cache directly into the decode instance's host pool via RDMA, bypassing the GPU entirely on the decode side. This eliminates the transient GPU memory spike during KV transfer and removes the staging DMA step. ``` Prefill GPU ──RDMA──▶ Decode Host Pool (CPU pinned memory) │ ▼ alloc device buffer (4KB) │ ▼ swap-in kernel (on-demand top-k) ``` For DeepSeek V4, the direct-to-host path writes only C4 KV into the decode host pool. The c4\_indexer and C128 KV remain device-to-device transfers. ## Server Arguments
Argument Type / Default Description
--enable-hisparse flag; default: disabled Enable HiSparse on the decode instance
--hisparse-config JSON string Configuration for HiSparse (see below)
### HiSparse Config Parameters Pass as a JSON string via `--hisparse-config`:
Parameter Type / Default Description
top\_k int Number of topk entries
device\_buffer\_size int Number of token slots in the per-request GPU device buffer
host\_to\_device\_ratio int Ratio of logical pool size to device pool size, determining host memory capacity
swap\_in\_block\_size int / 960 CUDA thread-block size for the HiSparse swap-in kernel
Example: `--hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10, "swap_in_block_size": 960}'` ### Shared-index prefetch (automatic) When a model reuses one anchor layer's top-k selection across a run of subsequent "skip" layers (DSA `index_topk_freq` / `index_topk_pattern`; native in GLM-5.2 as IndexShare), the working set of every skip layer is known the moment the anchor's index is computed. HiSparse exploits this automatically: the anchor's swap-in kernel records its miss plan (which host slots go to which device-buffer slots), and each skip layer replays that plan with a copy-only kernel issued ahead on a side stream, so the skip layers' host→device IO overlaps the intervening layers' compute instead of sitting on the decode critical path. The replay kernel uses a small fixed grid to keep its SM footprint low while overlapped. The prefetch is enabled automatically for eligible models (no pipeline parallelism, no speculative decoding) and can be turned off for A/B comparison with `SGLANG_DISABLE_HISPARSE_PREFETCH=1`. ## Deployment HiSparse currently requires **PD disaggregation mode** and is enabled only on the **decode instance**. ### Prefill Instance ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path /path/to/model \ --trust-remote-code \ --port 8000 --host 0.0.0.0 \ --context-length 81920 \ --chunked-prefill-size 65536 \ --tp-size 8 --attn-dp-size 8 \ --mem-fraction-static 0.85 \ --disaggregation-mode prefill \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \ --nnodes 1 --node-rank 0 ``` ### Decode Instance (with HiSparse) ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path /path/to/model \ --trust-remote-code \ --port 8000 --host 0.0.0.0 \ --context-length 81920 \ --tp-size 8 --attn-dp-size 8 \ --mem-fraction-static 0.85 \ --disable-radix-cache \ --disaggregation-mode decode \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \ --dist-init-addr 127.0.0.1:5757 \ --nnodes 1 --node-rank 0 \ --enable-hisparse \ --hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10, "swap_in_block_size": 960}' ``` > **Note**: For DSA models, `--kv-cache-dtype` defaults to `auto`, which resolves to `fp8_e4m3` on SM100+ (Blackwell) and `bfloat16` on older architectures. The DSA decode backend is automatically selected based on KV dtype (`bfloat16` → `flashmla_sparse`, `fp8_e4m3` → `flashmla_kv`), except for GLM DSA models on SM120/SM121 with `fp8_e4m3`, which use `flashinfer_sparse_mla`. DSA backend flags apply only to DSA models; DeepSeek V4 uses its own `dsv4` attention backend. ### MiniMax M3 Dense-layer K/V and index K stay on GPU; sparse-layer K/V use host memory plus a GPU working set. * Use TP 4 or greater, with the same TP size and PP 1 on both PD instances. * Use `--attention-backend triton`, `--mm-attention-backend triton_attn`, `--disable-prefill-cuda-graph`, and `--disable-radix-cache`. * Set `device_buffer_size` to at least 2048 in `--hisparse-config`; `top_k` does not override the model's selection width. * PD retraction backup is not supported. Use `--num-reserved-decode-tokens` to reserve capacity for the expected output length. ### Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \ --dataset-name random \ --random-input 40000 \ --random-output 20000 \ --num-prompts 200 \ --max-concurrency 200 \ --request-rate 40 \ --random-range-ratio 1.0 \ --host 127.0.0.1 \ --port 20000 \ --model /path/to/model \ --flush-cache \ ``` ### Key Notes * The prefill instance does not need `--enable-hisparse`; it is unaware of HiSparse. * On the decode instance, `--enable-hisparse` and `--hisparse-config` are required for HiSparse. * For DSA models, `--kv-cache-dtype bfloat16` uses `flashmla_sparse`, and `--kv-cache-dtype fp8_e4m3` uses `flashmla_kv`. * On SM120/SM121 (e.g. RTX PRO 6000, RTX 5090) with GLM DSA models and `--kv-cache-dtype fp8_e4m3`, both DSA backends resolve to `flashinfer_sparse_mla`, which is the only DSA kernel available on that architecture. HiSparse accepts it there; no extra flag is needed. * For DeepSeek V4, DSA backend flags are not applicable. DeepSeek V4 uses the `dsv4` attention backend and `fp8_e4m3` KV cache by default. * `host_to_device_ratio` should be configured based on the host machine's available memory. For example: * **\~1 TB** host memory → `host_to_device_ratio: 5` * **\~2 TB** host memory → `host_to_device_ratio: 10` ## Acknowledgments We would like to thank the SGLang team and community for the implementation and generous support, especially Zhiqiang Xie, Zhangheng Huang, Tingwei Huang, Shangming Cai, Teng Ma, and many others. We also thank the Alibaba Cloud TairKVCache team and the AntGroup SCT Inference team for their valuable contributions. # Hyperparameter Tuning Source: https://docs.sglang.io/docs/advanced_features/hyperparameter_tuning ## Achieving high throughput for offline batch inference Achieving a large batch size is the most important thing for attaining high throughput in offline batch inference. When the server is running at full load in a steady state, look for the following in the log: ```text Output theme={null} Decode batch. #running-req: 233, #token: 370959, token usage: 0.82, cuda graph: True, gen throughput (token/s): 4594.01, #queue-req: 317 ``` ### Adjust the request submission speed to control `#queue-req` `#queue-req` indicates the number of requests in the queue. If you frequently see `#queue-req: 0`, it suggests that your client code is submitting requests too slowly. A healthy range for `#queue-req` is `100 - 2000`. However, avoid making `#queue-req` too large, as this will increase the scheduling overhead on the server. ### Achieve a high `token usage` `token usage` indicates the KV cache memory utilization of the server. `token usage > 0.9` means good utilization. If you frequently see `token usage < 0.9` and `#queue-req > 0`, it means the server is too conservative about taking in new requests. You can decrease `--schedule-conservativeness` to a value like 0.3. The case of a server being too conservative can happen when users send many requests with a large `max_new_tokens` but the requests stop very early due to EOS or stop strings. On the other hand, if you see `token usage` very high and you frequently see warnings like `KV cache pool is full. Retract requests. #retracted_reqs: 1, #new_token_ratio: 0.9998 -> 1.0000`, you can increase `--schedule-conservativeness` to a value like 1.3. If you see `KV cache pool is full. Retract requests.` occasionally but not frequently (\~1 time per minute), it is okay. ### Tune `--mem-fraction-static` to increase KV cache pool capacity SGLang allocates memory as follows: Total memory usage = model weights + KV cache pool + CUDA graph buffers + activations The `--mem-fraction-static` parameter determines how much memory is allocated to the first two components: mem\_fraction\_static = (model weights + KV cache pool) / GPU memory capacity To support higher concurrency, you should maximize the KV cache pool capacity by setting `--mem-fraction-static` as high as possible while still reserving enough memory for activations and CUDA graph buffers. SGLang uses simple heuristics to set the default value of `--mem-fraction-static`, but you can optimize it for your use cases. As a rule of thumb, reserving 5–8 GB of memory for activations is typically sufficient. You can check this by inspecting the logs just before the server is ready. Look for log entries like this: ```text Output theme={null} [2025-08-11 17:17:03] max_total_num_tokens=665690, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4096, context_len=65536, available_gpu_mem=13.50 GB ``` Check the `available_gpu_mem` value. * If it is between 5–8 GB, the setting is good. * If it is too high (e.g., 10 - 20 GB), increase `--mem-fraction-static` to allocate more memory to the KV cache. * If it is too low, you risk out-of-memory (OOM) errors later, so decrease `--mem-fraction-static`. Another straightforward approach is to increase `--mem-fraction-static` in increments of 0.01 until you encounter OOM errors for your workloads. ### Avoid out-of-memory errors by tuning `--chunked-prefill-size`, `--mem-fraction-static`, and `--max-running-requests` If you encounter out-of-memory (OOM) errors, you can adjust the following parameters: * If OOM occurs during prefill, try reducing `--chunked-prefill-size` to `4096` or `2048`. This saves memory but slows down the prefill speed for long prompts. * If OOM occurs during decoding, try lowering `--max-running-requests`. * You can also reduce `--mem-fraction-static` to a smaller value, such as 0.8 or 0.7. This decreases the memory usage of the KV cache memory pool and helps prevent OOM errors during both prefill and decoding. However, it limits maximum concurrency and reduces peak throughput. ### Tune `--cuda-graph-max-bs-decode` By default, CUDA graph is enabled only for small batch sizes (e.g., less than 160 or 256). However, for some models, especially at large tensor parallelism sizes, CUDA graph can be useful for batch sizes up to 512 or 768. Therefore, it may be beneficial to increase `--cuda-graph-max-bs-decode` to a larger value. Note that CUDA graph consumes more memory, so you may need to reduce `--mem-fraction-static` at the same time. ### Tune `--dp-size` and `--tp-size` Data parallelism is better for throughput. When there is enough GPU memory, always favor data parallelism for throughput. Refer to [SGLang Model Gateway (former Router)](../advanced_features/sgl_model_gateway) for a better data parallelism rather than using `dp_size` parameter. ### Try other options * `torch.compile` accelerates small models on small batch sizes. You can enable it with `--enable-torch-compile`. * Try other quantization (e.g. FP8 quantization with `--quantization fp8`) * Try other parallelism strategies (e.g. [expert parallelism](https://lmsys.org/blog/2025-05-05-large-scale-ep/)) or DP attention for deepseek models (with `--attn-dp-size 8`). * If the workload has many shared prefixes, try `--schedule-policy lpm`. Here, `lpm` stands for longest prefix match. It reorders requests to encourage more cache hits but introduces more scheduling overhead. # llm-d Source: https://docs.sglang.io/docs/advanced_features/llm-d [llm-d](https://llm-d.ai/) is a Kubernetes-native distributed inference framework for serving large language models at scale across a fleet of inference servers. SGLang is a supported inference engine in llm-d: llm-d coordinates a fleet of SGLang instances across a cluster so that performance holds up under real production traffic, achieving the fastest "time to state-of-the-art (SOTA) performance" for key OSS models across most hardware accelerators. llm-d is a [CNCF Sandbox project](https://www.cncf.io/blog/2026/03/24/welcome-llm-d-to-the-cncf-evolving-kubernetes-into-sota-ai-infrastructure/) founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. ## What llm-d adds to an SGLang deployment A single SGLang server is fast, and RadixAttention already maximizes cache reuse within each replica. But at scale the picture changes: across many replicas, cache locality breaks under round-robin load balancing as related requests scatter and radix-cache hit rates collapse, long prompts inflate time-to-first-token, and accelerators sit underused. llm-d adds the cluster-level layer that the engine does not aim to provide on its own: * **[Prefix-aware routing](https://llm-d.ai/docs/guides/precise-prefix-cache-aware).** Instead of round-robin, the llm-d Router scores each replica on prefix-cache locality and current load, routing each request to the replica most likely to already hold its prefix — raising RadixAttention hit rates on multi-turn and shared-prefix workloads while avoiding saturated servers. * **[Distributed KV-cache management](https://llm-d.ai/docs/guides#advanced-kv-cache-management).** A global index tracks which token blocks live on which replica, and tiered offloading spills cache to CPU memory or local SSD, extending the working set beyond accelerator HBM. * **[Prefill/decode disaggregation](https://llm-d.ai/docs/guides/pd-disaggregation).** Prompt processing and token generation run on separate workers, with KV-cache moved over high-speed interconnects, lowering TTFT and steadying per-token latency on long prompts. * **[SLO-aware autoscaling](https://llm-d.ai/docs/guides/workload-autoscaling) and [flow control](https://llm-d.ai/docs/guides/flow-control).** Scale SGLang pools on real inference signals (queue depth, true demand) rather than raw GPU utilization, with multi-tenant fairness and priority dispatch. * **One control plane for mixed fleets.** llm-d schedules across engines, so platform teams can serve SGLang and vLLM pools behind the same gateway, policies, and observability instead of running parallel stacks. These capabilities are composable. Most teams start by adding prefix-aware routing over an existing SGLang pool, then layer in the rest as specific bottlenecks appear. ## Kubernetes-native gateway llm-d builds on the Gateway API Inference Extension, so SGLang pools are managed through standard Kubernetes resources (Gateway, HTTPRoute, InferencePool) and work with supported gateway providers such as Istio, GKE Inference Gateway, and agentgateway, rather than a bespoke routing tier. ## Performance llm-d publishes reproducible benchmarks from production-scale deployments on [Prism](https://prism.llm-d.ai/). One representative result: prefix-aware routing delivered **3x higher output throughput and 2x faster TTFT** than round-robin load balancing (Llama 3.1 70B). The mechanism carries over directly to SGLang, where RadixAttention makes the cluster-level cache hit rate a function of routing. ## Get started * **Deploy the optimized baseline with the [Quickstart](https://llm-d.ai/docs/getting-started/quickstart), selecting SGLang as the inference server.** It stands up an intelligent router (the llm-d Router) over an SGLang pool on Kubernetes in a tested configuration. * Browse the [well-lit path guides](https://llm-d.ai/docs/guides), each a tested recipe for one of the capabilities above, and add the optimization that fits your workload. * Read the [Introduction](https://llm-d.ai/docs/getting-started) and [Architecture overview](https://llm-d.ai/docs/architecture) to see how the scheduler, gateway, and model servers wrap your SGLang deployment. Questions and contributions are welcome on [GitHub](https://github.com/llm-d/llm-d) and [Slack](https://llm-d.ai/slack). ## Current scope SGLang is supported across the well-lit paths — including intelligent inference scheduling, precise prefix-cache routing (SGLang publishes KV-cache events that the llm-d Router subscribes to), tiered KV-cache management, prefill/decode disaggregation, flow control, and autoscaling. The one current exception is Multi-Node Wide Expert Parallelism, which is vLLM-specific today. See the llm-d documentation for the latest per-engine support status. # LoRA Serving Source: https://docs.sglang.io/docs/advanced_features/lora SGLang enables the use of [LoRA adapters](https://arxiv.org/abs/2106.09685) with a base model. By incorporating techniques from [S-LoRA](https://arxiv.org/pdf/2311.03285) and [Punica](https://arxiv.org/pdf/2310.18547), SGLang can efficiently support multiple LoRA adapters for different sequences within a single batch of inputs. ## Arguments for LoRA Serving The following server arguments are relevant for multi-LoRA serving: * `enable_lora`: Enable LoRA support for the model. This argument is automatically set to True if `--lora-paths` is provided for backward compatibility. * `enable_lora_overlap_loading`: Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters. * `lora_paths`: The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: \ | \=\ | JSON with schema \{"lora\_name":str,"lora\_path":str,"pinned":bool}. * `max_loras_per_batch`: Maximum number of adaptors used by each batch. This argument can affect the amount of GPU memory reserved for multi-LoRA serving, so it should be set to a smaller value when memory is scarce. Defaults to be 8. * `max_loaded_loras`: If specified, it limits the maximum number of LoRA adapters loaded in CPU memory at a time. The value must be greater than or equal to `max-loras-per-batch`. * `lora_eviction_policy`: LoRA adapter eviction policy when GPU memory pool is full. `lru`: Least Recently Used (default, better cache efficiency). `fifo`: First-In-First-Out. * `lora_backend`: The backend of running GEMM kernels for Lora modules. Currently we support Triton LoRA backend (`triton`) and Chunked SGMV backend (`csgmv`). In the future, faster backend built upon Cutlass or Cuda kernels will be added. * `max_lora_rank`: The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup. * `lora_target_modules`: The union set of all target modules where LoRA should be applied (e.g., `q_proj`, `k_proj`, `gate_proj`). If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of different target modules after server startup. You can also set it to `all` to enable LoRA for all supported modules. However, enabling LoRA on additional modules introduces a minor performance overhead. If your application is performance-sensitive, we recommend only specifying the modules for which you plan to load adapters. * `max_lora_chunk_size`: Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when --lora-backend is 'csgmv'. Choosing a larger value might improve performance. Please tune this value based on your hardware and workload as needed. Defaults to 16. * `lora_drain_wait_threshold`: When any LoRA adapter request waits longer than this threshold (in seconds), the scheduler will selectively drain one running adapter to make room. This mitigates extreme tail latency under high or skewed workloads by preventing a small set of adapters from monopolizing batch slots. Set to 0 to disable draining (default). * `tp_size`: LoRA serving along with Tensor Parallelism is supported by SGLang. `tp_size` controls the number of GPUs for tensor parallelism. More details on the tensor sharding strategy can be found in [S-Lora](https://arxiv.org/pdf/2311.03285) paper. From client side, the user needs to provide a list of strings as input batch, and a list of adaptor names that each input sequence corresponds to. ## Usage ### Serving Single Adaptor **Note:** SGLang supports LoRA adapters through two APIs: 1. **OpenAI-Compatible API** (`/v1/chat/completions`, `/v1/completions`): Use the `model:adapter-name` syntax. See [OpenAI API with LoRA](../basic_usage/openai_api_completions#using-lora-adapters) for examples. 2. **Native API** (`/generate`): Pass `lora_path` in the request body (shown below). ```python Example theme={null} import json import requests from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, terminate_process ``` ```python Example theme={null} server_process, port = launch_server_cmd( # Here we set max-loras-per-batch to 2: one slot for adaptor and another one for base model """ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --enable-lora \ --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \ --max-loras-per-batch 2 \ --log-level warning \ """ ) wait_for_server(f"http://localhost:{port}") ``` ```python Example theme={null} url = f"http://127.0.0.1:{port}" json_data = { "text": [ "List 3 countries and their capitals.", "List 3 countries and their capitals.", ], "sampling_params": {"max_new_tokens": 32, "temperature": 0}, # The first input uses lora0, and the second input uses the base model "lora_path": ["lora0", None], } response = requests.post( url + "/generate", json=json_data, ) print(f"Output 0: {response.json()[0]['text']}") print(f"Output 1: {response.json()[1]['text']}") ``` ```python Example theme={null} terminate_process(server_process) ``` ### Serving Multiple Adaptors ```python Example theme={null} server_process, port = launch_server_cmd( """ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --enable-lora \ --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \ lora1=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \ --max-loras-per-batch 2 \ --log-level warning \ """ ) wait_for_server(f"http://localhost:{port}") ``` ```python Example theme={null} url = f"http://127.0.0.1:{port}" json_data = { "text": [ "List 3 countries and their capitals.", "List 3 countries and their capitals.", ], "sampling_params": {"max_new_tokens": 32, "temperature": 0}, # The first input uses lora0, and the second input uses lora1 "lora_path": ["lora0", "lora1"], } response = requests.post( url + "/generate", json=json_data, ) print(f"Output 0: {response.json()[0]['text']}") print(f"Output 1: {response.json()[1]['text']}") ``` ```python Example theme={null} terminate_process(server_process) ``` ### Dynamic LoRA loading Instead of specifying all adapters during server startup via `--lora-paths`. You can also load & unload LoRA adapters dynamically via the `/load_lora_adapter` and `/unload_lora_adapter` API. When using dynamic LoRA loading, it's recommended to explicitly specify both `--max-lora-rank` and `--lora-target-modules` at startup. For backward compatibility, SGLang will infer these values from `--lora-paths` if they are not explicitly provided. However, in that case, you would have to ensure that all dynamically loaded adapters share the same shape (rank and target modules) as those in the initial `--lora-paths` or are strictly "smaller". ```python Example theme={null} lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json" # rank - 4, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj lora1 = "algoprog/fact-generation-llama-3.1-8b-instruct-lora" # rank - 64, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj lora0_new = "philschmid/code-llama-3-1-8b-text-to-sql-lora" # rank - 256, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj # The `--target-lora-modules` param below is technically not needed, as the server will infer it from lora0 which already has all the target modules specified. # We are adding it here just to demonstrate usage. server_process, port = launch_server_cmd( """ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --enable-lora \ --cuda-graph-max-bs-decode 2 \ --max-loras-per-batch 2 \ --max-lora-rank 256 --lora-target-modules all --log-level warning """ ) url = f"http://127.0.0.1:{port}" wait_for_server(url) ``` Load adapter lora0 ```python Example theme={null} response = requests.post( url + "/load_lora_adapter", json={ "lora_name": "lora0", "lora_path": lora0, }, ) if response.status_code == 200: print("LoRA adapter loaded successfully.", response.json()) else: print("Failed to load LoRA adapter.", response.json()) ``` Load adapter lora1: ```python Example theme={null} response = requests.post( url + "/load_lora_adapter", json={ "lora_name": "lora1", "lora_path": lora1, }, ) if response.status_code == 200: print("LoRA adapter loaded successfully.", response.json()) else: print("Failed to load LoRA adapter.", response.json()) ``` Check inference output: ```python Example theme={null} url = f"http://127.0.0.1:{port}" json_data = { "text": [ "List 3 countries and their capitals.", "List 3 countries and their capitals.", ], "sampling_params": {"max_new_tokens": 32, "temperature": 0}, # The first input uses lora0, and the second input uses lora1 "lora_path": ["lora0", "lora1"], } response = requests.post( url + "/generate", json=json_data, ) print(f"Output from lora0: \n{response.json()[0]['text']}\n") print(f"Output from lora1 (updated): \n{response.json()[1]['text']}\n") ``` Unload lora0 and replace it with a different adapter: ```python Example theme={null} response = requests.post( url + "/unload_lora_adapter", json={ "lora_name": "lora0", }, ) response = requests.post( url + "/load_lora_adapter", json={ "lora_name": "lora0", "lora_path": lora0_new, }, ) if response.status_code == 200: print("LoRA adapter loaded successfully.", response.json()) else: print("Failed to load LoRA adapter.", response.json()) ``` Check output again: ```python Example theme={null} url = f"http://127.0.0.1:{port}" json_data = { "text": [ "List 3 countries and their capitals.", "List 3 countries and their capitals.", ], "sampling_params": {"max_new_tokens": 32, "temperature": 0}, # The first input uses lora0, and the second input uses lora1 "lora_path": ["lora0", "lora1"], } response = requests.post( url + "/generate", json=json_data, ) print(f"Output from lora0: \n{response.json()[0]['text']}\n") print(f"Output from lora1 (updated): \n{response.json()[1]['text']}\n") ``` ```python Example theme={null} terminate_process(server_process) ``` ### OpenAI-compatible API usage You can use LoRA adapters via the OpenAI-compatible APIs by specifying the adapter in the `model` field using the `base-model:adapter-name` syntax (for example, `qwen/qwen2.5-0.5b-instruct:adapter_a`). For more details and examples, see the “Using LoRA Adapters” section in the OpenAI API documentation: [openai\_api\_completions](../basic_usage/openai_api_completions). ### LoRA GPU Pinning Another advanced option is to specify adapters as `pinned` during loading. When an adapter is pinned, it is permanently assigned to one of the available GPU pool slots (as configured by `--max-loras-per-batch`) and will not be evicted from GPU memory during runtime. Instead, it remains resident until it is explicitly unloaded. This can improve performance in scenarios where the same adapter is frequently used across requests, by avoiding repeated memory transfers and reinitialization overhead. However, since GPU pool slots are limited, pinning adapters reduces the flexibility of the system to dynamically load other adapters on demand. If too many adapters are pinned, it may lead to degraded performance, or in the most extreme case (`Number of pinned adapters == max-loras-per-batch`), halt all unpinned requests. Therefore, currently SGLang limits maximal number of pinned adapters to `max-loras-per-batch - 1` to prevent unexpected starvations. In the example below, we start a server with `lora1` loaded as pinned, `lora2` and `lora3` loaded as regular (unpinned) adapters. Please note that, we intentionally specify `lora2` and `lora3` in two different formats to demonstrate that both are supported. ```python Example theme={null} server_process, port = launch_server_cmd( """ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --enable-lora \ --cuda-graph-max-bs-decode 8 \ --max-loras-per-batch 3 \ --max-lora-rank 256 \ --lora-target-modules all \ --lora-paths \ {"lora_name":"lora0","lora_path":"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json","pinned":true} \ {"lora_name":"lora1","lora_path":"algoprog/fact-generation-llama-3.1-8b-instruct-lora"} \ lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora --log-level warning """ ) url = f"http://127.0.0.1:{port}" wait_for_server(url) ``` You can also specify adapter as pinned during dynamic adapter loading. In the example below, we reload `lora2` as pinned adapter: ```python Example theme={null} response = requests.post( url + "/unload_lora_adapter", json={ "lora_name": "lora1", }, ) response = requests.post( url + "/load_lora_adapter", json={ "lora_name": "lora1", "lora_path": "algoprog/fact-generation-llama-3.1-8b-instruct-lora", "pinned": True, # Pin the adapter to GPU }, ) ``` Verify that the results are expected: ```python Example theme={null} url = f"http://127.0.0.1:{port}" json_data = { "text": [ "List 3 countries and their capitals.", "List 3 countries and their capitals.", "List 3 countries and their capitals.", ], "sampling_params": {"max_new_tokens": 32, "temperature": 0}, # The first input uses lora0, and the second input uses lora1 "lora_path": ["lora0", "lora1", "lora2"], } response = requests.post( url + "/generate", json=json_data, ) print(f"Output from lora0 (pinned): \n{response.json()[0]['text']}\n") print(f"Output from lora1 (pinned): \n{response.json()[1]['text']}\n") print(f"Output from lora2 (not pinned): \n{response.json()[2]['text']}\n") ``` ```python Example theme={null} terminate_process(server_process) ``` ## Choosing LoRA Backend SGLang supports two LoRA backends that you can choose from using the `--lora-backend` argument: * `triton`: Basic Triton-based backend. * `csgmv`: Default chunked SGMV backend optimized for high concurrency scenarios. The `csgmv` backend was recently introduced to improve performance especially at high-concurrency scenarios. Our benchmark shows that it achieves 20% to 80% latency improvements over the basic triton backend. ```python Example theme={null} server_process, port = launch_server_cmd( """ python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --enable-lora \ --lora-backend csgmv \ --max-loras-per-batch 16 \ --lora-paths lora1=path/to/lora1 lora2=path/to/lora2 """ ) ``` ```python Example theme={null} terminate_process(server_process) ``` ## LoRA with DP attention SGLang supports LoRA serving with [data parallel attention (DPA)](/docs/advanced_features/dp_dpa_smg_guide). The model architecture must support DPA, and you must currently select the Triton LoRA backend. Other built-in LoRA backends reject this configuration at startup. The following NVIDIA CUDA example uses four attention-DP ranks: ```bash theme={null} python3 -m sglang.launch_server \ --model-path MODEL_PATH \ --tp 4 \ --attn-dp-size 4 \ --enable-lora \ --lora-backend triton \ --lora-paths '{"lora_name":"ADAPTER_NAME","lora_path":"ADAPTER_PATH","pinned":true}' ``` Keep these constraints in mind: * `--attn-dp-size` must equal `--tp-size`, so every attention DP rank owns its whole token batch. The server rejects other layouts at startup. * All adapters must be GPU-pinned (`pinned: true`), including dynamically loaded adapters. You can pin at most `--max-loras-per-batch - 1` adapters; the remaining slot is reserved for the base model. * Decode CUDA graphs are supported. LoRA prefill runs eagerly when DPA is enabled. * `--max-loras-per-batch` limits the union of adapters active across all DP ranks in a synchronized forward, not the adapters on one rank alone. * Dynamic load and unload requests are sent to every DP rank. An operation succeeds only if every rank reports success. After a partially successful load, retry unloading the same adapter name before loading it again. * The built-in LoRA memory pool does not support `--enable-lora-overlap-loading` with DPA. The server rejects that combination rather than allowing DP ranks to assign different GPU slots. ## LoRA Overlap Loading By using the `--enable-lora-overlap-loading` server argument, the SGLang engine is able to overlap the loading of LoRA weights with prefill and decode compute, essentially hiding the data movement for LoRA weights behind GPU computation. Our benchmarks show that under adversarial conditions, enabling this feature can result in a \~35% reduction in median TTFT - (see the [LoRA overlap loading PR](https://github.com/sgl-project/sglang/pull/15512) for detailed benchmarks). ```python Example theme={null} lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json" lora1 = "algoprog/fact-generation-llama-3.1-8b-instruct-lora" lora2 = "philschmid/code-llama-3-1-8b-text-to-sql-lora" server_process, port = launch_server_cmd( """ python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --enable-lora \ --enable-lora-overlap-loading \ --lora-paths lora0=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \ lora1=algoprog/fact-generation-llama-3.1-8b-instruct-lora \ lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora \ --max-lora-rank 256 \ --max-loras-per-batch 2 \ --max-loaded-loras 4 """ ) url = f"http://127.0.0.1:{port}" wait_for_server(url) ``` ```python Example theme={null} json_data = { "text": [ "Write a very long fairy-tale.", "List 3 countries and their capitals.", "List 3 countries and their capitals.", ], "sampling_params": [ {"max_new_tokens": 1024, "temperature": 0}, {"max_new_tokens": 64, "temperature": 0}, {"max_new_tokens": 64, "temperature": 0}, ], "lora_path": ["lora0", "lora1", "lora2"], } # lora0 and lora1 will be loaded into the memory pool first, and because max_loras_per_batch = 2, lora2's request will remain in the queue. # lora1's request will likely finish first, and once it does, lora2 will be loaded. With --enable-lora-overlap-loading, this loading will # occur asynchronously and thus decoding for lora0's request won't be blocked. response = requests.post( url + "/generate", json=json_data, ) for i in range(3): print(f"Output from lora{i}: \n{response.json()[i]['text']}\n") ``` ```python Example theme={null} terminate_process(server_process) ``` #### Limitations of LoRA Overlap Loading However, LoRA overlap loading is not free and comes with two important caveats: 1. **Pinned CPU memory requirement**: Asynchronous H2D memory copies require LoRA weights to be pinned in CPU memory, which is a finite system resource. To mitigate excessive pinned-memory usage, SGLang currently restricts `max_loaded_loras` to be at most 2× `max_loras_per_batch` when LoRA overlap loading is enabled. 2. **Reduced multi-adapter prefill batching**: With overlap loading, adapters become available on the GPU at different times because each adapter is loaded asynchronously. This can reduce the scheduler’s ability to form multi-adapter prefill batches, since only requests whose adapters are currently loaded can be grouped together. As a result, requests for different adapters will be scheduled in separate (or smaller) prefill batches, which can increase TTFT when adapter load time is small compared to prefill compute time. This is why LoRA overlap loading is disabled by default: it should only be enabled when users have determined that LoRA weight loading is a bottleneck (EG high adapter churn, heavy adapter weights, or PCIe-bottlenecked workloads). #### Example When Overlap Loading Results in Higher Latency For instance, suppose we have four LoRA adapters: `lora0`, `lora1`, `lora2`, and `lora3`. Loading any adapter takes 2ms, while the prefill step for requests for that adapter takes 20ms. 1. **Baseline**: The engine loads all four adapters synchronously, then runs one combined prefill batch, giving us a total time of ≈ `2 * 4 + 20 = 28ms` 2. **With LoRA overlap loading enabled**: The engine begins loading `lora0` and, once it is ready, schedules a prefill batch containing only `lora0` while `lora1` loads in the background. Then it schedules `lora1`’s prefill while `lora2` loads, and so on. In the worst case where prefill cannot be batched across adapters, total time is ≈ `2 + 4 * 20 = 82ms` In this scenario, overlap loading reduces adapter-load overhead, but the loss of multi-adapter prefill batching dominates and leads to higher TTFT. ## Future Works The development roadmap for LoRA-related features can be found in this [issue](https://github.com/sgl-project/sglang/issues/2929). Other features, including Embedding Layer, Unified Paging, Cutlass backend are still under development. # Model Loading Source: https://docs.sglang.io/docs/advanced_features/model_loading Control how SGLang loads model weights: load formats, model loader extra config, multithreaded loading, prefetching, and remote/streaming loaders. `--model-path` selects the checkpoint to serve; `--load-format` and the weight-loading flags below control how those weights are read into memory. To stream weights from cloud object storage (S3/GCS/Azure), see [Loading Models from Object Storage](./object_storage). ## How loading works SGLang picks a loader from `--load-format`, falling back to auto-detection from the checkpoint or model path. The default `auto` loader reads `safetensors` and falls back to PyTorch `.bin`. ```bash theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3.6-35B-A3B \ --load-format auto ``` Some formats are auto-detected and override `auto`: * A Mistral native checkpoint is detected and loaded with `mistral`. * A `.gguf` model path is detected and loaded with `gguf`. * An object storage URI (`s3://`, `gs://`, `az://`) is loaded with `runai_streamer`. * A remote URI is loaded with `remote`. ## Load formats Set with `--load-format`:
Format Description
auto Default. Load safetensors if available, otherwise fall back to the PyTorch .bin format.
safetensors Load weights in the safetensors format.
instanttensor Load Safetensors weights quickly with InstantTensor's distributed loading and pipelined prefetching. Supports CUDA and ROCm; requires PyTorch's cuda device interface. Distributed loading may retain communicator memory for the server lifetime.
pt Load weights in the PyTorch .bin format.
npcache Load PyTorch-format weights and store a numpy cache to speed up subsequent loads. Only supports .bin checkpoints.
dummy Initialize weights with random values, for profiling.
sharded\_state Each tensor-parallel worker reads only its own pre-sharded shard rather than the full checkpoint, giving a fast load path for large TP models. See examples/runtime/engine/save\_sharded\_state.py for creating a sharded checkpoint.
fastsafetensors Load safetensors using the fastsafetensors iterator.
layered Load weights layer by layer, so a layer can be quantized before the next is loaded, lowering the peak memory envelope.
gguf Load weights in the GGUF format. Auto-detected from a .gguf model path.
bitsandbytes Load weights using bitsandbytes quantization.
mistral Load a Mistral native-format checkpoint. Auto-detected for such checkpoints.
flash\_rl Load a BF16/FP16 checkpoint with native SGLang FP8 quantization for RL training. Requires --rl-quant-profile.
runai\_streamer Stream weights from SSDs, shared filesystems, or object storage. See Loading Models from Object Storage.
remote Load tensors from a remote KV/filesystem connector. Auto-detected for remote URIs.
remote\_instance Pull weights over the network from another running SGLang instance (the "seed") rather than from disk. Configured with the --remote-instance-weight-loader-\* flags.
## Model loader extra config `--model-loader-extra-config` takes a JSON string passed to the loader selected by `--load-format`. ```bash theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3.6-35B-A3B \ --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 16}' ```
Load format Key Description Default
auto / safetensors / pt / npcache enable\_multithread\_load (bool) Read weight shards with a thread pool instead of sequentially. Disabled by default when --weight-loader-prefetch-checkpoints is set (to avoid I/O oversubscription with the prefetch threads); set this to true to opt back in. true
auto / safetensors / pt / npcache num\_threads (int) Number of worker threads when multithreaded loading is enabled. 8
sharded\_state pattern (str) Filename pattern for per-rank shards. model-rank-\{rank}-part-\{part}.safetensors
fastsafetensors enable\_gds (bool) Use GPU Direct Storage. Set to false when the host does not provide the NVIDIA GPUDirect Storage kernel driver, such as in a gVisor sandbox. true
instanttensor buffer\_size (int) GPU tensor buffer size in bytes. See the InstantTensor GitHub homepage for the latest optional parameters. null
instanttensor chunk\_size (int) File I/O operation size in bytes. null
instanttensor io\_depth (int) Maximum number of rank-local I/O operations in flight. null
instanttensor backend (str / list) A backend name or an ordered list of candidates: AIO, AIO\_BUFFERED, URING, URING\_BUFFERED, CUFILE, MMAP, or the BUFFERED policy. null
bitsandbytes qlora\_adapter\_name\_or\_path (str) QLoRA adapter to apply on top of the bitsandbytes-quantized base weights. —
runai\_streamer distributed, concurrency, memory\_limit Streaming controls. See Loading Models from Object Storage. See linked page
## Weight-loading performance flags Top-level arguments that tune how safetensors weights are read, independent of `--load-format`.
Flag Description Default
--download-dir Directory used to download and cache Hugging Face model files. HF default
--weight-loader-disable-mmap Disable mmap while loading safetensors. Can help on filesystems where mmap is slow. off
--weight-loader-prefetch-checkpoints Prefetch checkpoint files into the OS page cache before loading. Each rank prefetches a fraction of the shards, cutting total network I/O on shared filesystems (NFS/Lustre) from N×checkpoint to 1×checkpoint. Recommended for models on network storage. When enabled, multi-threaded safetensors loading is disabled by default to avoid I/O oversubscription with the prefetch threads; set enable\_multithread\_load=true in --model-loader-extra-config to keep multi-threaded loading (e.g. on local NVMe where prefetch is a no-op). off
--weight-loader-prefetch-num-threads Threads per rank for checkpoint prefetching. 4
--weight-loader-drop-cache-after-load Call posix\_fadvise(DONTNEED) after successfully loading each shard, freeing page cache. Supported by the standard safetensors and fastsafetensors loaders. off
--custom-weight-loader Import path(s) of a custom weight-loading function, e.g. my\_package.weight\_load\_func. —
## See also * [Loading Models from Object Storage](./object_storage) * [Quantization](./quantization) * [Server Arguments](./server_arguments) # Loading Models from Object Storage Source: https://docs.sglang.io/docs/advanced_features/object_storage SGLang can load models directly from object storage without a full local download. It uses the `runai_streamer` load format to stream model weights from cloud storage, reducing startup time and local storage requirements. ## Overview When loading models from object storage, SGLang uses a two-phase approach: 1. **Metadata Download** (once, before process launch): Configuration files and tokenizer files are downloaded to a local cache 2. **Weight Streaming** (lazy, during model loading): Model weights are streamed directly from object storage as needed ## Supported Storage Backends 1. **Amazon S3**: `s3://bucket-name/path/to/model/` 2. **Google Cloud Storage**: `gs://bucket-name/path/to/model/` 3. **Azure Blob**: `az://some-azure-container/path/` 4. **S3 compatible**: `s3://bucket-name/path/to/model/` ## Quick Start ### Basic Usage Simply provide an object storage URI as the model path: ```bash theme={null} # S3 python -m sglang.launch_server \ --model-path s3://my-bucket/models/llama-3-8b/ \ --load-format runai_streamer # Google Cloud Storage python -m sglang.launch_server \ --model-path gs://my-bucket/models/llama-3-8b/ \ --load-format runai_streamer ``` **Note**: The `--load-format runai_streamer` is automatically detected when using object storage URIs, so you can omit it: ```bash theme={null} python -m sglang.launch_server \ --model-path s3://my-bucket/models/llama-3-8b/ ``` ### With Tensor Parallelism ```bash theme={null} python -m sglang.launch_server \ --model-path gs://my-bucket/models/llama-70b/ \ --tp 4 \ --model-loader-extra-config '{"distributed": true}' ``` ## Configuration ### Load Format The `runai_streamer` load format is designed for object storage, SSDs, and shared filesystems. ```bash theme={null} python -m sglang.launch_server \ --model-path s3://bucket/model/ \ --load-format runai_streamer ``` ### Extended Configuration Parameters Use `--model-loader-extra-config` to pass additional configuration as a JSON string: ```bash theme={null} python -m sglang.launch_server \ --model-path s3://bucket/model/ \ --model-loader-extra-config '{ "distributed": true, "concurrency": 8, "memory_limit": 2147483648 }' ``` #### Available Parameters
Parameter Type Description Default
distributed bool Enable distributed streaming for multi-GPU setups. Automatically set to true for object storage paths on CUDA-like devices. Auto-detected
concurrency int Number of concurrent download streams. Higher values can improve throughput for large models. 4
memory\_limit int Memory limit (in bytes) for the streaming buffer. System-dependent
## Performance Considerations ### Distributed Streaming For multi-GPU setups, enable distributed streaming to parallelize weight loading across processes: ```bash theme={null} python -m sglang.launch_server \ --model-path s3://bucket/model/ \ --tp 8 \ --model-loader-extra-config '{"distributed": true}' ``` ## Limitations * **Supported formats**: Only the `.safetensors` weight format is supported. * **Supported devices**: Distributed streaming is supported on CUDA-like devices; otherwise it falls back to non-distributed streaming. ## See Also * [Runai model streamer documentation](https://github.com/run-ai/runai-model-streamer) # Observability Source: https://docs.sglang.io/docs/advanced_features/observability ## Production Metrics SGLang exposes the following metrics via Prometheus. You can enable them by adding `--enable-metrics` when launching the server. You can query them by: ```bash Command theme={null} curl http://localhost:30000/metrics ``` See [Production Metrics](../references/production_metrics) and [Production Request Tracing](../references/production_request_trace) for more details. ## Logging By default, SGLang does not log any request contents. You can log them by using `--log-requests`. You can control the verbosity by using `--log-request-level`. See [Logging](./server_arguments#logging) for more details. You can change verbosity at runtime: ```bash Command theme={null} python3 -m sglang.srt.managers.configure_logging --url http://localhost:30000 --log-level=debug ``` ## Request Dump and Replay You can dump all requests and replay them later for benchmarking or other purposes. To start dumping, use the following command to send a request to a server: ```bash Command theme={null} python3 -m sglang.srt.managers.configure_logging --url http://localhost:30000 --dump-requests-folder /tmp/sglang_request_dump --dump-requests-threshold 100 ``` The server will dump the requests into a pickle file for every 100 requests. To replay the request dump, use `scripts/playground/replay_request_dump.py`. ## Crash Dump and Replay Sometimes the server might crash, and you may want to debug the cause of the crash. SGLang can preserve recent request data for replay and collect CUDA device coredumps for low-level debugging. Set the crash diagnostics folder with `--crash-dump-folder /tmp/crash_dump`. When SGLang handles a crash, it writes completed requests retained by the crash-dump buffer plus in-flight requests to `/tmp/crash_dump//crash_dump_.pkl`. The file also contains the server arguments and launch command. Replay it with `scripts/playground/replay_request_dump.py`. On NVIDIA CUDA, the option also sets default environment variables before CUDA initializes. These defaults enable device coredumps on CUDA exceptions and allow SGLang to trigger device coredumps for live scheduler processes when it handles a crash. CUDA device coredumps are written to `/tmp/crash_dump//core.cuda..`. Explicitly configured CUDA coredump environment variables take precedence, including a custom `CUDA_COREDUMP_FILE` path. This option does not configure OS process core dumps. # Advanced Features Source: https://docs.sglang.io/docs/advanced_features/overview Advanced configuration, optimization, and deployment features for SGLang. * [Server Arguments](./server_arguments) * [Session-Aware Radix Cache](./session_radix_cache) * [Hyperparameter Tuning](./hyperparameter_tuning) * [Attention Backend](./attention_backend) * [Speculative Decoding](./speculative_decoding) * [Structured Outputs](./structured_outputs) * [Quantization](./quantization) * [Expert Parallelism](./expert_parallelism) * [Prefill Context Parallelism](./prefill_cp) * [Decode Context Parallelism](./dcp) * [LoRA](./lora) * [PD Disaggregation](./pd_disaggregation) * [Pipeline Parallelism](./pipeline_parallelism) * [HiCache](./hicache_best_practices) * [SGLang Simulator](./sglang_simulator) * [Observability](./observability) * [And more…](./server_arguments) # PD Disaggregation Source: https://docs.sglang.io/docs/advanced_features/pd_disaggregation ## Why and What is PD Disaggregation? Large Language Model (LLM) inference comprises two distinct phases: **Prefill** and **Decode**. The Prefill phase is computation-intensive, processing the entire input sequence, while the Decode phase is memory-intensive, managing the Key-Value (KV) cache for token generation. Traditionally, these phases are handled within a unified engine, where combined scheduling of prefill and decode batches introduces inefficiencies. To address these challenges, we introduce **Prefill and Decoding (PD) Disaggregation** in SGLang. ### Issues with Unified Scheduling The conventional unified engine, which processes prefill and decode batches together, results in two significant problems: 1. **Prefill Interruption**: Incoming prefill batches frequently interrupt ongoing decode batches, causing substantial delays in token generation. 2. **DP Attention Imbalance**: In data-parallel (DP) attention, one DP worker may process a prefill batch while another handles a decode batch simultaneously, leading to increased decode latency. PD Disaggregation resolves these by separating the two stages, enabling tailored optimizations for each. For the design details, please refer to [link](https://docs.google.com/document/d/1rQXJwKd5b9b1aOzLh98mnyMhBMhlxXA5ATZTHoQrwvc/edit?tab=t.0). Currently, we support Mooncake and NIXL as the transfer engine. ## Decode allocation policy By default, decode reserves each request's destination KV slots before prefill finishes. Long prefills can hold these slots while decode waits for the KV data. To delay this reservation, set the following flag on **both prefill and decode**: ```bash theme={null} --disaggregation-decode-allocation-policy prefill_complete ``` With `prefill_complete`, prefill computes without waiting for destination slots, then signals readiness after accepting its final prefill result. Decode reserves slots and starts the existing KV transfer handshake only after that signal. This changes per-request KV reservation, not the size or creation of the global KV pool. The default `early` policy preserves the existing behavior. Deferring allocation can help when long prefills compete with active decode requests for destination KV capacity. It also retains KV on prefill longer and removes overlap between chunked prefill and KV transfer. Compare both policies for your workload; this option is not a general throughput improvement. This initial implementation requires Mooncake (RDMA or `mooncake_tcp`), the Python bootstrap registry, and attention TP/CP and pipeline sizes of one. Attention DP is allowed when each DP rank has attention TP size one. Prefill automatically resolves `--optimistic-prefill-attempts` from zero to one; configurations that disable optimistic prefill after model validation are rejected. Chunked prefill does not yield just because decode has not admitted the request. Completed requests retain their KV in the existing optimistic-prefill inflight queue until decode supplies destination indices; readiness alone never starts a KV write. After a successful transfer, prefill clears the room's readiness and transfer state before notifying decode of success. Retraction recovery can then reuse the same bootstrap room. This notification waits for worker and scheduler cleanup; failure and cancellation records remain until their timeout expires. Retries remain eligible for prefill after the optimistic attempt budget is exhausted and remain subject to the existing cancellation and timeout handling. On prefill, `SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT` starts after the final prefill result, giving decode admission its own budget. On decode, `SGLANG_DISAGGREGATION_WAITING_TIMEOUT` bounds the wait for prefill completion, then starts afresh for admission and again when destination indices are sent for transfer. The option does not support staging buffers, runtime PD role switching, PD multiplexing, encoder disaggregation, decoupled speculation, HiSparse, decode radix cache, decode HiCache, or non-DSA auxiliary state pools. Prefill HiCache is limited to L2 with `write_back` and no storage backend. Unsupported configurations fail validation. All communicating workers must use the same revision and enable the same policy. This version of the decode worker rejects policy or peer-layout mismatches before destination registration. Older decode workers do not enforce this check. Keep `early` on both sides until every worker has been upgraded, then enable the new policy on both sides. ## Profiling in PD Disaggregation Mode When you need to profile prefill or decode workers in PD disaggregation mode, please refer to the [Profile In PD Disaggregation Mode](../developer_guide/benchmark_and_profiling#profile-in-pd-disaggregation-mode) section in the Benchmark and Profiling guide. Due to torch profiler limitations, prefill and decode workers must be profiled separately using dedicated command-line options. ## Router Integration For deploying PD disaggregation at scale with load balancing and fault tolerance, SGLang provides a router. The router can distribute requests between prefill and decode instances using various routing policies. For detailed information on setting up routing with PD disaggregation, including configuration options and deployment patterns, see the [SGLang Model Gateway (former Router)](./sgl_model_gateway#prefill-decode-disaggregation). ### Responses API under HTTP PD HTTP PD supports foreground generation through `POST /v1/responses`, including streaming generation with `stream=true` through the HTTP PD router. This requires both the router's Responses dispatch support and serving workers that accept Responses PD routing metadata. Store-backed and stateful Responses features are **not supported under PD**: * Response persistence (`store`) and response history. * Conversation chaining with `previous_response_id`. * Background Responses (`background=true`), including background streams. * Retrieval by response ID (`GET /v1/responses/{id}`). * Cancel-by-ID (`POST /v1/responses/{id}/cancel`). * Other workflows requiring shared process-local Responses history. Prefill and decode run in separate serving processes. Their process-local Responses stores cannot provide coherent shared state: both legs need the same conversation history to reconstruct the prompt, and prefill's stored response is not available to decode. Send explicit conversation history in foreground requests instead of relying on stored response IDs. Built-in tools (`web_search`, `code_interpreter`) are also unsupported under PD: each tool call requires another generation, but the router dispatches exactly one prefill/decode pair per HTTP request. Serving-side capability enforcement is tracked in [#39122](https://github.com/sgl-project/sglang/pull/39122). That change introduces opt-in standalone storage with `--enable-response-store`, disallows it under PD, and rejects stateful requests with HTTP 400 before generation. These checks require a serving version containing that change; older versions must not be assumed to reject every unsupported workflow at admission. The HTTP PD router also rejects detached background requests because it does not implement their retrieval/cancel lifecycle. ## Mooncake ### Requirements ```bash theme={null} uv pip install mooncake-transfer-engine ``` ### AWS EFA On an x86\_64 AWS EFA instance, replace the standard CUDA 13 Mooncake package with the EFA build. The two packages install the same `mooncake` Python module, so do not install them side by side. The following commands replace the package in an SGLang CUDA 13 environment while preserving its existing Python dependencies: ```bash theme={null} python3 -m pip uninstall -y \ mooncake-transfer-engine-cuda13 \ mooncake-transfer-engine-efa-cuda13 python3 -m pip install --no-deps \ mooncake-transfer-engine-efa-cuda13==0.3.13.post1 export LD_LIBRARY_PATH="/opt/amazon/efa/lib:/opt/amazon/efa/lib64:${LD_LIBRARY_PATH:-}" export MOONCAKE_PROTOCOL=efa python3 -c 'from mooncake import engine; assert engine.SUPPORT_EFA' fi_info -p efa -t FI_EP_RDM ``` The EFA wheel expects a compatible AWS EFA userspace installation on the host and does not bundle `libfabric` or `libefa`. The current wheel is available only for x86\_64. It is built without Mooncake EP/PG and the NVLink/MNNVL components, so use the standard Mooncake CUDA package for those features. The current libfabric EFA provider cannot transfer CUDA VMM-backed buffers. Until [libfabric issue #12853](https://github.com/ofiwg/libfabric/issues/12853) is resolved, apply these settings to both prefill and decode workers: * Do not set `SGLANG_MOONCAKE_CUSTOM_MEM_POOL` to `NVLINK` or `BAREX`. `INTRA_NODE_NVLINK` is supported because it uses the default CUDA allocator. * Do not enable `expandable_segments:True` in `PYTORCH_CUDA_ALLOC_CONF` or `PYTORCH_ALLOC_CONF`. * Post-capture KV sizing is automatically disabled for the EFA protocol because its KV cache uses CUDA VMM allocations. * Do not set `MC_FORCE_MNNVL` or `NCCL_MNNVL_ENABLE`; EFA is not an MNNVL transport. SGLang validates the first two constraints at startup to avoid a later `fi_read` or `fi_write` failure from libfabric. ### IB Device Configuration `--disaggregation-ib-device` supports the following formats when using the Mooncake backend: 1. Shared device list for all GPUs: `mlx5_0` or `mlx5_0,mlx5_1` 2. Per-GPU JSON mapping: `{"0": "mlx5_0,mlx5_1", "1": "mlx5_2,mlx5_3"}` 3. Path to a JSON file containing the same per-GPU mapping Each JSON value uses the same comma-separated device list format as the shared configuration. ### Usage ### Llama Single Node ```bash theme={null} python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode prefill \ --port 30000 \ --disaggregation-ib-device mlx5_roce0 python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode decode \ --port 30001 \ --base-gpu-id 1 \ --disaggregation-ib-device mlx5_roce0 python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000 ``` ### DeepSeek Multi-Node ```bash theme={null} # prefill 0 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-ib-device ${device_name} \ --disaggregation-mode prefill \ --host ${local_ip} \ --port 30000 \ --trust-remote-code \ --dist-init-addr ${prefill_master_ip}:5000 \ --nnodes 2 \ --node-rank 0 \ --tp-size 16 \ --attn-dp-size 8 \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 # prefill 1 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-ib-device ${device_name} \ --disaggregation-mode prefill \ --host ${local_ip} \ --port 30000 \ --trust-remote-code \ --dist-init-addr ${prefill_master_ip}:5000 \ --nnodes 2 \ --node-rank 1 \ --tp-size 16 \ --attn-dp-size 8 \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 # decode 0 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-ib-device ${device_name} \ --disaggregation-mode decode \ --host ${local_ip} \ --port 30001 \ --trust-remote-code \ --dist-init-addr ${decode_master_ip}:5000 \ --nnodes 2 \ --node-rank 0 \ --tp-size 16 \ --attn-dp-size 8 \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 \ --max-running-requests 128 # decode 1 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-ib-device ${device_name} \ --disaggregation-mode decode \ --host ${local_ip} \ --port 30001 \ --trust-remote-code \ --dist-init-addr ${decode_master_ip}:5000 \ --nnodes 2 \ --node-rank 1 \ --tp-size 16 \ --attn-dp-size 8 \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 \ --max-running-requests 128 ``` ### Advanced Configuration PD Disaggregation with Mooncake supports the following environment variables for fine-grained control over system behavior. #### NVLink Transport Configuration To enable NVLink transport for KV cache transfers with the mooncake backend (recommended for NVL72 deployments), set the following environment variables. Note that auxiliary data transfer will still use TCP as a temporary workaround. ```bash Command theme={null} export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK export MC_FORCE_MNNVL=True ``` To utilize Intra-Node NVLink for KV cache transfers with the Mooncake backend (recommended for A100, H20, H100, etc.), set the following environment variables. Please note that auxiliary data still needs to be transferred via TCP. ```bash theme={null} export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK export MC_INTRANODE_NVLINK=true ``` The `SGLANG_MOONCAKE_CUSTOM_MEM_POOL` environment variable enables the custom memory pool. Supported values are `NVLINK` (or `True`), `BAREX`, and `INTRA_NODE_NVLINK`. #### Prefill Server Configuration
Variable Description Default
**`SGLANG_DISAGGREGATION_THREAD_POOL_SIZE`** Controls the total number of worker threads for KVCache transfer operations per TP rank A dynamic value calculated by int(0.75 \* os.cpu\_count()) // 8, which is limited to be larger than 4 and less than 12 to ensure efficiency and prevent thread race conditions
**`SGLANG_DISAGGREGATION_QUEUE_SIZE`** Sets the number of parallel transfer queues. KVCache transfer requests from multiple decode instances will be sharded into these queues so that they can share the threads and the transfer bandwidth at the same time. If it is set to 1, then we transfer requests one by one according to fcfs strategy `4`
**`SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES`** Opt-in limit for the number of KV cache indices represented by one synchronous all-layer Mooncake transfer batch. Set it to a positive value to slice larger index arrays into ordered sub-batches before contiguous address ranges are formed. The custom-memory-pool layerwise path is unchanged. 0 (disabled)
**`SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT`** Timeout (seconds) for receiving destination KV indices during request initialization `300`
SGLANG\_DISAGGREGATION\_BOOTSTRAP\_ENTRY\_CLEANUP\_INTERVAL Interval (seconds) between cleanups of bootstrap entries 120
If a greater mean TTFT is acceptable, you can `export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600` (10 minutes) to relax the timeout condition. Please be aware that this setting will cause prefill instances to take a longer time to clean up the affected memory resources when a running decode node loses connection. #### Decode Server Configuration
Variable Description Default
**`SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL`** Interval (seconds) between health checks to prefill bootstrap servers `5.0`
**`SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE`** Consecutive heartbeat failures before marking prefill server offline `2`
**`SGLANG_DISAGGREGATION_WAITING_TIMEOUT`** Timeout (seconds) for receiving KV Cache after request initialization `300`
If a greater mean TTFT is acceptable, you can `export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600` (10 minutes) to relax the timeout condition. ## Heterogeneous TP with GPU Staging Buffer When prefill and decode use different tensor parallelism (TP) sizes (e.g., prefill TP=4, decode DP attention with TP=1), the KV cache memory layout differs between the two sides. The **GPU staging buffer** solves this by gathering KV head slices into a contiguous buffer on the prefill side, performing bulk RDMA transfer, then scattering into the correct KV cache pages on the decode side. This provides **2–5x throughput improvement** over the default per-token slice approach at high concurrency and matches homogeneous TP baselines within \~5%. Enable the staging buffer when prefill and decode use **different TP sizes** with the **Mooncake** transfer backend. When both sides use the same TP size, staging is automatically bypassed even if enabled. > **Note:** The staging buffer is designed for non-MLA models (e.g. GQA, MHA). MLA models (e.g. DeepSeek-V2/V3) should not enable this flag. ### Environment Variables
Variable Description Default
SGLANG\_DISAGG\_STAGING\_BUFFER Enable GPU staging buffer for heterogeneous TP KV transfer False
SGLANG\_DISAGG\_STAGING\_POOL\_SIZE\_MB Decode-side ring buffer pool total size in MB 4096
### Usage Example ```bash Command theme={null} # Set staging buffer environment variables on BOTH prefill and decode export SGLANG_DISAGG_STAGING_BUFFER=1 export SGLANG_DISAGG_STAGING_POOL_SIZE_MB=4096 # Prefill with TP=4 python -m sglang.launch_server \ --model-path $MODEL_PATH \ --disaggregation-mode prefill \ --port 30000 \ --tp 4 \ --trust-remote-code \ --disaggregation-ib-device mlx5_1,mlx5_2 # Decode with TP=1 (or DP attention with effective attention TP=1) python -m sglang.launch_server \ --model-path $MODEL_PATH \ --disaggregation-mode decode \ --port 30001 \ --tp 4 \ --attn-dp-size 4 \ --trust-remote-code \ --disaggregation-ib-device mlx5_3,mlx5_4 # Router python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://127.0.0.1:30000 \ --decode http://127.0.0.1:30001 \ --host 0.0.0.0 --port 8000 ``` ## NIXL ### Requirements Install via pip. ```bash theme={null} pip install nixl ``` Or build from source - may be required if you already have UCX installed. ```bash theme={null} git clone https://github.com/ai-dynamo/nixl.git cd nixl pip install . --config-settings=setup-args="-Ducx_path=/path/to/ucx" ``` ### Usage ### Llama Single Node ```bash theme={null} python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode prefill \ --port 30000 \ --disaggregation-transfer-backend nixl python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode decode \ --port 30001 \ --base-gpu-id 1 \ --disaggregation-transfer-backend nixl python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000 ``` ### DeepSeek Multi-Node ```bash theme={null} # prefill 0 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-transfer-backend nixl \ --disaggregation-mode prefill \ --host ${local_ip} \ --port 30000 \ --trust-remote-code \ --dist-init-addr ${prefill_master_ip}:5000 \ --nnodes 2 \ --node-rank 0 \ --tp-size 16 \ --attn-dp-size 8 \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 # prefill 1 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-transfer-backend nixl \ --disaggregation-mode prefill \ --host ${local_ip} \ --port 30000 \ --trust-remote-code \ --dist-init-addr ${prefill_master_ip}:5000 \ --nnodes 2 \ --node-rank 1 \ --tp-size 16 \ --attn-dp-size 8 \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 # decode 0 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-transfer-backend nixl \ --disaggregation-mode decode \ --host ${local_ip} \ --port 30001 \ --trust-remote-code \ --dist-init-addr ${decode_master_ip}:5000 \ --nnodes 2 \ --node-rank 0 \ --tp-size 16 \ --attn-dp-size 8 \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 \ --max-running-requests 128 # decode 1 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-transfer-backend nixl \ --disaggregation-mode decode \ --host ${local_ip} \ --port 30001 \ --trust-remote-code \ --dist-init-addr ${decode_master_ip}:5000 \ --nnodes 2 \ --node-rank 1 \ --tp-size 16 \ --attn-dp-size 8 \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 \ --max-running-requests 128 ``` ### Advanced Configuration #### NIXL Backend Selection By default, NIXL uses the **UCX** backend for KV cache transfers. You can select a different NIXL plugin backend depending on your infrastructure using the environment variable `SGLANG_DISAGGREGATION_NIXL_BACKEND`. Example: `export SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC` **Available backends:** UCX (default), LIBFABRIC, or any installed NIXL plugin. Example usage: ```bash theme={null} export SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode prefill \ --disaggregation-transfer-backend nixl \ --port 30000 ``` ## ASCEND ### Usage Use ascend backend with [memfabric\_hybrid](https://gitcode.com/Ascend/memfabric_hybrid) and ASCEND\_MF\_STORE\_URL being set ```bash Command theme={null} pip install memfabric-hybrid==1.0.0 export ASCEND_MF_STORE_URL="tcp://xxx.xx.xxx.xxx:xxxx" ``` Use mooncake backend, more details can be found in mooncake section. ```bash theme={null} export ENABLE_ASCEND_TRANSFER_WITH_MOONCAKE=true ``` ASCEND\_NPU\_PHY\_ID need to be set in container env ```bash theme={null} export ASCEND_NPU_PHY_ID=xxx ``` ### MIMO Single Node #### prefill ```bash theme={null} # high performance cpu echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 # bind cpu export SGLANG_SET_CPU_AFFINITY=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING # cann source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export STREAMS_PER_DEVICE=32 export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export HCCL_BUFFSIZE=1600 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME=lo export GLOO_SOCKET_IFNAME=lo export SGLANG_NPU_PROFILING=0 export SGLANG_NPU_PROFILING_STAGE="prefill" export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export ASCEND_MF_STORE_URL="tcp://127.0.0.1:24669" export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export SGLANG_DEEPEP_BF16_DISPATCH=0 export ASCEND_USE_FIA=1 python3 -m sglang.launch_server \ --model-path /path/to/MiMo-V2-Flash-w8a8-all-0512 \ --attention-backend ascend \ --device npu \ --tp-size 8 --nnodes 1 --node-rank 0 \ --chunked-prefill-size -1 \ --trust-remote-code --port 10000 \ --host 127.0.0.1 --max-running-requests 16 \ --mem-fraction-static 0.8 \ --disaggregation-mode prefill --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8996 \ --base-gpu-id 0 \ --disable-radix-cache \ --disable-cuda-graph \ --moe-a2a-backend deepep --deepep-mode normal \ # 2>&1 | tee $SGLANG_LOG_PATH ``` #### decode ```bash theme={null} # high performance cpu echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 # bind cpu export SGLANG_SET_CPU_AFFINITY=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING # export ASCEND_LAUNCH_BLOCKING=1 # cann source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export STREAMS_PER_DEVICE=32 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256 export HCCL_BUFFSIZE=1600 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME=lo export GLOO_SOCKET_IFNAME=lo export SGLANG_NPU_PROFILING=0 export SGLANG_NPU_PROFILING_STAGE="prefill" export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export ASCEND_MF_STORE_URL="tcp://127.0.0.1:24669" export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export SGLANG_DEEPEP_BF16_DISPATCH=0 export ASCEND_USE_FIA=1 python3 -m sglang.launch_server \ --model-path /path/to/MiMo-V2-Flash-w8a8-all-0512 \ --attention-backend ascend \ --device npu \ --tp-size 8 --nnodes 1 --node-rank 0 \ --trust-remote-code --port 10001 \ --host 127.0.0.1 --max-running-requests 16 \ --mem-fraction-static 0.8 \ --disaggregation-mode decode --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8996 \ --base-gpu-id 8 \ --disable-radix-cache \ --cuda-graph-bs-decode 1 2 4 8 10 12 14 16 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --moe-a2a-backend deepep --deepep-mode low_latency \ # 2>&1 | tee $SGLANG_LOG_PATH ``` #### router ```bash theme={null} python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://127.0.0.1:10000 \ --decode http://127.0.0.1:10001 \ --host 127.0.0.1 \ --port 9903 \ --health-check-interval-secs 3600 \ --mini-lb \ ``` ### DeepSeek Multi-Node #### Environment ```bash theme={null} echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 export SGLANG_SET_CPU_AFFINITY=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH} export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export STREAMS_PER_DEVICE=32 # IP set to p first node ip export ASCEND_MF_STORE_URL="tcp://XXXXXX:24670" # p node IP P_IP=('XXXXX') # D node IP D_IP=('XXXXX') # enable mlapo export SGLANG_NPU_USE_MLAPO=1 export SGLANG_USE_FIA_NZ=1 export ENABLE_MOE_NZ=1 #export SGLANG_NPU_USE_MULTI_STREAM=1 LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" ``` #### prefill ```bash theme={null} MODEL_PATH=/path/to/deepseekr1_w4a8_pertoken for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export HCCL_BUFFSIZE=2600 export HCCL_SOCKET_IFNAME=lo export GLOO_SOCKET_IFNAME=lo python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \ --tp-size 16 --mem-fraction-static 0.7 --attention-backend ascend --device npu --quantization modelslim \ --disaggregation-transfer-backend ascend --max-running-requests 32 --context-length 8192 --disable-radix-cache \ --chunked-prefill-size -1 --max-prefill-tokens 10240 --moe-a2a-backend deepep --deepep-mode normal \ --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ --attn-dp-size 8 --disable-shared-experts-fusion --dtype bfloat16 NODE_RANK=$i break fi done ``` #### decode ```bash theme={null} MODEL_PATH=/path/to/deepseekr1_w4a8_pertoken for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export HCCL_BUFFSIZE=900 export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=112 export TASK_QUEUE_ENABLE=1 export HCCL_SOCKET_IFNAME=data0.3001 export GLOO_SOCKET_IFNAME=data0.3001 python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ --port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --attn-dp-size 16 \ --mem-fraction-static 0.8 --max-running-requests 448 --attention-backend ascend --device npu --quantization modelslim \ --moe-a2a-backend deepep --deepep-mode low_latency --enable-dp-lm-head \ --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 28 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \ --load-balance-method round_robin NODE_RANK=$i break fi done ``` #### router ```bash theme={null} python -m sglang_router.launch_router --prefill ${P_IP}:8000 \ --decode ${D_IP}:8001 \ --host ${D_IP} --port 6688 \ --pd-disaggregation \ --health-check-interval-secs 3600 \ ``` # Pipeline Parallelism for Long Context Source: https://docs.sglang.io/docs/advanced_features/pipeline_parallelism ## Why Pipeline Parallelism? As Large Language Models (LLMs) scale toward trillion-parameter architectures and "infinite" context windows, the underlying serving infrastructure must evolve toward more granular, cross-node parallelization strategies. While KV cache techniques effectively mitigate redundant computation, they cannot circumvent the prohibitive Time to First Token (TTFT) inherent in ultra-long sequences with extremely large initial Input Token Length (ITL). Although Tensor Parallelism (TP) remains the conventional approach for intra-node scaling, it frequently encounters communication bottlenecks during multi-node deployments. On the other hand, pipeline parallelism only requires cross-node communication at the boundaries of each pipeline stage, which can achieve better computation-communication overlap compared to a large TP. Therefore, it is also a promising parallelization strategy for improving throughput. Detailed analysis can be found in this [blog](https://lmsys.org/blog/2026-01-15-chunked-pipeline/). ## Implementation Refactoring based on Async Communication With Dynamic Chunked Prefill, pipeline parallelism has the potential to reduce the TTFT of long-context inputs. For each request, its input tokens can be partitioned into multiple chunks, each no longer than the chunked prefill size. Different chunks of the same request can be processed simultaneously by different nodes, thus parallelizing the processing and reducing TTFT. SGLang has supported Pipeline Parallelism (#5724) for some time and made it compatible with the PD Disaggregation feature (#8846), but the implementation was not perfect and had significant room for performance improvements. To eliminate this performance hazard, SGLang implements a Micro-batching Event Loop with non-blocking asynchronous peer-to-peer (P2P) communication to overlap GPU computation with CPU metadata processing and PP communication. This ensures that while one micro-batch is being computed on the GPU, the next one is already being prepared and moved into position effectively, ensuring the pipeline remains as saturated as possible. This approach was first proposed in #7979 and has been redesigned and included in #11852. The key mechanisms of the implementation include: * **Decoupled Sync/Async Logic in the Event Loop:** The scheduler uses `async_send` in `_pp_send_pyobj_to_next_stage`. Instead of waiting for a transfer to complete, it returns a `P2PWork` handle. The actual synchronization (`P2PWork.work.wait()`) is deferred until `_pp_commit_comm_work` is called, allowing the CPU to perform other work—like scheduling the next batch or processing metadata—while data is in flight. * **Multi-Stream Execution:** In addition to the main `default_stream`, which serves as the synchronization stream, SGLang utilizes dedicated `forward_stream` and `copy_stream` to execute forward pass GPU computation and Data-to-Host (D2H) memory transfers separately for better overlapping. While `_pp_launch_batch` is executing the current micro-batch on the GPU for the current stage, the CPU processes the previous micro-batch's results using `_pp_process_batch_result`. ## Guidance about Dynamic Chunking ### Why Dynamic Chunking Chunked prefill with a fixed size can cause bubbles in the pipeline, especially when the pp size is large. The main reason behind this phenomenon is that the model has a non-uniform running time, even though each chunk size is identical (brought by the Transformer structure). The larger the prefix sequence length, the longer the running time of the chunk. And these bubbles will be propagated to the next stage, and will significantly degrade the scale efficiency of larger pp ranks. To address this issue, SGLang introduces a dynamic chunking mechanism to predict the optimal size for the next chunk such that it satisfies this condition: Runtime(L + Next Chunk Size) - Runtime(L) = Runtime(Initial Chunk Size) where ***L*** denotes the Prefix Sequence Length. By profiling a series of requests with different ITLs, we model the cumulative runtime as a quadratic function of sequence length. Using this model, we solve the optimal next chunk size for any given prefix length ***L***. Since the computation complexity of the Attention mechanism scales with ***L***, the next chunk size will be progressively reduced as ***L*** grows to maintain an aligned chunk execution time across pipeline stages. Based on this method, the scheduler can predict and dynamically reduce the chunk size during runtime to minimize the bubbles caused by the stage misalignment. To be noticed, the scheduler does not use the raw predicted value. To facilitate efficient KVCache memory management and ensure affinity with hardware execution efficiency, the value is aligned downward to the nearest multiple of max(`--page-size`, 64). ### Chunked Prefill Size and Smoothing Factor When `--enable-dynamic-chunking` is enabled, each chunk size of a sequence is determined dynamically based on the quadratic model that predicts the next chunk size based on the estimated runtime of the initial chunk length. In this case, we use `--chunked-prefill-size` to set up the initial chunk size. When switching to the dynamic chunking mode, the initial chunk size (`--chunked-prefill-size`) should be set to a larger value comparable to the original chunked prefill size, so that there won't be too many chunks. **`SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR`** is an environmental variable that controls the smoothing factor for the dynamic chunking algorithm, defaulting to 0.75. It determines how much the chunk size can change during the prefill phase. A larger value means a more aggressive chunk size change, which may lead to better performance but also to greater chunk size changes (the chunk size at the end may become very small, which could lead to performance degradation) and more total chunks. When it is set to 1, the chunk size will be adjusted strictly based on the aforementioned quadratic model that predicts the next chunk size. A smaller value means a more conservative chunk size change, which may lead to smaller chunk size changes and fewer total chunks. When it is set to 0, the chunk size will not be adjusted dynamically, so it is identical to the traditional way with a fixed chunked prefill size. Due to the variation in hardware, models, and target workloads, a static configuration is seldom optimal across all scenarios. Consequently, achieving peak performance necessitates a degree of hyperparameter tuning when switching to the dynamic chunking mode. **Tuning Guidance for Dynamic Chunked Prefill** * **Step 1 - Iterate to find the optimal fixed chunked prefill size for the targeted PP size**: Different PP sizes for targeted ITL may have different optimal chunked prefill sizes. Therefore, users should iterate to obtain the baseline according to the available resources for scaling. * **Step 2 - Initial Chunk Size Selection for Dynamic Chunking**: Set the initial size to 2× or 3× the optimal fixed chunked prefill size. This reduces the total number of chunks and prevents "tail chunks" from underutilizing hardware. To maintain efficiency for extremely large Input Token Lengths (ITL), the dynamic predictor automatically ensures subsequent chunks are at least 1/4 of this initial size. In addition, it is recommended to use a larger initial chunk size (e.g., 4× the optimal fixed chunked prefill size) for such cases as well. * **Step 3 - Smooth Factor Adjustment**: This factor controls how strictly the chunk size adjusts the prediction given by the quadratic performance fitting model. * 1.0: Follows the model strictly. * **0.6 – 0.85 (Recommended)**: Typical range for the best balance between dynamic scaling and hardware stability. Through experiments, we find that a range between 0.6 and 0.85 typically yields the best performance for dynamic chunking. * 0: Disables dynamic adjustment, reverting to traditional fixed-size chunking. * **Another small optimization tip:** Put the larger partition in the higher PP rank when the layers are not evenly divisible across ranks. It can increase the GPU utilization when a larger PP rank is waiting for the previous stage’s result, hence reducing the bubbles on higher PP ranks. If we take DeepSeek-V3.1 as an example, `SGLANG_PP_LAYER_PARTITION=15,15,15,16` usually performs better than `16,15,15,15`. ## Best Practice for Long Context ### Tuning the Chunked Prefill Size Optimizing the chunked prefill size is crucial for balancing pipeline efficiency and resource utilization. The ideal size depends on factors including model architecture, hardware configuration, and typical input lengths. We recommend starting with a small chunk size, such as 4K, and gradually increasing it until you find the optimal size for your specific use case (Different targeted ITL and PP Sizes may have different optimal chunked prefill sizes. Therefore, users should iterate to obtain the baseline according to the available resources for scaling). Alternatively, you can analyze the hardware capacity and determine the optimal chunk size based on the roofline model. ### Enable Dynamic Chunking and Adjust Smoothing Factor for Ultra-long ITL SGLang also offers a dynamic chunking solution that could further improve performance. This feature is currently an experimental feature that requires a certain amount of tuning experimentation and may not be suitable for all workloads. In addition, fine-tuning the smoothing factor can help optimize performance for specific workloads and model characteristics. ### Case Study on NVIDIA H20 When evaluating pipeline parallelism with fixed chunked prefill sizes from 2K to 16K, experiment results show that a 4K chunk size delivered optimal prefill TTFT performance for the DeepSeek-V3.1, and a 6K chunk size delivered optimal prefill TTFT performance for the Qwen3-235B-A22B-FP8. When enabling dynamic chunking, we first scale the optimal fixed chunked prefill size by a factor of 3 as the initial chunk size. Through experimentation, we found that a multiplier of 2-3 provides an appropriate balance—avoiding excessive initial pipeline bubbles while ensuring that subsequent chunks don't become too small as context length increases. With the default dynamic chunking smoothing factor of 0.75, we performed parameter tuning and determined that a value of 0.65 works optimally with the 12K initial chunk size for the DeepSeek-V3.1, while a value of 0.8 works optimally with the 18K initial chunk size for the Qwen3-235B-A22B-FP8. #### DeepSeek-V3.1 with 128K Input Token Length ```bash Command theme={null} # prefill node 0 (fixed chunked prefill size) python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3.1 --trust-remote-code \ --nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \ --port 30000 --dist-init-addr \ --disable-radix-cache --mem-fraction-static 0.8 \ --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \ --max-running-requests 128 --chunked-prefill-size 4096 ``` ```bash Command theme={null} # prefill node 0 (with dynamic chunking) export SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR=0.65 python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3.1 --trust-remote-code \ --nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \ --port 30000 --dist-init-addr \ --disable-radix-cache --mem-fraction-static 0.8 \ --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \ --max-running-requests 128 --chunked-prefill-size 12288 --enable-dynamic-chunking ``` #### Qwen3-235B-A22B-FP8 with 128K Input Token Length ```bash Command theme={null} # prefill node 0 (fixed chunked prefill size) python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-235B-A22B-FP8 --trust-remote-code \ --nnodes 4 --node-rank 0 --tp 4 --pp-size 8 \ --port 30000 --dist-init-addr \ --disable-radix-cache --mem-fraction-static 0.8 \ --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \ --max-running-requests 128 --chunked-prefill-size 6144 ``` ```bash Command theme={null} # prefill node 0 (with dynamic chunking) export SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR=0.8 python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-235B-A22B-FP8 --trust-remote-code \ --nnodes 4 --node-rank 0 --tp 4 --pp-size 8 \ --port 30000 --dist-init-addr \ --disable-radix-cache --mem-fraction-static 0.8 \ --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \ --max-running-requests 128 --chunked-prefill-size 18432 --enable-dynamic-chunking ``` Note: `--disable-radix-cache` is enabled only for reproducible benchmarking purposes. It is not recommended to use it in production. ## Best Practice for Pipeline Parallelism with PD Disaggregation To be added. Stay tuned for the latest updates on Pipeline Parallelism with PD Disaggregation. # Prefill Context Parallelism Source: https://docs.sglang.io/docs/advanced_features/prefill_cp Prefill context parallelism (CP) distributes the tokens of a prompt across an attention CP group. Each rank computes attention and, for models with an indexer, indexer operations for its local queries, while all-gather on the KV cache makes the required keys and values available. This lets multiple GPUs share the work of processing a long prompt. To compare prefill CP with other common parallelism strategy: Tensor parallelism (TP) splits attention heads and weights, [Decode context parallelism (DCP)](https://docs.sglang.io/docs/advanced_features/dcp) shards the KV cache and its attention workload during decode, while prefill CP (PCP) splits prompt tokens across GPUs. Note that PCP and DCP are independent optimizations for different phases of inference. ## Why prefill context parallelism? Prefill CP offers three benefits: * It partitions work that would otherwise be repeated across ranks, such as indexer computation in DSA models and the DeepSeek V4 series. * It combines naturally with KV-cache sharding to reduce per-rank cache memory. Layer-wise sharding is available through DSA cache LayerSplit; additional sharding techniques are under development. * Combined with expert parallelism (EP), it can reduce communication time by using all-to-all token dispatch and combine instead of TP all-reduce. The benefit depends on the workload and interconnect. ## Architecture ### Strategy interface `ContextParallelStrategy` separates token-layout policy from model execution and attention kernels. Server argument resolution initializes one strategy per process. Per-forward state lives in `ForwardBatch.attn_cp_metadata`, rather than in the model's token-layout logic. ### Zigzag strategy For a CP group of size `C`, split each request's newly extended tokens into `2C` contiguous blocks. Rank `r` receives block `r` and block `2C - 1 - r`. The split restarts for every request in a batch. For example, with CP=4 and 16 new tokens, each block has two tokens: ```mermaid theme={null} flowchart TB S["One request, original token order
B0: 0–1 · B1: 2–3 · B2: 4–5 · B3: 6–7
B4: 8–9 · B5: 10–11 · B6: 12–13 · B7: 14–15"] S --> R0["Rank 0
B0 + B7
tokens 0–1, 14–15"] S --> R1["Rank 1
B1 + B6
tokens 2–3, 12–13"] S --> R2["Rank 2
B2 + B5
tokens 4–5, 10–11"] S --> R3["Rank 3
B3 + B4
tokens 6–7, 8–9"] ``` Pairing an early, cheaper block with a late, more expensive block balances causal attention work while retaining contiguous query blocks. Lengths need not divide evenly: early blocks receive the remainder, and communication buffers are padded as needed. The metadata records separate query and KV lengths for the early and late blocks, including any cached prefix. Each block therefore attends to the correct causal history, not just the other tokens assigned to its rank. ### Interleave strategy For CP size `C`, rank `r` receives flattened token indices `r, r+C, r+2C, ...`. The index runs across the batch's newly extended tokens; it does not reset at request boundaries. Position IDs keep their original values. With CP=4 and 16 new tokens: ```mermaid theme={null} flowchart TB S["Flattened new tokens in batch order
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15"] S --> R0["Rank 0
0, 4, 8, 12"] S --> R1["Rank 1
1, 5, 9, 13"] S --> R2["Rank 2
2, 6, 10, 14"] S --> R3["Rank 3
3, 7, 11, 15"] ``` This gives each rank queries spread across the context. For a batch containing requests of lengths 5 and 3, rank 0 owns flattened indices 0 and 4, while rank 1 owns indices 1 and 5; index 5 is the first new token of the second request. Per-request sharding metadata preserves those boundaries and each query's causal extent. ### Attention backend integration | Path | Integration | | - | - | | Zigzag with FlashAttention | Gather and reorder new KV; execute attention for the early and late query blocks with their respective sequence lengths. | | Zigzag with `trtllm_mha` | Materialize KV, build zigzag page tables, and execute a combined variable-length attention call. | | Interleave with DSA | Shard query/indexer metadata by request, gather indexer keys and MLA KV as required, and execute sparse attention in the backend. | | DeepSeek V4 interleave | Uses model-specific sparse-attention integration; the CP regression suite includes DeepSeek-V4-Flash on B200. Do not infer its backend requirements from the dense zigzag path. | The CP and attention-backend compatibility matrix is being expanded, with support for more combinations planned. ## Compositions ### Prefill CP × TP, DP, and EP Attention TP splits heads within each CP partition; attention DP separates request batches. Dense FFNs and MoE layers can use a different layout, bridged by the layer boundaries between stages (see [Layer boundaries](/docs/developer_guide/layer_boundary)). The Qwen3 example in [Usage and Examples](#usage-and-examples) combines CP=2 with attention TP=2 and EP=4. For interleave DSA on DeepSeek V3.2 and the GLM-5 series, keep DP=1 and use the resolved attention CP=TP topology. ### Prefill CP × pipeline parallelism The eager runner preserves CP-local hidden states between pipeline stages and gathers on the final stage. PP can place stages on different nodes while keeping each DSA attention CP group within one node; it does not remove the single-machine DSA CP restriction. ### Prefill CP × speculative decoding Prefill CP can coexist with supported speculative-decoding configurations. The CP runner handles aligned speculative hidden-state inputs and gathers supported auxiliary outputs. ### Prefill CP × CUDA graphs The shared CP breakable-prefill-graph path currently requires `zigzag`, the `trtllm_mha` prefill backend, PP=1, and attention CP size equal to TP size. It uses CP-local static input buffers and selects capture buckets large enough for each rank's padded input. Support for combining other attention backends with prefill CP and Breakable CUDA Graph is under development. ### Prefill CP × PD disaggregation and DSA cache LayerSplit In supported DSA deployments, `--enable-dsa-cache-layer-split` distributes GPU KV/indexer cache layers across CP ranks to reduce per-rank cache memory. Each rank owns a range of layers and uses scratch storage for remote-layer data. This is layer-wise cache ownership, separate from the interleave token assignment used for computation, and is not applied to draft workers. LayerSplit requires a PD prefill worker (`--disaggregation-mode prefill`), `--enable-prefill-cp --cp-strategy interleave`, and PP=1. It currently supports the `mooncake` and `mooncake_tcp` transfer backends. Do not enable LayerSplit on decode or non-PD workers; the decode side receives full cache shards through PD transfer. Use the [PD disaggregation guide](https://docs.sglang.io/docs/advanced_features/pd_disaggregation) and model-specific launch recipes to configure both sides of the transfer; enabling prefill CP alone does not configure a PD deployment. ## Usage and Examples Use NVIDIA CUDA GPUs and select a strategy supported by your model and attention backend. The examples below target a single Linux host with Hopper GPUs and an SGLang version that includes the selected backend. | Argument | Default | Description | | - | - | - | | `--enable-prefill-cp` | `False` | Enable context-parallel prefill. | | `--cp-strategy` | Unset | Required when enabling prefill CP. Choose `zigzag` or `interleave`. | | `--attn-cp-size` | `1` | Number of ranks in an attention CP group. Use a value greater than 1 to distribute work; model-specific argument resolution may adjust it. | CP subdivides the existing TP world; it does not add another GPU multiplier. With attention DP enabled, the attention topology is: ```text theme={null} TP world size = attention DP size × attention CP size × attention TP size Example: --tp 4 --dp 1 --attn-cp-size 2 → one attention DP group, CP=2, attention TP=2 ``` The server requires `--tp` to be divisible by `--dp × --attn-cp-size`. The attention layout does not by itself specify the dense FFN or MoE layout; those use their own parallelism settings. For DSA models, such as the GLM-5 series and DeepSeek V3.2, use `interleave`; `zigzag` is currently unavailable. Do not use a smaller `--attn-cp-size` to request hybrid attention TP/CP for DSA models. Broader compatibility is planned for future support. ### Zigzag example: GQA with attention TP and CP This Qwen3 configuration uses four Hopper GPUs, with attention TP=2, CP=2, and EP=4 for the MoE layers. The model is a Hugging Face repository ID passed to `--model-path`. ```bash theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3-30B-A3B-FP8 \ --tp 4 \ --ep 4 \ --attn-cp-size 2 \ --enable-prefill-cp \ --cp-strategy zigzag \ --attention-backend fa3 ``` ### Interleave example: DSA This GLM-5.2 configuration uses eight H200 GPUs with attention CP=8 and attention TP=1. The DSA backend is selected automatically for this model; the command makes that choice explicit. ```bash theme={null} python -m sglang.launch_server \ --model-path zai-org/GLM-5.2-FP8 \ --tp 8 \ --attn-cp-size 8 \ --enable-prefill-cp \ --cp-strategy interleave \ --attention-backend dsa \ --trust-remote-code \ --mem-fraction-static 0.85 ``` For model-specific deployment combinations, see [context parallelism in the GLM-5.3 cookbook](https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3#3-5-context-parallelism). ## References * [CP strategies and metadata](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/layers/cp): strategy interface, zigzag/interleave layouts, padding, and graph helpers. * [Prefill Context Parallelism Roadmap](https://github.com/sgl-project/sglang/issues/21788). * [Prefill Context Parallelism Refactor Design](https://github.com/sgl-project/sglang/issues/27252). # Quantization Source: https://docs.sglang.io/docs/advanced_features/quantization SGLang supports various quantization methods, including offline quantization and online dynamic quantization. Offline quantization loads pre-quantized model weights directly during inference. This is required for quantization methods such as GPTQ and AWQ, which collect and pre-compute various statistics from the original weights using the calibration dataset. Online quantization dynamically computes scaling parameters—such as the maximum/minimum values of model weights—during runtime. Like NVIDIA FP8 training's [delayed scaling](https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html#Mixed-precision-training-with-FP8) mechanism, online quantization calculates the appropriate scaling factors on-the-fly to convert high-precision weights into a lower-precision format. **Note: For better performance, usability and convenience, offline quantization is recommended over online quantization.** If you use a pre-quantized model, **do not add `--quantization` to enable online quantization at the same time**. For popular pre-quantized models, please visit [Unsloth](https://huggingface.co/unsloth), [NVIDIA ModelOpt](https://huggingface.co/collections/nvidia/inference-optimized-checkpoints-with-model-optimizer) or [NeuralMagic](https://huggingface.co/collections/neuralmagic) collections on HF for some popular quality validated quantized models. Quantized models must be validated via benchmarks post-quantization to guard against abnormal quantization loss regressions. ## Platform Compatibility The following table summarizes quantization method support across NVIDIA and AMD GPUs, Ascend NPUs.
Method NVIDIA GPUs AMD GPUs (MI300X/MI325X/MI350X) Ascend NPUs (A2/A3 Series, 950PR/DT Series) Notes
fp8 Yes Yes WIP Aiter or Triton backend on AMD
mxfp4 Yes Yes Yes (950PR/DT Series) On GPU: requires CDNA3/CDNA4 with MXFP support (uses Aiter). On Ascend NPU (950PR/DT Series): W4A4 MXFP4 for Qwen3 dense and MoE LLMs (MXFP4 weights + activations) — dense models support online dual-level MXFP4; offline W4A4\_MXFP4 dense and MoE checkpoints (single-level) are auto-detected via modelslim. On Intel GPUs (XPU): W4A16 MoE experts on Xe2/BMG via sgl-kernel-xpu, enabled automatically with --device xpu (see Intel GPUs (XPU))
mxfp8 No No Yes (950PR/DT Series for Diffusion, LLM Dense Linear and LLM MoE) Ascend NPU only; online + offline MXFP8 for Diffusion models (e.g., Wan2.2), LLM Dense Linear, and LLM MoE (FusedMoE, e.g. Qwen3-30B-A3B) on 950PR/DT Series; uses CANN npu\_dynamic\_mx\_quant / npu\_quant\_matmul (dense) and npu\_grouped\_matmul\_swiglu\_quant\_v2 / npu\_grouped\_matmul (MoE) kernels
mxfp\_w4a8 No No Yes (950PR/DT Series) Ascend NPU only; online W4A8 for Qwen3 dense LLM (MXFP4 weights + MXFP8 activations) on 950PR/DT Series; offline W4A8\_MXFP dense and MoE checkpoints are auto-detected via modelslim
blockwise\_int8 Yes Yes No Triton-based, works on both platforms
w8a8\_int8 Yes Yes No
w8a8\_fp8 Yes Yes No Aiter or Triton FP8 on AMD
awq Yes Yes Yes Uses a JIT-compiled CUDA kernel on NVIDIA, Triton dequantize on AMD. Uses CANN kernels on Ascend
gptq No No Yes Removed on NVIDIA and AMD GPUs — use gptq\_marlin instead. Uses CANN kernels on Ascend. Still supported on Intel CPUs with AMX
compressed-tensors Yes Yes Partial Aiter paths for FP8/MoE on AMD. Uses CANN kernels on Ascend, FP8 not supported yet
quark Yes Yes No AMD Quark quantization; Aiter GEMM paths on AMD
auto-round Yes Yes Partial Platform-agnostic (Intel auto-round). Uses CANN kernels on Ascend
quark\_int4fp8\_moe No Yes No AMD-only; online INT4-to-FP8 MoE quantization (CDNA3/CDNA4)
awq\_marlin Yes No No Marlin kernels are CUDA-only
gptq\_marlin Yes No No Marlin kernels are CUDA-only
gguf Yes No Yes CUDA kernels in sgl-kernel; Ascend uses CPU pre-dequantization at load time
modelopt / modelopt\_fp8 Yes (Hopper/SM90+) No No NVIDIA ModelOpt; requires NVIDIA hardware
modelopt\_fp4 Yes (SM80-SM90 via Marlin; SM100+ native FP4) No No NVIDIA ModelOpt; use Marlin W4A16 fallback on Ampere/Hopper and native FP4 backends on Blackwell; supports load-time BF16/FP16/FP8 MoE conversion with per-tensor FP32 activation scales
nvfp4\_online Yes (Blackwell/SM100 or SM103) No No Online MoE-only NVFP4 weight quantization with per-token FP32 activation scales for BF16/FP16/FP8 checkpoints; use modelopt\_fp4 for per-tensor FP32 activation scales; requires flashinfer\_trtllm or flashinfer\_trtllm\_routed
petit\_nvfp4 No Yes (MI250/MI300X/MI325X) No Enables NVFP4 on ROCm via Petit; use modelopt\_fp4 on NVIDIA Blackwell. Auto-selected when loading NVFP4 models on AMD. See LMSYS blog and AMD ROCm blog.
bitsandbytes Yes Experimental No Depends on bitsandbytes ROCm support
modelslim No No Yes Ascend quantization; Uses CANN kernels
On AMD, several of these methods use [Aiter](https://github.com/ROCm/aiter) for acceleration -- set `SGLANG_USE_AITER=1` where noted. See [AMD GPU setup](../hardware-platforms/amd_gpu) for installation and configuration details. On Ascend, various layers quantization configurations are supported, see [Ascend NPU quantization](../hardware-platforms/ascend-npus/optimization/quantization) for details. ## GEMM Backends for FP4/FP8 Quantization Backend selection applies to **blockwise FP8**, **MXFP8** (dense linear), and **NVFP4** GEMM. When running offline or online FP8 or FP4 quantized models, you can select the GEMM backend via `--fp8-gemm-backend` and `--fp4-gemm-backend`. ### `--fp8-gemm-backend` (Blockwise FP8 GEMM)
Backend Hardware Description
auto All Auto-selects based on hardware
deep\_gemm SM90, SM100 JIT-compiled; enabled when DeepGEMM is installed
flashinfer\_trtllm SM100 FlashInfer TensorRT-LLM backend; optimal for low-latency
flashinfer\_cutlass SM100/120 FlashInfer CUTLASS groupwise FP8 GEMM
flashinfer\_deepgemm SM90 Uses swapAB optimization for small M dimensions in decoding
cutlass SM120 sgl-kernel CUTLASS
triton All Fallback; widely compatible
aiter ROCm AMD AITER backend
**`auto` selection order:** 1) DeepGEMM (SM90/SM100, installed); 2) FlashInfer TRTLLM (SM100, FlashInfer available); 3) CUTLASS (SM120); 4) AITER (AMD); 5) Triton (fallback). **MXFP8 dense linear:** `auto` uses `flashinfer_cutlass` on SM100 (else `triton`). `flashinfer_cutlass` is fastest on most shapes; `flashinfer_trtllm` is faster only at small M. ### `--fp4-gemm-backend` (NVFP4 GEMM)
Backend Hardware Description
auto SM80+ Auto-selects: flashinfer\_cutedsl on SM100; marlin on SM80-SM90; flashinfer\_cutlass otherwise (including SM120)
flashinfer\_cutlass SM100/120 FlashInfer CUTLASS backend
flashinfer\_cudnn SM100/120 (CUDA 13+, cuDNN 9.15+) FlashInfer cuDNN backend
flashinfer\_cutedsl SM100 FlashInfer CuTe DSL backend
flashinfer\_trtllm SM100 FlashInfer TensorRT-LLM backend
marlin SM80-SM90 Weight-only W4A16 fallback for NVFP4 checkpoints
On SM80-SM90, `auto` selects Marlin for NVFP4. NVFP4 GEMM requires FlashInfer to be installed. ## Offline Quantization To load already quantized models, simply load the model weights and config. **Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be parsed from the downloaded Hugging Face or msModelSlim config. For example, DeepSeek V3/R1 models are already in FP8, so do not add redundant parameters.** ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 \ --port 30000 --host 0.0.0.0 ``` Take note, if your model is **per-channel quantized (INT8 or FP8) with per-token dynamic quantization activation**, you can opt to include `--quantization w8a8_int8` or `--quantization w8a8_fp8` to invoke the corresponding CUTLASS int8\_kernel or fp8\_kernel in sgl-kernel. This action will ignore the Hugging Face config's quantization settings. For instance, with `neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic`, if you execute with `--quantization w8a8_fp8`, the system will use the `W8A8Fp8Config` from SGLang to invoke the sgl-kernel, rather than the `CompressedTensorsConfig` for vLLM kernels. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic \ --quantization w8a8_fp8 \ --port 30000 --host 0.0.0.0 ``` ### Examples of Offline Model Quantization #### Using [Unsloth](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide) We strongly suggest the use of Unsloth to quantize and load the model. Please refer to [SGLang Deployment & Inference Guide with Unsloth](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide). #### Using [auto-round](https://github.com/intel/auto-round) ```bash Command theme={null} # Install pip install auto-round ``` * LLM quantization ```py Example theme={null} # for LLM from auto_round import AutoRound model_id = "meta-llama/Llama-3.2-1B-Instruct" quant_path = "Llama-3.2-1B-Instruct-autoround-4bit" # Scheme examples: "W2A16", "W3A16", "W4A16", "W8A16", "NVFP4", "MXFP4" (no real kernels), "GGUF:Q4_K_M", etc. scheme = "W4A16" format = "auto_round" autoround = AutoRound(model_id, scheme=scheme) autoround.quantize_and_save(quant_path, format=format) # quantize and save ``` * VLM quantization ```py Example theme={null} # for VLMs from auto_round import AutoRoundMLLM model_name = "Qwen/Qwen2-VL-2B-Instruct" quant_path = "Qwen2-VL-2B-Instruct-autoround-4bit" scheme = "W4A16" format = "auto_round" autoround = AutoRoundMLLM(model_name, scheme) autoround.quantize_and_save(quant_path, format=format) # quantize and save ``` * Command Line Usage (Gaudi/CPU/Intel GPU/CUDA) ```bash Command theme={null} auto-round \ --model meta-llama/Llama-3.2-1B-Instruct \ --bits 4 \ --group_size 128 \ --format "auto_round" \ --output_dir ./tmp_autoround ``` * SGlang API Usage (CPU/CUDA) ```python Example theme={null} from sglang.srt.configs.load_config import LoadConfig from sglang.srt.configs.model_config import ModelConfig from sglang.srt.model_loader.loader import get_model_loader from sglang.srt.configs.device_config import DeviceConfig # Configure model with inc quantization and saving model_config = ModelConfig( model_path="meta-llama/Llama-3.2-3B-Instruct", quantization="auto-round-int8", trust_remote_code=True, ) load_config = LoadConfig( inc_save_path="./quantized_model", ) device_config = DeviceConfig(device="cpu") # Load and quantize the model model_loader = get_model_loader(load_config, model_config) quantized_model = model_loader.load_model( model_config=model_config, device_config=device_config, ) ``` * known issues Several limitations currently affect offline quantized model loading in sglang, These issues might be resolved in future updates of sglang. If you experience any problems, consider using Hugging Face Transformers as an alternative. 1. Mixed-bit Quantization Limitations Mixed-bit quantization is not fully supported. Due to vLLM's layer fusion (e.g., QKV fusion), applying different bit-widths to components within the same fused layer can lead to compatibility issues. 2. Limited Support for Quantized MoE Models Quantized MoE models may encounter inference issues due to kernel limitations (e.g., lack of support for mlp.gate layer quantization). please try to skip quantizing these layers to avoid such errors. 3. Limited Support for Quantized VLMs Qwen2.5-VL-7B auto\_round:auto\_gptq format: Accuracy is close to zero. GPTQ format: Fails with: ```text Output theme={null} The output size is not aligned with the quantized weight shape ``` auto\_round:auto\_awq and AWQ format: These work as expected. 4. Limited Support for SGlang API Usage SGlang API Usage only supports `auto-round-int8` quantization method now, more quantization methods are on the way. * CPU serving AutoRound INT4 checkpoints (both `auto_round:auto_gptq` and `auto_round:auto_awq` packing formats) can be served on Intel CPUs with AMX support: ```bash theme={null} SGLANG_USE_CPU_ENGINE=1 python3 -m sglang.launch_server \ --model-path OPEA/Qwen2.5-0.5B-Instruct-int4-sym-inc \ --quantization auto-round \ --device cpu --trust-remote-code ``` The current SGLang CPU backend supports only 4-bit AutoRound checkpoints on Intel AMX. Other AutoRound bit-widths and non-AMX CPU backends are not supported by this path. #### Using [GPTQModel](https://github.com/ModelCloud/GPTQModel) ```bash Command theme={null} # install pip install gptqmodel --no-build-isolation -v ``` ```py Example theme={null} from datasets import load_dataset from gptqmodel import GPTQModel, QuantizeConfig model_id = "meta-llama/Llama-3.2-1B-Instruct" quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit" calibration_dataset = load_dataset( "allenai/c4", data_files="en/c4-train.00001-of-01024.json.gz", split="train" ).select(range(1024))["text"] quant_config = QuantizeConfig(bits=4, group_size=128) # quantization config model = GPTQModel.load(model_id, quant_config) # load model model.quantize(calibration_dataset, batch_size=2) # quantize model.save(quant_path) # save model ``` #### Using [LLM Compressor](https://github.com/vllm-project/llm-compressor/) ```bash Command theme={null} # install pip install llmcompressor ``` Here, we take quantize `meta-llama/Meta-Llama-3-8B-Instruct` to `FP8` as an example to elaborate on how to do offline quantization. ```python Example theme={null} from transformers import AutoTokenizer from llmcompressor.transformers import SparseAutoModelForCausalLM from llmcompressor.transformers import oneshot from llmcompressor.modifiers.quantization import QuantizationModifier # Step 1: Load the original model. MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct" model = SparseAutoModelForCausalLM.from_pretrained( MODEL_ID, device_map="auto", torch_dtype="auto") tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) # Step 2: Perform offline quantization. # Step 2.1: Configure the simple PTQ quantization. recipe = QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"]) # Step 2.2: Apply the quantization algorithm. oneshot(model=model, recipe=recipe) # Step 3: Save the model. SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic" model.save_pretrained(SAVE_DIR) tokenizer.save_pretrained(SAVE_DIR) ``` Then, you can directly use the quantized model with `SGLang`, by using the following command: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path $PWD/Meta-Llama-3-8B-Instruct-FP8-Dynamic \ --port 30000 --host 0.0.0.0 ``` #### Using [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer) NVIDIA Model Optimizer (ModelOpt) provides advanced quantization techniques optimized for NVIDIA hardware. **Offline vs. Online Quantization:** SGLang supports two modes for ModelOpt. * **Offline Quantization (pre-quantized):** * **Usage:** Download a pre-quantized model from Hugging Face or run `hf_ptq.py` once to create a new quantized checkpoint. Then load this quantized checkpoint. * **Pros:** Fast server startup, quantization can be validated before deployment, efficient resource usage. * **Cons:** Requires an extra preparation step. * **Online Quantization (quant and serve):** * **Usage:** Load a standard BF16/FP16 model and add a flag. The engine applies quantization *on startup*. * **Pros:** Convenient (no new checkpoint needed). * **Cons:** **High startup time**, increases VRAM usage during initialization (risk of OOM). The following sections guide you through using the Offline path: loading pre-quantized models or creating your own checkpoints. ##### Using Pre-Quantized Checkpoints If a model is already quantized (e.g., from Hugging Face), you can load it directly. * **FP8 Models:** Use `--quantization modelopt_fp8`. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/Llama-3.1-8B-Instruct-FP8 \ --quantization modelopt_fp8 \ --port 30000 ``` * **FP4 Models:** Use `--quantization modelopt_fp4`. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/Llama-3.3-70B-Instruct-NVFP4 \ --quantization modelopt_fp4 \ --port 30000 ``` ##### Creating Your Own Quantized Checkpoints If a pre-quantized checkpoint is not available for your model, you can create one using NVIDIA Model Optimizer's `hf_ptq.py` script. **Why quantize?** * Reduce VRAM usage * Higher throughput and lower latency * More flexible deployment (on smaller GPUs) **What can be quantized?** * The entire model * MLP layers only * KV cache **Key options in `hf_ptq.py`:** `--qformat`: Quantization formats `fp8`, `nvfp4`, `nvfp4_mlp_only` `--kv_cache_qformat`: KV cache quantization format (default: `fp8`) **Note:** The default `kv_cache_qformat` may not be optimal for all use cases. Consider setting this explicitly. **Hardware requirements:** Hopper and higher are recommended. Insufficient GPU memory may cause weight offloading, resulting in extremely long quantization time. For detailed usage and supported model architectures, see [NVIDIA Model Optimizer LLM PTQ](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq). SGLang includes a streamlined workflow for quantizing models with ModelOpt and automatically exporting them for deployment. ##### Installation First, install ModelOpt: ```bash Command theme={null} pip install nvidia-modelopt ``` ##### Quantization and Export Workflow SGLang provides an example script that demonstrates the complete ModelOpt quantization and export workflow. Run from the SGLang repository root (see [modelopt\_quantize\_and\_export.py](https://github.com/sgl-project/sglang/blob/main/examples/usage/modelopt_quantize_and_export.py)): ```bash Command theme={null} # Quantize and export a model using ModelOpt FP8 quantization python examples/usage/modelopt_quantize_and_export.py quantize \ --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \ --export-dir ./quantized_tinyllama_fp8 \ --quantization-method modelopt_fp8 # For FP4 quantization (requires Blackwell GPU) python examples/usage/modelopt_quantize_and_export.py quantize \ --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \ --export-dir ./quantized_tinyllama_fp4 \ --quantization-method modelopt_fp4 ``` ##### Available Quantization Methods * `modelopt_fp8`: FP8 quantization with optimal performance on NVIDIA Hopper and Blackwell GPUs * `modelopt_fp4`: FP4 quantization with optimal performance on Nvidia Blackwell GPUs ##### Python API Usage You can also use ModelOpt quantization programmatically: ```python Example theme={null} import sglang as sgl from sglang.srt.configs.device_config import DeviceConfig from sglang.srt.configs.load_config import LoadConfig from sglang.srt.configs.model_config import ModelConfig from sglang.srt.model_loader.loader import get_model_loader # Configure model with ModelOpt quantization and export model_config = ModelConfig( model_path="TinyLlama/TinyLlama-1.1B-Chat-v1.0", quantization="modelopt_fp8", # or "modelopt_fp4" trust_remote_code=True, ) load_config = LoadConfig( modelopt_export_path="./exported_model", modelopt_checkpoint_save_path="./checkpoint.pth", # optional, fake quantized checkpoint ) device_config = DeviceConfig(device="cuda") # Load and quantize the model (export happens automatically) model_loader = get_model_loader(load_config, model_config) quantized_model = model_loader.load_model( model_config=model_config, device_config=device_config, ) ``` ##### Deploying Quantized Models After quantization and export, you can deploy the model with SGLang: ```bash Command theme={null} # Deploy the exported quantized model python -m sglang.launch_server \ --model-path ./quantized_tinyllama_fp8 \ --quantization modelopt \ --port 30000 --host 0.0.0.0 ``` Or using the Python API (use the same path as `modelopt_export_path` from the quantize step): ```python Example theme={null} import sglang as sgl def main(): # Deploy exported ModelOpt quantized model # Path must match modelopt_export_path from quantize step (e.g., ./exported_model) llm = sgl.Engine( model_path="./exported_model", quantization="modelopt", ) # Run inference prompts = [ "Hello, how are you?", "What is the capital of France?", ] sampling_params = { "temperature": 0.8, "top_p": 0.95, "max_new_tokens": 100, } outputs = llm.generate(prompts, sampling_params) for i, output in enumerate(outputs): print(f"Prompt: {prompts[i]}") print(f"Output: {output['text']}") if __name__ == "__main__": main() ``` ##### Advanced Features **Checkpoint Management**: Save and restore fake quantized checkpoints for reuse: ```bash Command theme={null} # Save the fake quantized checkpoint during quantization python examples/usage/modelopt_quantize_and_export.py quantize \ --model-path meta-llama/Llama-3.2-1B-Instruct \ --export-dir ./quantized_model \ --quantization-method modelopt_fp8 \ --checkpoint-save-path ./my_checkpoint.pth # The checkpoint can be reused for future quantization runs and skip calibration ``` **Export-only Workflow**: If you have a pre-existing fake quantized ModelOpt checkpoint, you can export it directly. See [LoadConfig](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/configs/load_config.py) for the full API: ```python Example theme={null} from sglang.srt.configs.device_config import DeviceConfig from sglang.srt.configs.load_config import LoadConfig from sglang.srt.configs.model_config import ModelConfig from sglang.srt.model_loader.loader import get_model_loader model_config = ModelConfig( model_path="meta-llama/Llama-3.2-1B-Instruct", quantization="modelopt_fp8", trust_remote_code=True, ) load_config = LoadConfig( modelopt_checkpoint_restore_path="./my_checkpoint.pth", modelopt_export_path="./exported_model", ) # Load and export the model (DeviceConfig defaults to device="cuda") model_loader = get_model_loader(load_config, model_config) model_loader.load_model(model_config=model_config, device_config=DeviceConfig()) ``` ##### Benefits of ModelOpt * **Hardware Optimization**: Specifically optimized for NVIDIA GPU architectures * **Advanced Quantization**: Supports cutting-edge FP8 and FP4 quantization techniques * **Seamless Integration**: Automatic export to HuggingFace format for easy deployment * **Calibration-based**: Uses calibration datasets for optimal quantization quality * **Production Ready**: Enterprise-grade quantization with NVIDIA support #### Using [ModelSlim](https://gitcode.com/Ascend/msmodelslim) MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware. * **Installation** ```bash Command theme={null} # Clone repo and install msmodelslim: git clone https://gitcode.com/Ascend/msmodelslim.git cd msmodelslim bash install.sh ``` * **LLM quantization** Download the original floating-point weights of the large model. Taking Qwen3-32B as an example, you can go to [Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) to obtain the original model weights. Then install other dependencies (related to the model, refer to the huggingface model card). > Note: You can find pre-quantized validated models on [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech). *Traditional quantification methods require the preparation of calibration data files (`.jsonl` formats) for calibration in the quantification process.* ````bash Command theme={null} Qwen3-32B/ # floating-point model downloaded from official HF (or modelscope) repo msmodelslim/ # msmodelslim repo |----- lab_calib # calibration date folder (put your dataset here in ```.jsonl``` format or use pre-prepared ones) |----- some file (such as laos_calib.jsonl) |----- lab_practice # best practice folder with configs for quantization |----- model folder (such as qwen3_5_moe folder) # folder with quantization configs |----- quant_config (such as qwen3_5_moe_w8a8.yaml) # quantization config |----- another folders output_folder/ # generated by below command |----- quant_model_weights-00001-of-0001.safetensors # quantized weights |----- quant_model_description.json # file with description of the quantization methods for each layer (```W4A4_DYNAMIC```, etc.) |----- another files (such as config.json, tokenizer.json, etc.) ```` Run quantization using one-click quantization (recommended): ```bash Command theme={null} msmodelslim quant \ --model_path ${MODEL_PATH} \ --save_path ${SAVE_PATH} \ --device npu:0,1 \ --model_type Qwen3-32B \ --quant_type w8a8 \ --trust_remote_code True ``` * **Usage Example** ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path $PWD/Qwen3-32B-w8a8 \ --port 30000 --host 0.0.0.0 ``` * **Available Quantization Methods**: * [x] `W4A4_DYNAMIC` linear with online quantization of activations * [x] `W8A8` linear with offline quantization of activations * [x] `W8A8_DYNAMIC` linear with online quantization of activations * [x] `W4A4_DYNAMIC` MOE with online quantization of activations * [x] `W4A4_MXFP4` MOE with dynamic MXFP4 activation quantization * [x] `W4A8_DYNAMIC` MOE with online quantization of activations * [x] `W4A8_MXFP` MOE with dynamic MXFP8 activation quantization * [x] `W8A8_DYNAMIC` MOE with online quantization of activations * [ ] `W4A8` linear TBD * [ ] `W4A16` linear TBD * [ ] `W48A16` linear TBD * [ ] `W4A16` MoE in progress * [ ] `W8A16` MoE in progress * [ ] `KV Cache` in progress * [ ] `Attention` in progress For more detailed examples of quantization of models, as well as information about their support, see the [examples](https://gitcode.com/Ascend/msmodelslim/blob/master/example/README.md) section in ModelSLim repo. ## Online Quantization To enable online quantization, you can simply specify `--quantization` in the command line. For example, you can launch the server with the following command to enable `FP8` quantization for model `meta-llama/Meta-Llama-3.1-8B-Instruct`: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --quantization fp8 \ --port 30000 --host 0.0.0.0 ``` Our team is working on supporting more online quantization methods. SGLang will soon support methods including but not limited to `["awq", "gptq", "marlin", "gptq_marlin", "awq_marlin", "bitsandbytes", "gguf"]`. ### `nvfp4_online` online quantization method Use `--quantization nvfp4_online` to convert eligible BF16, FP16, or FP8 MoE expert weights to NVFP4 at load time with per-token FP32 activation scales. Use `modelopt_fp4` for serialized NVFP4 checkpoints or the same load-time conversion with per-tensor FP32 activation scales. The design separates weight quantization from activation scaling: * **Weights:** SGLang quantizes each eligible MoE expert weight tensor as it is loaded, using standard 2D NVFP4 weight quantization. The generated NVFP4 weights use static E4M3 block scales plus static per-tensor FP32 scales derived from the weight amax. For gated MoE experts, the w1/w3 pair shares one per-tensor FP32 scale. * **Activations:** FlashInfer computes and propagates one FP32 scale per token at runtime. Backends that accept one per-tensor FP32 activation scale must use `modelopt_fp4`. * **FP8 checkpoints:** If an eligible expert weight is stored as FP8, SGLang first dequantizes that tensor with the checkpoint scale and then requantizes it to NVFP4 during loading. * **Other layers:** Dense linear layers stay in their source checkpoint precision or checkpoint quantization path. Only `--moe-runner-backend flashinfer_trtllm` and `--moe-runner-backend flashinfer_trtllm_routed` are supported. If `--moe-runner-backend` is omitted, SGLang selects `flashinfer_trtllm`. Tensor parallelism is supported; activation per-token scales are computed locally on each TP rank, while online weight quantization still uses the loaded expert tensor's per-tensor amax-derived FP32 scale. FlashInfer TRTLLM MoE backends disable shared-expert fusion, so online quantization applies to routed MoE experts while shared experts stay in the checkpoint precision. Both online modes honor `SGLANG_FP4_IGNORED_LAYERS`; for FP8 source checkpoints, listed experts remain FP8 instead of being converted to NVFP4. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 \ --tp-size 2 \ --ep-size 2 \ --quantization nvfp4_online \ --port 30000 --host 0.0.0.0 ``` ### `quark_int4fp8_moe` online quantization method SGLang running on AMD GPUs (CDNA3 or CDNA4 architecture) supports the quantization method `--quantization quark_int4fp8_moe`, that will replace [MoE layers](https://github.com/sgl-project/sglang/blob/v0.4.8/python/sglang/srt/layers/moe/fused_moe_triton/layer.py#L271) originally in high precision (bfloat16, float16 or float32) to use weights dynamically quantized to int4, that are upcasted to float8 during inference to run compute in float8 precision with activations dynamically quantized on the fly to float8. Other layers (e.g. projections in the attention layers) have their weights quantized online to float8 directly. ### `quark_mxfp4` online quantization method SGLang running on AMD GPUs with hardware FP4 support (CDNA4 architecture, e.g. MI355x) supports the quantization method `--quantization quark_mxfp4`, that will quantize BF16 or NVFP4 model weights to MXFP4 at load time, use dynamic MXFP4 quantization for activations and MXFP4 GEMMs instead of BF16 GEMMs. Example (BF16 to MXFP4 requantization): ```bash theme={null} sglang serve --model-path Qwen/Qwen3-30B-A3B \ --tensor-parallel-size 1 \ --quantization quark_mxfp4 ``` #### Online NVFP4 to MXFP4 Requantization The option `--quantization quark_mxfp4` supports converting NVFP4 checkpoints (e.g. `nvidia/Kimi-K2.6-NVFP4`) to MXFP4 at load time to allow efficient inference using supported AMD hardware (gfx95x+): * The quantization metadata of the source NVFP4 checkpoint is read from either `config.json` (`quantization_config`) or a standalone `hf_quant_config.json`; * Producer-declared excluded modules will remain in higher precision; * NVFP4 checkpoints with mixed precision (`"quant_algo": "MIXED_PRECISION"`, e.g. `nvidia/Qwen3.5-397B-A17B-NVFP4-V2`) are also supported. Example (NVFP4 to MXFP4 requantization): ```bash theme={null} sglang serve --model-path nvidia/Kimi-K2.6-NVFP4 \ --tensor-parallel-size 4 \ --quantization quark_mxfp4 \ ``` #### Online FP8 to MXFP4 Requantization The option `--quantization quark_mxfp4` supports converting FP8 dense and MOE models to MXFP4, following this logic: 1. Load an FP8 weight tensor, 2. Dequantize it to BF16, 3. Requantize it to MXFP4 progressively during weight loading. Example (FP8 to MXFP4 requantization): ```bash theme={null} sglang serve --model-path Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \ --tensor-parallel-size 1 \ --quantization quark_mxfp4 ``` ### Intel® Neural Compressor online quantization method SGLang supports quantization methods based on the advanced algorithm [auto-round](https://github.com/intel/auto-round) in [Intel® Neural Compressor](https://github.com/intel/neural-compressor). You can simply specify `--quantization auto-round-int8` to use this feature. It will quantize the model on the fly to target format. More online quantization methods are on the way. ##### Available Quantization Methods | Quantization Method | Schemes | Validated Hardware Environment | | :- | :- | :- | | auto-round-int8 | INT8 per-channel quantized weight
INT8 per-token dynamic quantized activation | Intel Xeon Scalable processor
Nvidia A100 GPU | ## Diffusion Model Quantization on Ascend NPU SGLang-Diffusion supports MXFP8 quantization for diffusion models (such as Wan2.2) on Ascend 950PR/DT Series NPUs, in both online and offline (ModelSlim) modes. This is separate from the LLM serving path and uses the `sglang serve` / `sglang generate` CLI. **Requirements:** Ascend 950PR/DT Series, CANN ≥ 8.0.RC3 ### Online MXFP8 Pass `--quantization mxfp8` to dynamically quantize FP16/BF16 transformer weights to MXFP8 at load time: ```bash theme={null} sglang serve \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --quantization mxfp8 \ --num-gpus 4 ``` ### Offline MXFP8 (ModelSlim) Pre-quantize with [msModelSlim](https://gitcode.com/Ascend/msmodelslim) and load the checkpoint directly — the quantization scheme is auto-detected from `quant_model_description.json`: ```bash theme={null} sglang generate \ --model-path /path/to/wan2_2_mxfp8_diffusers \ --prompt "a beautiful sunset" \ --save-output ``` For the full quantization + format conversion workflow and a complete list of supported schemes, see [Diffusion Quantization on Ascend NPU](../hardware-platforms/ascend-npus/optimization/quantization#diffusion-model-quantization-on-ascend-npu) and [SGLang-Diffusion Quantization](../sglang-diffusion/quantization#modelslim). ## Reference * [GPTQModel](https://github.com/ModelCloud/GPTQModel) * [LLM Compressor](https://github.com/vllm-project/llm-compressor/) * [NVIDIA Model Optimizer (ModelOpt)](https://github.com/NVIDIA/Model-Optimizer) * [NVIDIA Model Optimizer LLM PTQ](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq) * [Petit: NVFP4 on ROCm](https://github.com/causalflow-ai/petit-kernel) — [LMSYS blog](https://lmsys.org/blog/2025-09-21-petit-amdgpu/), [AMD ROCm blog](https://rocm.blogs.amd.com/artificial-intelligence/fp4-mixed-precision/README.html) * [vLLM Quantization](https://docs.vllm.ai/en/latest/quantization/) * [auto-round](https://github.com/intel/auto-round) * [ModelSlim](https://gitcode.com/Ascend/msmodelslim) # Quantized KV Cache Source: https://docs.sglang.io/docs/advanced_features/quantized_kv_cache Quantized KV cache reduces the memory footprint of key-value cache storage by using lower-precision data types (FP8 or FP4) instead of the default model precision in BF16. During autoregressive generation, LLMs cache previously computed key-value pairs to avoid redundant calculations. The KV cache typically consumes a significant portion of GPU memory, especially for long sequences. Quantized KV cache is a memory optimization technique that primarily benefits throughput by allowing more tokens to be cached, but may introduce minimal accuracy degradation depending on the quantization format used. **Performance Warning**: When quantized KV cache must be dequantized before use in attention operations, performance can be extremely slow if dequantization is not fused with the attention kernel. Always verify that your chosen attention backend supports quantized KV cache. Backends without fused support may experience significant throughput degradation, potentially negating the memory benefits. **Backend Support**: Not all attention backends support quantized KV cache. Refer to [Attention Backend](./attention_backend) for which backends support it. ## Supported Formats SGLang supports the following quantized KV cache formats: ### FP8 Format [OCP (Open Compute Project)](https://www.opencompute.org) specifies two common 8-bit floating point formats: * **E5M2** (5 exponent bits, 2 mantissa bits): Larger dynamic range (±57344.0), lower precision * **E4M3** (4 exponent bits, 3 mantissa bits): Higher precision, smaller dynamic range (±240.0) ### FP4 Format FP4 quantization is currently experimental. [OCP (Open Compute Project)](https://www.opencompute.org) specifies MXFP4 (Microscaling FP4), a 4-bit floating-point format. SGLang exposes two experimental E2M1 KV-cache recipes: * **`nvfp4`**: NVIDIA FP4 with 16-value blocks, E4M3 block scales, and a per-tensor global scale. * **`fp4_mx_block16`**: An E2M1 block-size-16 compatibility recipe. It is distinct from the standard OCP MXFP4 block-size-32 format. ## Usage ### Enabling Quantized KV Cache To enable quantized KV cache, use the `--kv-cache-dtype` argument when launching the server: ```bash Command theme={null} # Enable FP8 E5M2 KV cache python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-0528 \ --kv-cache-dtype fp8_e5m2 \ # Enable FP8 E4M3 KV cache python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-0528 \ --kv-cache-dtype fp8_e4m3 \ # Enable NVFP4 FP4 E2M1 KV cache python3 -m sglang.launch_server \ --model-path nvidia/DeepSeek-R1-0528-NVFP4 \ --kv-cache-dtype nvfp4 \ # Enable block-size-16 FP4 E2M1 KV cache python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-0528 \ --kv-cache-dtype fp4_mx_block16 \ ``` ### SM100 native NVFP4 recipes On SM100, prefill can either consume packed NVFP4 directly or dequantize it into an FP8 E4M3 workspace. Select the online dequantization dtype with `--prefill-kv-cache-dequant-dtype`; SGLang chooses the corresponding attention implementation. For native NVFP4 prefill and decode: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3.5-35B-A3B-FP8 \ --tp-size 1 \ --kv-cache-dtype nvfp4 \ --prefill-kv-cache-dequant-dtype nvfp4 \ --page-size 16 ``` For FP8 prefill backed by a temporary dequantization workspace, with native NVFP4 decode: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3.5-35B-A3B-FP8 \ --tp-size 1 \ --kv-cache-dtype nvfp4 \ --prefill-kv-cache-dequant-dtype fp8_e4m3 \ --page-size 16 ``` `nvfp4` means that prefill consumes the packed native NVFP4 cache directly without any additional dequantization. The default value, `auto`, selects this native NVFP4 mode on SM100 and FP8 E4M3 dequantization on SM120. Decode consumes native NVFP4 in both recipes. The native recipe avoids the FP8 workspace and its token-linear scale copy; the FP8 recipe retains both the linear scales used during prefill and the physical scale layout used during decode. KV data remains stored as packed FP4 either way. Native NVFP4 prefill requires SM100, a page size divisible by 4, and an attention head dimension divisible by 64. TRT-LLM GenMHA uses FP8 query and output buffers internally; SGLang converts the result back to the model activation dtype. Top-k-1 EAGLE/EAGLE3/NEXTN and breadth-1 NGRAM speculative decoding are supported: an EAGLE-family draft worker uses `trtllm_mha`, and target verification consumes the physical NVFP4 cache directly in eager execution and CUDA Graphs. Set `--speculative-ngram-max-bfs-breadth=1` for NGRAM. With the mixed FlashInfer-prefill/TRT-LLM-decode recipe, SGLang resolves `--speculative-attention-mode` to `decode` so verification does not depend on FlashInfer's transient dequantization workspace. Other speculative algorithms, PD disaggregation, hierarchical KV cache, and LMCache are not currently supported by the SM100 native NVFP4 path. SM120 XQA continues to use its architecture-specific linear scale layout and BF16 query/output path. ### Scaling Factors FP8 quantization requires scaling factors to properly quantize and dequantize the KV cache. Currently, only per-tensor (scalar) scaling factors are supported. Scaling factors can be: * **Loaded from checkpoints**: Pre-quantized models (e.g., ModelOpt) may include `k_scale` and `v_scale` parameters that are automatically loaded * **Provided via JSON**: Supply scaling factors via `--quantization-param-path`. The JSON file should follow this format: ```json Config theme={null} { "kv_cache": { "dtype": "float8_e4m3fn", "scaling_factor": { "0": { "0": 1.0, "1": 1.0 } } } } ``` Where the outer keys in `scaling_factor` are tensor parallel ranks and inner keys are layer indices. If scaling factors are not provided and not found in the checkpoint, it will default to 1.0, which may cause accuracy issues. **FP4 (MXFP4)**: Unlike FP8, FP4 quantization handles scaling factors automatically on-the-fly during quantization and dequantization. No pre-quantized models or external scaling factor files are required—the block-based scaling factors are computed dynamically as needed. ## Performance Considerations ### Memory Savings Quantized KV cache provides significant memory savings: * **BF16 → FP4**: Supports approximately 3.56× more tokens than BF16 (accounting for scaling factor overhead) FP4 and FP8 quantization require additional memory for block-based scaling factors, which reduces the effective memory savings compared to the raw bit-width reduction. FP4 with block size 16 supports approximately 1.78× more tokens than FP8, and approximately 3.56× more tokens than BF16. The relative token capacity between FP8 and BF16 can be derived from these ratios. This enables longer context lengths or more concurrent requests within the same memory budget. For native `nvfp4`, each logical scalar costs 0.5 bytes of packed FP4 data plus 1/16 byte of block-scale storage, compared with 2 bytes for BF16. The resulting theoretical KV-token capacity multiplier is `2 / 0.5625 = 3.56×`. The mixed SM100 recipe owns a second scale layout and a shared one-layer FP8 prefill workspace, so its exact capacity depends on the number of full-attention layers. SGLang includes those auxiliary buffers in its KV-pool sizing calculation. ### Accuracy Impact #### FP8 Accuracy FP8 E4M3 quantization typically introduces minimal accuracy degradation. The impact depends on model architecture, sequence length, and quantization format (generally, E4M3 has better accuracy than E5M2). #### FP4 Accuracy FP4 (MXFP4) quantization provides significant memory savings with varying accuracy impact depending on model size and dataset complexity. Preliminary accuracy test results from [PR #10078](https://github.com/sgl-project/sglang/pull/10078) (MLA) and [PR #12612](https://github.com/sgl-project/sglang/pull/12612) (MHA) show: **Large Models (e.g., Qwen3-235B-A22B, DeepSeek-R1-0528)** On large-scale models, FP4 maintains accuracy close to FP8/BF16, especially on simpler datasets:
Model Dataset KV16 KV8 (FP8 E4M3) KV4 (FP4 E2M1)
Qwen3-235B-A22B gsm8k 0.9168 0.9181 0.9186
Qwen3-235B-A22B aime25 0.7733 0.7333 0.6000
Qwen3-235B-A22B gpqa\_diamond 0.7010 0.6899 0.6778
DeepSeek-R1-0528 gsm8k 0.9157 0.9154 0.9124
DeepSeek-R1-0528 aime25 0.5067 0.4934 0.4000
DeepSeek-R1-0528 gpqa\_diamond 0.7707 0.7697 0.7273
**Smaller Models (e.g., GPT-OSS-120B)** On smaller models, FP4 shows more pronounced accuracy drops, particularly on challenging datasets:
Model Dataset KV16 KV8 (FP8 E4M3) KV4 (FP4 E2M1)
GPT-OSS-120B gsm8k 0.9161 0.9163 0.9152
GPT-OSS-120B aime25 0.7533 0.7667 0.3533
GPT-OSS-120B gpqa\_diamond 0.5081 0.5434 0.3202
**Key Observations:** * **Simple datasets (e.g., gsm8k)**: FP4 maintains accuracy close to FP8/BF16 across model sizes * **Model size matters**: Large models (200B+ parameters) generally tolerate FP4 quantization better than smaller models * **Context length**: Accuracy degradation may be more pronounced in long-context scenarios, as the accumulation of the quantization error may become significant. Evaluate FP4 accuracy on your specific model and workload. Large models on simpler tasks typically show minimal degradation, while smaller models or complex reasoning tasks may require FP8 or BF16 for acceptable accuracy. ## Best Practices * **Use pre-quantized models**: Prefer models quantized offline with scaling factors included in the checkpoint. * **Choose the right format**: Use `fp8_e4m3` for better accuracy (recommended), `fp8_e5m2` for larger dynamic range, or `nvfp4` / `fp4_mx_block16` for maximum memory savings (experimental) * **Check backend compatibility**: Verify that your chosen attention backend supports quantized KV cache See also: * [Quantization](./quantization) * [Attention Backend](./attention_backend) * [Server Arguments](./server_arguments) # Radix Cache Eviction Policies Source: https://docs.sglang.io/docs/advanced_features/radix_eviction_policy When the KV cache pool is full, the radix cache reclaims space by evicting cached prefixes. The eviction policy decides which prefix goes first. The default, `lru`, is the right choice for most workloads; the others trade recency for hit frequency, reuse history, or request priority. ## How eviction picks a victim Eviction only ever considers **evictable leaves**: nodes whose KV is present, unlocked (no in-flight request holds them), and not shadowed by a child that still holds KV. The root is never evictable. The policy scores each candidate and the **lowest score is evicted first**. Once a leaf is evicted, its parent may become a leaf and re-enter the candidate set, so eviction walks a branch from its tip toward the root. A policy only scores. It does not decide how much to free, and it cannot pin KV in memory — a node protected by the policy is still evicted if reclaiming everything else is not enough. ## Available policies Select one with `--radix-eviction-policy`. All of them fall back to least-recently-used order within a tie. | Policy | Evicts first | Use when | | - | - | - | | `lru` (default) | The prefix unused for longest. | General serving. Matches how prefix reuse decays with time. | | `lfu` | The prefix with the fewest cache hits, then the least recent. | A small set of prompts is reused far more than the rest, and you want those to survive bursts of one-off traffic. | | `slru` | Prefixes still in the probationary segment, then the least recent within a segment. | Like `lfu`, but you want a hard floor on how much a one-off prefix can displace a proven one. See [`slru` parameters](#slru-parameters). | | `priority` | The prefix belonging to the lowest-priority request, then the least recent. | You run [priority scheduling](/docs/advanced_features/server_arguments) and want cache retention to follow the same ranking as admission. | | `tlru` | The tail of a conversation beyond what its next prefill needs to meet the TTFT budget ("TEL-safe" tokens), then the least recent. | Agentic / multi-turn workloads where tail TTFT matters more than raw hit rate. See [`tlru` parameters](#tlru-parameters). Requires the unified radix cache (the default tree). | Notes on the scoring inputs: * **Hit count** (`lfu`, `slru`) counts how many times a node was matched by a later request. It is not incremented for chunked-prefill steps, for evicted nodes, or under write-back HiCache. * **Request priority** (`priority`) is the `priority` field of the request that inserted the prefix; a node reached by several requests keeps the highest priority among them. Without priority scheduling every node is priority `0` and this policy is equivalent to `lru`. `fifo`, `mru`, and `filo` also exist in the policy registry but are not offered on the command line. They are reachable only by out-of-tree code that extends the choice list, and exist for experiments rather than serving. ## Tuning a policy Some policies take parameters. Pass them as a json object to `--radix-eviction-policy-config`; the keys are the policy's own, so they are only valid for the policy you selected. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path MODEL_PATH \ --radix-eviction-policy slru \ --radix-eviction-policy-config '{"protected_threshold": 4}' ``` Omit the flag to accept every default. An unrecognized key fails at startup rather than being ignored: ``` TypeError: SLRUStrategy.__init__() got an unexpected keyword argument 'protected_treshold' ``` ### Policy parameters `slru` and `tlru` take parameters. `lru`, `lfu`, and `priority` take none, so `--radix-eviction-policy-config` has no effect with them and any key is an error. #### `slru` parameters `slru` splits the cache into a **probationary** segment and a **protected** segment. A prefix enters probationary, and is promoted to protected once it has been hit enough times. Everything probationary is evicted before anything protected. | Key | Type | Default | Meaning | | - | - | - | - | | `protected_threshold` | int | `2` | Hit count at which a prefix is promoted to the protected segment. | Raising it makes promotion harder, so the protected set stays small and closer to your genuinely hot prefixes; a prefix hit three times stays probationary at `4` but is protected at the default `2`. Lowering it to `1` promotes any prefix that is reused even once, which approaches `lru` with a one-hit grace period. #### `tlru` parameters `tlru` (Tail-Optimized LRU, [arXiv:2510.15152](https://arxiv.org/abs/2510.15152)) protects only the cached history a conversation's next prefill needs to stay under a TTFT budget; the rest of its tail is evicted first, and eviction continues in plain recency order once the TEL-safe tokens run out. | Key | Type | Meaning | | - | - | - | | `threshold` | int | Tail-latency threshold ξ, in tokens: a conversation only keeps enough cache to hold its next prefill under ξ uncached tokens. Convert from a TTFT target by dividing it by the measured ms per uncached token. The paper states ξ in blocks, so multiply its values by `--page-size`. | | `next_prompt_estimate` | int | Estimated tokens the next turn of a conversation will add; use the trace's empirical mean. | Both keys are required in practice: `threshold` must be greater than `next_prompt_estimate` (only the difference affects behaviour), and values at or above it reduce T-LRU to plain LRU, which startup rejects. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path MODEL_PATH \ --radix-eviction-policy tlru \ --radix-eviction-policy-config '{"threshold": 4096, "next_prompt_estimate": 512}' ``` ## Which policy to pick Start with `lru` and change it only against a measured cache hit rate — the counters are exposed under `--enable-metrics`. `lfu` and `slru` help when your traffic has a stable hot set that a recency-only policy keeps flushing; they hurt when prefix popularity shifts over time, because a prefix that earned a high hit count keeps its advantage after it stops being useful. # Reasoning-Aware Compression Source: https://docs.sglang.io/docs/advanced_features/reasoning_aware_compression Pruning a reasoning model with a standard calibration set does more damage than pruning a conventional LLM — and it can make the model *slower*. Reasoning-Aware Compression (RAC) fixes this by changing what the pruning solver calibrates on: the model's own chain of thought, generated with SGLang. From [*Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction*](https://arxiv.org/abs/2509.12464) (ICLR 2026). ## The problem One-shot pruning methods such as SparseGPT and Wanda choose which weights to remove by minimizing a layer-wise reconstruction error against a calibration activation matrix `X`: ``` min_{W'} || W X - W' X ||_F^2 s.t. ||W'||_0 <= S ``` `X` is conventionally built from **prompt** tokens — a slice of C4, or a set of task prompts. That is a fair proxy for a typical serving workload, where the prompt dominates the token count. Reasoning models invert that ratio. They emit thousands of chain-of-thought tokens per query, so almost every forward pass the pruned model will ever run is over a token it generated itself. Calibrating only on prompts optimizes the pruned weights for a distribution the model barely visits. The result is not a graceful accuracy decay. The pruned model starts to ramble: it produces longer chains of thought *and* answers less accurately, so pruning increases end-to-end latency instead of reducing it. At 50% sparsity on MATH-500, C4-calibrated DeepSeek-R1-Distill-Qwen-7B takes almost six times as long to evaluate as the dense model it was meant to accelerate. ## The fix RAC samples the dense model's own on-policy rollout during calibration and reconstructs the prompt and decode activations jointly: ``` X_RAC = [ X_prompt , X_decode ] ``` The solver is untouched, so this is a drop-in change to any existing SparseGPT or Wanda workflow. DeepSeek-R1-Distill-Qwen-7B, MATH-500, SparseGPT at 50% sparsity, 1M calibration tokens: | Calibration set | acc\@1 | Eval wall clock | | - | - | - | | Dense (no pruning) | 0.936 | 23.3 min | | C4 | 0.744 | 135.0 min | | Task prompts only | 0.812 | 115.6 min | | **RAC (prompts + on-policy CoT)** | **0.900** | **35.3 min** | Across DeepSeek-R1-Distill-Qwen (1.5B–32B) and Qwen3 (1.7B–14B), the paper reports that RAC keeps up to 95% of dense accuracy at 50% sparsity, improving on prompt-only calibration by up to 17 points. ## Using it SGLang ships the recipe as a runnable example at [`examples/usage/reasoning_aware_compression`](https://github.com/sgl-project/sglang/tree/main/examples/usage/reasoning_aware_compression), in three phases: | Phase | Script | What it does | | - | - | - | | I | `rac_collect_traces.py` | `sgl.Engine` samples on-policy CoT traces into a calibration set | | II | `rac_prune.py` | `llm-compressor` runs SparseGPT/Wanda against those activations | | III | `rac_serve_and_eval.py` | SGLang serves the sparse checkpoint and scores MATH-500 | Phase I is the expensive step — the paper's budget is 1M on-policy CoT tokens — and is where SGLang's batched generation does the work. Phase II delegates the pruning solver to [`llm-compressor`](https://github.com/vllm-project/llm-compressor), which is **not** an SGLang dependency; install it separately with `pip install "llmcompressor>=0.12.0"`. ```bash theme={null} cd examples/usage/reasoning_aware_compression python rac_collect_traces.py \ --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \ --dataset open-r1/OpenR1-Math-220k --prompt-column problem \ --target-tokens 1000000 --output-dir ./rac_traces_math python rac_prune.py \ --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \ --calibration ./rac_traces_math/traces.jsonl \ --sparsity 0.5 --output-dir ./rac_pruned_50 python -m sglang.launch_server --model-path ./rac_pruned_50 ``` The example README walks through building the paper's prompt-only baseline from the same prompts so you can compare calibration strategies head to head. ## Evaluating a pruned reasoning model Accuracy alone will hide the failure mode described above. Always report **mean completion length** and **wall clock** alongside accuracy when comparing pruned reasoning checkpoints — a model that scores two points lower while emitting three times the chain of thought is not a good trade. `rac_serve_and_eval.py` reports all three. ## Related * [Quantization](/docs/advanced_features/quantization) — the other axis of model compression, applied at serving time. # Reasoning Parser Source: https://docs.sglang.io/docs/advanced_features/separate_reasoning SGLang supports parsing reasoning content out from "normal" content for reasoning models such as [DeepSeek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1). ## Supported Models & Parsers
Model Reasoning tags Parser Notes
[Apertus 2509 models](https://huggingface.co/swiss-ai/Apertus-8B-Instruct-2509) `<|inner_prefix|>` … `<|inner_suffix|>` `apertus2509` Supports `enable_thinking` parameter
[DeepSeek‑R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d) `` … `` `deepseek-r1` Supports all variants (R1, R1-0528, R1-Distill)
[DeepSeek‑V3 series](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) `` … `` `deepseek-v3` Including [DeepSeek‑V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp). Supports `thinking` parameter
[Standard Qwen3 models](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) `` … `` `qwen3` Supports `enable_thinking` parameter
[Qwen3-Thinking models](https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507) `` … `` `qwen3` or `qwen3-thinking` Always generates thinking content
[Kimi K2 Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking) `◁think▷` … `◁/think▷` `kimi_k2` Uses special thinking delimiters. Also requires `--tool-call-parser kimi_k2` for tool use.
[GPT OSS](https://huggingface.co/openai/gpt-oss-120b) `<|channel|>analysis<|message|>` … `<|end|>` `gpt-oss` N/A
### Model-Specific Behaviors **Apertus 2509:** * Uses `<|inner_prefix|>` and `<|inner_suffix|>` to delimit reasoning content. For agentic tool use, also specify `--tool-call-parser apertus2509`. **DeepSeek-R1 Family:** * DeepSeek-R1: No `` start tag, jumps directly to thinking content * DeepSeek-R1-0528: Generates both `` start and `` end tags * Both are handled by the same `deepseek-r1` parser **DeepSeek-V3 Family:** * DeepSeek-V3.1/V3.2: Hybrid model supporting both thinking and non-thinking modes, use the `deepseek-v3` parser and `thinking` parameter (NOTE: not `enable_thinking`) **Qwen3 Family:** * Standard Qwen3 (e.g., Qwen3-2507): Use `qwen3` parser, supports `enable_thinking` in chat templates * Qwen3-Thinking (e.g., Qwen3-235B-A22B-Thinking-2507): Use `qwen3` or `qwen3-thinking` parser, always thinks **Kimi K2:** * Kimi K2 Thinking: Uses special `◁think▷` and `◁/think▷` tags. For agentic tool use, also specify `--tool-call-parser kimi_k2`. **GPT OSS:** * GPT OSS: Uses special `<|channel|>analysis<|message|>` and `<|end|>` tags ## Usage ### Launching the Server Specify the `--reasoning-parser` option. ```python Example theme={null} import requests from openai import OpenAI from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process server_process, port = launch_server_cmd( "python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning" ) wait_for_server(f"http://localhost:{port}") ``` Note that `--reasoning-parser` defines the parser used to interpret responses. To cap how many tokens the model spends thinking, see [Thinking Budget](/docs/advanced_features/thinking_budget). ### OpenAI Compatible API Using the OpenAI compatible API, the contract follows the [DeepSeek API design](https://api-docs.deepseek.com/guides/reasoning_model) established with the release of DeepSeek-R1: * `reasoning_content`: The content of the CoT. * `content`: The content of the final answer. ```python Example theme={null} # Initialize OpenAI-like client client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:{port}/v1") model_name = client.models.list().data[0].id messages = [ { "role": "user", "content": "What is 1+3?", } ] ``` #### Non-Streaming Request ```python Example theme={null} response_non_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0.6, top_p=0.95, stream=False, # Non-streaming extra_body={"separate_reasoning": True}, ) print_highlight("==== Reasoning ====") print_highlight(response_non_stream.choices[0].message.reasoning_content) print_highlight("==== Text ====") print_highlight(response_non_stream.choices[0].message.content) ``` #### Streaming Request ```python Example theme={null} response_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0.6, top_p=0.95, stream=True, # Non-streaming extra_body={"separate_reasoning": True}, ) reasoning_content = "" content = "" for chunk in response_stream: if chunk.choices[0].delta.content: content += chunk.choices[0].delta.content if chunk.choices[0].delta.reasoning_content: reasoning_content += chunk.choices[0].delta.reasoning_content print_highlight("==== Reasoning ====") print_highlight(reasoning_content) print_highlight("==== Text ====") print_highlight(content) ``` Optionally, you can buffer the reasoning content to the last reasoning chunk (or the first chunk after the reasoning content). ```python Example theme={null} response_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0.6, top_p=0.95, stream=True, # Non-streaming extra_body={"separate_reasoning": True, "stream_reasoning": False}, ) reasoning_content = "" content = "" for chunk in response_stream: if chunk.choices[0].delta.content: content += chunk.choices[0].delta.content if chunk.choices[0].delta.reasoning_content: reasoning_content += chunk.choices[0].delta.reasoning_content print_highlight("==== Reasoning ====") print_highlight(reasoning_content) print_highlight("==== Text ====") print_highlight(content) ``` The reasoning separation is enable by default when specify . **To disable it, set the `separate_reasoning` option to `False` in request.** ```python Example theme={null} response_non_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0.6, top_p=0.95, stream=False, # Non-streaming extra_body={"separate_reasoning": False}, ) print_highlight("==== Original Output ====") print_highlight(response_non_stream.choices[0].message.content) ``` ### SGLang Native API ```python Example theme={null} from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B") input = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) gen_url = f"http://localhost:{port}/generate" gen_data = { "text": input, "sampling_params": { "skip_special_tokens": False, "max_new_tokens": 1024, "temperature": 0.6, "top_p": 0.95, }, } gen_response = requests.post(gen_url, json=gen_data).json()["text"] print_highlight("==== Original Output ====") print_highlight(gen_response) parse_url = f"http://localhost:{port}/separate_reasoning" separate_reasoning_data = { "text": gen_response, "reasoning_parser": "deepseek-r1", } separate_reasoning_response_json = requests.post( parse_url, json=separate_reasoning_data ).json() print_highlight("==== Reasoning ====") print_highlight(separate_reasoning_response_json["reasoning_text"]) print_highlight("==== Text ====") print_highlight(separate_reasoning_response_json["text"]) ``` ```python Example theme={null} terminate_process(server_process) ``` ### Offline Engine API ```python Example theme={null} import sglang as sgl from sglang.srt.parser.reasoning_parser import ReasoningParser from sglang.utils import print_highlight llm = sgl.Engine(model_path="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B") tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B") input = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) sampling_params = { "max_new_tokens": 1024, "skip_special_tokens": False, "temperature": 0.6, "top_p": 0.95, } result = llm.generate(prompt=input, sampling_params=sampling_params) generated_text = result["text"] # Assume there is only one prompt print_highlight("==== Original Output ====") print_highlight(generated_text) parser = ReasoningParser("deepseek-r1") reasoning_text, text = parser.parse_non_stream(generated_text) print_highlight("==== Reasoning ====") print_highlight(reasoning_text) print_highlight("==== Text ====") print_highlight(text) ``` ```python Example theme={null} llm.shutdown() ``` ## Supporting New Reasoning Model Schemas For future reasoning models, you can implement the reasoning parser as a subclass of `BaseReasoningFormatDetector` in `python/sglang/srt/reasoning_parser.py` and specify the reasoning parser for new reasoning model schemas accordingly. # Server Arguments Source: https://docs.sglang.io/docs/advanced_features/server_arguments This page provides a list of server arguments used in the command line to configure the behavior and performance of the language model server during deployment. These arguments enable users to customize key aspects of the server, including model selection, parallelism policies, memory management, and optimization techniques. You can find all arguments by `python3 -m sglang.launch_server --help` ## Common launch commands * To use a configuration file, create a YAML file with your server arguments and specify it with `--config`. CLI arguments will override config file values. ```bash Command theme={null} # Create config.yaml cat > config.yaml << EOF model-path: meta-llama/Meta-Llama-3-8B-Instruct host: 0.0.0.0 port: 30000 tensor-parallel-size: 2 enable-metrics: true log-requests: true EOF # Launch server with config file python -m sglang.launch_server --config config.yaml ``` * To enable multi-GPU tensor parallelism, add `--tp 2`. If it reports the error "peer access is not supported between these two devices", add `--enable-p2p-check` to the server launch command. ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 2 ``` * To enable multi-GPU data parallelism, add `--dp 2`. Data parallelism is better for throughput if there is enough memory. It can also be used together with tensor parallelism. The following command uses 4 GPUs in total. We recommend [SGLang Model Gateway (former Router)](../advanced_features/sgl_model_gateway) for data parallelism. ```bash Command theme={null} python -m sglang_router.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --dp 2 --tp 2 ``` * If you see out-of-memory errors during serving, try to reduce the memory usage of the KV cache pool by setting a smaller value of `--mem-fraction-static`. ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --mem-fraction-static 0.7 ``` * See [hyperparameter tuning](./hyperparameter_tuning) on tuning hyperparameters for better performance. * For docker and Kubernetes runs, you need to set up shared memory which is used for communication between processes. See `--shm-size` for docker and `/dev/shm` size update for Kubernetes manifests. * If you see out-of-memory errors during prefill for long prompts, try to set a smaller chunked prefill size. ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --chunked-prefill-size 4096 ``` * To enable fp8 weight quantization, add `--quantization fp8` on a fp16 checkpoint or directly load a fp8 checkpoint without specifying any arguments. * To enable fp8 kv cache quantization, add `--kv-cache-dtype fp8_e4m3` or `--kv-cache-dtype fp8_e5m2`. * To enable deterministic inference and batch invariant operations, add `--enable-deterministic-inference`. More details can be found in [deterministic inference document](./deterministic_inference). * If a multimodal server accepts requests from untrusted clients, restrict remote image, video, and audio URLs with `--allowed-media-domains`. SGLang checks the initial URL and every redirect destination against the exact-hostname allowlist. Remote media downloads are limited to 64 MiB by default; adjust `--media-url-max-file-size-mb` when larger trusted media is required. ```bash Command theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen2.5-VL-7B-Instruct \ --allowed-media-domains upload.wikimedia.org raw.githubusercontent.com ``` Without `--allowed-media-domains`, HTTP(S) media from any domain remains allowed for backward compatibility. Do not expose that configuration to untrusted users. Local paths and `data:` URLs are not governed by the domain allowlist. * To enable decode context parallelism for MLA models, add `--dcp-size N`. See [Decode Context Parallelism](./dcp). * If the model does not have a chat template in the Hugging Face tokenizer, you can specify a [custom chat template](../references/custom_chat_template). If the tokenizer has multiple named templates (e.g., 'default', 'tool\_use'), you can select one using `--hf-chat-template-name tool_use`. * To run tensor parallelism on multiple nodes, add `--nnodes 2`. If you have two nodes with two GPUs on each node and want to run TP=4, let `sgl-dev-0` be the hostname of the first node and `50000` be an available port, you can use the following commands. If you meet deadlock, please try to add `--disable-cuda-graph` * (Note: This feature is out of maintenance and might cause error) To enable `torch.compile` acceleration, add `--enable-torch-compile`. It accelerates small models on small batch sizes. By default, the cache path is located at `/tmp/torchinductor_root`, you can customize it using environment variable `TORCHINDUCTOR_CACHE_DIR`. For more details, please refer to [PyTorch official documentation](https://pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html) and [Enabling cache for torch.compile](../references/torch_compile_cache). ```bash Command theme={null} # Node 0 python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3-8B-Instruct \ --tp 4 \ --dist-init-addr sgl-dev-0:50000 \ --nnodes 2 \ --node-rank 0 # Node 1 python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3-8B-Instruct \ --tp 4 \ --dist-init-addr sgl-dev-0:50000 \ --nnodes 2 \ --node-rank 1 ``` Please consult the documentation below and [server\_args.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/server_args.py) to learn more about the arguments you may provide when launching a server. ## Model and tokenizer
Argument Description Defaults Options
`--model-path`
`--model`		 The path of the model weights. This can be a local folder or a Hugging Face repo ID. ` None` Type: str
`--tokenizer-path` The path of the tokenizer. ` None` Type: str
`--tokenizer-mode` Tokenizer mode. 'auto' will use the fast tokenizer if available, and 'slow' will always use the slow tokenizer. `auto` auto, slow
`--tokenizer-backend` Tokenizer backend. 'huggingface' uses the default HuggingFace tokenizers library; 'fastokens' uses the fastokens library for faster tokenization. Requires the fastokens package to be installed. `huggingface` huggingface, fastokens
`--tokenizer-worker-num` The worker num of the tokenizer manager. `1` Type: int
`--skip-tokenizer-init` If set, skip init tokenizer and pass input\_ids in generate request. `False` bool flag (set to enable)
`--load-format` The format of the model weights to load. "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. "pt" will load the weights in the pytorch bin format. "safetensors" will load the weights in the safetensors format. "instanttensor" will load Safetensors weights using InstantTensor's high-performance distributed loader. "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. "dummy" will initialize the weights with random values, which is mainly for profiling."gguf" will load the weights in the gguf format. "bitsandbytes" will load the weights using bitsandbytes quantization."layered" loads weights layer by layer so that one can quantize a layer before loading another to make the peak memory envelope smaller. "flash\_rl" will load the weights in flash\_rl format. "fastsafetensors" and "private" are also supported. "runai\_streamer" enables direct model loading from object storage and shared file systems. `auto` auto, pt, safetensors, instanttensor, npcache, dummy, sharded\_state, gguf, bitsandbytes, mistral, layered, flash\_rl, remote, remote\_instance, fastsafetensors, private, runai\_streamer
`--model-loader-extra-config` Extra config for model loader. This will be passed to the model loader corresponding to the chosen load\_format. `{}` Type: str
`--trust-remote-code` Whether or not to allow for custom models defined on the Hub in their own modeling files. `False` bool flag (set to enable)
`--context-length` The model's maximum context length. Defaults to None (will use the value from the model's config.json instead). ` None` Type: int
`--is-embedding` Whether to use a CausalLM as an embedding model. `False` bool flag (set to enable)
`--enable-multimodal` Enable the multimodal functionality for the served model. If the model being served is not multimodal, nothing will happen. ` None` bool flag (set to enable)
`--revision` The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. ` None` Type: str
`--model-impl` Which implementation of the model to use. "auto" will try to use the SGLang implementation if it exists and fall back to the Transformers implementation if no SGLang implementation is available. "sglang" will use the SGLang model implementation. "transformers" will use the Transformers model implementation. `auto` Type: str
`--detokenizer-worker-num` The worker num of the detokenizer manager. `1` Type: int
`--prefill-only-disable-kv-cache` Skip the physical KV cache allocation for embedding-mode prefill-only workloads. Currently only valid with --is-embedding, --chunked-prefill-size=-1, --disable-radix-cache, an FA prefill backend, and non-FP4 KV cache so the fa\_skip\_kv\_cache path is active (no layer reads or writes the cache). Other prefill-only workloads such as scoring/MIS may benefit from this later once their attention paths stop using paged KV. Scheduler admission accounting is unchanged; per-layer K/V tensors are sized to (page\_size, head\_num, head\_dim) placeholders so GPU memory is not wasted. `False` bool flag (set to enable)
`--model-config-parser` Which model-config parser to use. "auto" picks "mistral" via the is\_mistral\_model name heuristic, else "hf" (AutoConfig over config.json). Plugins can register additional parsers via @register\_model\_config\_parser. `auto` Type: str
## HTTP server
Argument Description Defaults Options
`--host` The host of the HTTP server. `127.0.0.1` Type: str
`--port` The port of the HTTP server. `30000` Type: int
`--fastapi-root-path` App is behind a path based routing proxy. `""` Type: str
`--grpc-mode` If set, use gRPC server instead of HTTP server. `False` bool flag (set to enable)
`--skip-server-warmup` If set, skip warmup. `False` bool flag (set to enable)
`--warmups` Specify custom warmup functions (csv) to run before server starts eg. --warmups=warmup\_name1,warmup\_name2 will run the functions `warmup_name1` and `warmup_name2` specified in warmup.py before the server starts listening for requests `None` Type: str
`--nccl-port` The port for NCCL distributed environment setup. Defaults to a random port. `None` Type: int
`--checkpoint-engine-wait-weights-before-ready` If set, the server will wait for initial weights to be loaded via checkpoint-engine or other update methods before serving inference requests. `False` bool flag (set to enable)
`--ssl-keyfile` The file path to the SSL key file. `None` Type: str
`--ssl-certfile` The file path to the SSL certificate file. `None` Type: str
`--ssl-ca-certs` The CA certificates file. `None` Type: str
`--ssl-keyfile-password` The password to decrypt the SSL keyfile. `None` Type: str
`--enable-ssl-refresh` Enable automatic SSL certificate hot-reloading when cert/key files change on disk. Requires --ssl-certfile and --ssl-keyfile. `False` bool flag (set to enable)
`--enable-http2` Use Granian instead of Uvicorn as the ASGI server, enabling HTTP/1.1 and HTTP/2 auto-negotiation. Clients may use h2c (cleartext HTTP/2) or plain HTTP/1.1. Requires 'pip install sglang\[http2]'. `False` bool flag (set to enable)
`--http2-max-concurrent-streams` Maximum number of concurrent streams advertised on each HTTP/2 connection (1 to 2^32 - 1). Only applies with --enable-http2. `200` Type: int
## Quantization and data type
Argument Description Defaults Options
`--dtype` Data type for model weights and activations. \* "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. \* "half" for FP16. Recommended for AWQ quantization. \* "float16" is the same as "half". \* "bfloat16" for a balance between precision and range. \* "float" is shorthand for FP32 precision. \* "float32" for FP32 precision. `auto` auto, half, float16, bfloat16, float, float32
`--quantization` The quantization method. `None` awq, fp8, mxfp8, gptq, marlin, gptq\_marlin, awq\_marlin, bitsandbytes, gguf, modelopt, modelopt\_fp8, modelopt\_fp4, nvfp4\_online, modelopt\_mixed, petit\_nvfp4, w8a8\_int8, w8a8\_fp8, moe\_wna16, w4afp8, mxfp4, auto-round, compressed-tensors, modelslim, quark, quark\_int4fp8\_moe, quark\_mxfp4, mlx\_q4, mlx\_q8, unquant
`--quantization-param-path` Path to the JSON file containing the KV cache scaling factors. This should generally be supplied, when KV cache dtype is FP8. Otherwise, KV cache scaling factors default to 1.0, which may cause accuracy issues. `None` Type: Optional\[str]
`--kv-cache-dtype` Data type for kv cache storage. "auto" will use model data type. "bf16" or "bfloat16" for BF16 KV cache. "fp8\_e5m2" and "fp8\_e4m3" are supported for CUDA 11.8+. "nvfp4" selects the NVFP4 FP4 E2M1 KV cache recipe; "fp4\_mx\_block16" selects the block-size-16 FP4 E2M1 KV cache recipe. Both require CUDA 12.8+ and PyTorch 2.8.0+ `auto` auto, fp8\_e5m2, fp8\_e4m3, bf16, bfloat16, nvfp4, fp4\_mx\_block16
`--enable-fp32-lm-head` If set, the LM head outputs (logits) are in FP32. `False` bool flag (set to enable)
`--modelopt-quant` The ModelOpt quantization configuration. Supported values: 'fp8', 'int4\_awq', 'w4a8\_awq', 'nvfp4', 'nvfp4\_awq'. This requires the NVIDIA Model Optimizer library to be installed: pip install nvidia-modelopt `None` Type: str
`--modelopt-checkpoint-restore-path` Path to restore a previously saved ModelOpt quantized checkpoint. If provided, the quantization process will be skipped and the model will be loaded from this checkpoint. `None` Type: str
`--modelopt-checkpoint-save-path` Path to save the ModelOpt quantized checkpoint after quantization. This allows reusing the quantized model in future runs. `None` Type: str
`--modelopt-export-path` Path to export the quantized model in HuggingFace format after ModelOpt quantization. The exported model can then be used directly with SGLang for inference. If not provided, the model will not be exported. `None` Type: str
`--quantize-and-serve` Quantize the model with ModelOpt and immediately serve it without exporting. This is useful for development and prototyping. For production, it's recommended to use separate quantization and deployment steps. `False` bool flag (set to enable)
`--rl-quant-profile` Path to the FlashRL quantization profile. Required when using --load-format flash\_rl. `None` Type: str
`--enable-quant-communications` Enable INT8 quantization of TP communications (Supported only for NPU for Qwen3 series). `False` bool flag (set to enable)
`--enable-tf32-matmul` Enable float32 matmuls to use TensorFloat32 precision for better performance (via torch.set\_float32\_matmul\_precision). CUDA only. Automatically enabled for MiniMax-M2 and GLM-4 models. `False` bool flag (set to enable)
## Memory and scheduling
Argument Description Defaults Options
`--mem-fraction-static` The fraction of the memory used for static allocation (model weights and KV cache memory pool). Use a smaller value if you see out-of-memory errors. When unset, it is computed as `(GPU memory - reserved memory) / GPU memory`, defaulting to `0.88` if GPU memory cannot be detected. `None` Type: float
`--max-running-requests` The maximum number of running requests. `None` Type: int
`--max-queued-requests` The maximum number of queued requests. This option is ignored when using disaggregation-mode. `None` Type: int
`--max-total-tokens` The maximum number of tokens in the memory pool. If not specified, it will be automatically calculated based on the memory usage fraction. This option is typically used for development and debugging purposes. `None` Type: int
`--chunked-prefill-size` The maximum number of tokens in a chunk for the chunked prefill. Setting this to -1 means disabling chunked prefill. `None` Type: int
`--prefill-max-requests` The maximum number of requests in a prefill batch. If not specified, there is no limit. `None` Type: int
`--enable-dynamic-chunking` Enable dynamic chunk size adjustment for pipeline parallelism. When enabled, chunk sizes are dynamically calculated based on fitted function to maintain consistent execution time across chunks. `False` bool flag (set to enable)
`--max-prefill-tokens` The maximum number of tokens in a prefill batch. The real bound will be the maximum of this value and the model's maximum context length. `16384` Type: int
`--schedule-policy` The scheduling policy of the requests. `fcfs` lpm, random, fcfs, dfs-weight, lof, priority, routing-key
`--enable-priority-scheduling` Enable priority scheduling. Requests with higher priority integer values will be scheduled first by default. `False` bool flag (set to enable)
`--abort-on-priority-when-disabled` If set, abort requests that specify a priority when priority scheduling is disabled. `False` bool flag (set to enable)
`--schedule-low-priority-values-first` If specified with --enable-priority-scheduling, the scheduler will schedule requests with lower priority integer values first. `False` bool flag (set to enable)
`--priority-scheduling-preemption-threshold` Minimum difference in priorities for an incoming request to have to preempt running request(s). `10` Type: int
`--schedule-conservativeness` How conservative the schedule policy is. A larger value means more conservative scheduling. Use a larger value if you see requests being retracted frequently. `1.0` Type: float
`--retraction-policy` The decode retraction policy to use when the KV cache is full. 'length' preserves the existing behavior and retracts short-output, long-input requests first. 'priority' retracts lower-priority requests first, using the same priority direction as priority scheduling. `length` length, priority
`--page-size` The number of tokens in a page. `1` Type: int
`--enable-page-major-kv-layout` Store the Mamba state and full/SWA KV caches as one entry per token slot, holding every layer's state, in a single page-granular buffer instead of one buffer per layer. Implied by `--enable-unified-memory`, and currently supported only with it. `False` bool flag (set to enable)
`--enable-unified-memory` For hybrid Mamba/GDN and hybrid SWA models, replace the statically-partitioned pools (full-attention KV + SWA/Mamba conv/SSM state) with a single byte buffer split dynamically between the sub-pools, so KV-vs-state capacity flexes with the workload instead of being fixed at startup. Implies `--enable-page-major-kv-layout`. Requires stride-aware attention, linear-attention and Mamba backends; the allowed backends and the supported PD-disaggregation and speculative-decoding configurations are validated at startup. `False` bool flag (set to enable)
`--swa-full-tokens-ratio` The ratio of SWA layer KV tokens / full layer KV tokens, regardless of the number of swa:full layers. It should be between 0 and 1. E.g. 0.5 means if each swa layer has 50 tokens, then each full layer has 100 tokens. `0.8` Type: float
`--disable-hybrid-swa-memory` Disable the hybrid SWA memory. `False` bool flag (set to enable)
`--radix-eviction-policy` The eviction policy of radix trees. 'lru' stands for Least Recently Used, 'lfu' stands for Least Frequently Used. `lru` lru, lfu, slru, priority
`--enable-prefill-delayer` Enable prefill delayer for DP attention to reduce idle time. `False` bool flag (set to enable)
`--prefill-delayer-max-delay-passes` Maximum forward passes to delay prefill. `30` Type: int
`--prefill-delayer-token-usage-low-watermark` Token usage low watermark for prefill delayer. `None` Type: float
`--prefill-delayer-queue-min-ratio` Opt-in to the adaptive queue-based delay trigger (independent of the slot-based one). Defers prefill until the waiting queue reaches `min(running_req * ratio, max_prefill_bs)` so small fragments batch into a larger prefill. Unset keeps the original slot-only behavior. Typical: `0.1`–`0.5`. `None` Type: float
`--prefill-delayer-max-delay-ms` Wall-clock cap (ms) on a single queue-trigger delay; once exceeded, prefill is force-released to bound worst-case TTFT. Only consulted when `--prefill-delayer-queue-min-ratio` is set. Typical: `1000`–`5000`. `None` Type: float
`--prefill-delayer-forward-passes-buckets` Custom buckets for prefill delayer forward passes histogram. 0 and max\_delay\_passes-1 will be auto-added. `None` List\[float]
`--prefill-delayer-wait-seconds-buckets` Custom buckets for prefill delayer wait seconds histogram. 0 will be auto-added. `None` List\[float]
`--disable-priority-preemption` Disable priority scheduling preemption. `False` bool flag (set to enable)
`--default-priority-value` Default priority for requests without explicit priority. `None` Type: int
## Runtime options
Argument Description Defaults Options
`--device` The device to use ('cuda', 'xpu', 'hpu', 'npu', 'cpu'). Defaults to auto-detection if not specified. `None` Type: str
`--tensor-parallel-size`
`--tp-size`		 The tensor parallelism size. `1` Type: int
`--pipeline-parallel-size`
`--pp-size`		 The pipeline parallelism size. `1` Type: int
`--attention-context-parallel-size`
`--attn-cp-size`		 The attention context parallelism size. `1` Type: int
`--dcp-size`
`--decode-context-parallel-size`		 The decode context parallelism size. See Decode Context Parallelism. `1` Type: int
`--dcp-comm-backend` Communication backend for the DCP attention reduction: AllGather + ReduceScatter, one All-to-All exchange, or FlashInfer MNNVL All-to-All. Unset resolves to fi\_a2a where the DCP group shares an MNNVL domain on Blackwell, else a2a on CUDA/ROCm/XPU, else ag\_rs. None ag\_rs, a2a, fi\_a2a
`--dcp-replicate-q-proj` For MLA DCP with the a2a/fi\_a2a backend: replicate the Q projection so each DCP rank computes the full-head query locally. Use --no-dcp-replicate-q-proj to disable the model-specific default. None bool flag (set to enable)
`--moe-data-parallel-size`
`--moe-dp-size`		 The moe data parallelism size. `1` Type: int
`--pp-max-micro-batch-size` The maximum micro batch size in pipeline parallelism. None Type: int
`--pp-async-batch-depth` The async batch depth of pipeline parallelism. `0` Type: int
`--stream-interval` The interval (or buffer size) for streaming in terms of the token length. A smaller value makes streaming smoother, while a larger value makes the throughput higher `1` Type: int
`--incremental-streaming-output` Whether to output as a sequence of disjoint segments. `False` bool flag (set to enable)
`--random-seed` The random seed. None Type: int
`--mlx-enable-sampling` MLX backend only: sample decode tokens (temperature / top-k / top-p / min-p) instead of greedy argmax. Sampling runs inside the lazy MLX graph, so it works with the overlap scheduler; first tokens from prefill/extend are sampled too. Greedy requests keep exact argmax behavior. Also enables on the MLX path: grammar vocab masks and custom logit processors (these break decode chaining per step; custom processors run on pure-decode steps only), logit\_bias, output logprobs (sampled token / top-k / token\_ids; prompt input logprobs are not computed), NaN sanitization (SGLANG\_SANITIZE\_NAN\_LOGITS), and per-request sampling\_seed under --enable-deterministic-inference (deterministic within MLX only). Penalties are not applied. `False` bool flag (set to enable)
`--constrained-json-whitespace-pattern` (outlines and llguidance backends only) Regex pattern for syntactic whitespaces allowed in JSON constrained output. For example, to allow the model to generate consecutive whitespaces, set the pattern to \[\n\t ]\* None Type: str
`--constrained-json-disable-any-whitespace` (xgrammar and llguidance backends only) Enforce compact representation in JSON constrained output. `False` bool flag (set to enable)
`--watchdog-timeout` Set watchdog timeout in seconds. If a forward batch takes longer than this, the server will crash to prevent hanging. `300` Type: float
`--soft-watchdog-timeout` Set soft watchdog timeout in seconds. If a forward batch takes longer than this, the server will dump information for debugging. `None` Type: float
`--dist-timeout` Set timeout for torch.distributed initialization. `None` Type: int
`--download-dir` Model download directory for huggingface. None Type: str
`--model-checksum` Model file integrity verification. If provided without value, uses model-path as HF repo ID. Otherwise, provide checksums JSON file path or HuggingFace repo ID. None Type: str
`--base-gpu-id` The base GPU ID to start allocating GPUs from. Useful when running multiple instances on the same machine. `0` Type: int
`--gpu-id-step` The delta between consecutive GPU IDs that are used. For example, setting it to 2 will use GPU 0,2,4,... `1` Type: int
`--sleep-on-idle` Reduce CPU usage when sglang is idle. `False` bool flag (set to enable)
`--batch-notify-size` Number of streaming notifications to batch before yielding to the event loop. Reduces asyncio wakeup overhead under high concurrency. `16` Type: int
`--stream-response-default-include-usage` Include usage in every streaming response (even when stream\_options is not specified). `False` bool flag (set to enable)
`--enable-streaming-session` Enable streaming session mode and StreamingSession wrapper. `False` bool flag (set to enable)
`--load-snapshot-publish-interval` Publish load snapshot to shared memory every N decode iterations. Prefill and idle always publish immediately. `15` Type: int
`--use-ray` Use Ray actors for scheduler process management. `False` bool flag (set to enable)
## Logging
Argument Description Defaults Options
`--log-level` The logging level of all loggers. `info` Type: str
`--log-level-http` The logging level of HTTP server. If not set, reuse --log-level by default. `None` Type: str
`--log-requests` Log metadata, inputs, outputs of all requests. The verbosity is decided by --log-requests-level `False` bool flag (set to enable)
`--log-requests-level` 0: Log metadata (no sampling parameters). 1: Log metadata and sampling parameters. 2: Log metadata, sampling parameters and partial input/output. 3: Log every input/output. `2` 0, 1, 2, 3
`--log-requests-format` Format for request logging: 'text' (human-readable) or 'json' (structured) `text` text, json
`--log-requests-target` Target(s) for request logging: 'stdout' and/or directory path(s) for file output. Can specify multiple targets, e.g., '--log-requests-target stdout /my/path'. `None` List\[str]
`--uvicorn-access-log-exclude-prefixes` Exclude uvicorn access logs whose request path starts with any of these prefixes. Defaults to empty (disabled). `[]` List\[str]
`--crash-dump-folder` Folder for crash diagnostics. Stores completed requests retained by the crash-dump buffer plus in-flight requests and, on NVIDIA CUDA, configures device coredumps under this folder. Existing CUDA coredump environment variables take precedence. If not specified, this option does not enable request crash dumping or set CUDA coredump defaults. `None` Type: str
`--show-time-cost` Show time cost of custom marks. `False` bool flag (set to enable)
`--enable-metrics` Enable log prometheus metrics. `False` bool flag (set to enable)
`--enable-mfu-metrics` Enable estimated MFU-related prometheus metrics. `False` bool flag (set to enable)
`--enable-metrics-for-all-schedulers` Enable --enable-metrics-for-all-schedulers when you want schedulers on all TP ranks (not just TP 0) to record request metrics separately. This is especially useful when dp\_attention is enabled, as otherwise all metrics appear to come from TP 0. `False` bool flag (set to enable)
`--tokenizer-metrics-custom-labels-header` Specify the HTTP header for passing custom labels for tokenizer metrics. `x-custom-labels` Type: str
`--tokenizer-metrics-allowed-custom-labels` The custom labels allowed for tokenizer metrics. The labels are specified via a dict in '--tokenizer-metrics-custom-labels-header' field in HTTP requests, e.g., \{'label1': 'value1', 'label2': 'value2'} is allowed if '--tokenizer-metrics-allowed-custom-labels label1 label2' is set. `None` List\[str]
`--bucket-time-to-first-token` The buckets of time to first token, specified as a list of floats. `None` List\[float]
`--bucket-inter-token-latency` The buckets of inter-token latency, specified as a list of floats. `None` List\[float]
`--bucket-e2e-request-latency` The buckets of end-to-end request latency, specified as a list of floats. None List\[float]
`--prompt-tokens-buckets` The buckets rule of prompt tokens. Supports 3 rule types: 'default' uses predefined buckets; 'tse \ \ \' generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets \[984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom \ \ ...' uses custom bucket values (e.g., 'custom 10 50 100 500'). `None` List\[str]
`--generation-tokens-buckets` The buckets rule for generation tokens histogram. Supports 3 rule types: 'default' uses predefined buckets; 'tse \ \ \' generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets \[984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom \ \ ...' uses custom bucket values (e.g., 'custom 10 50 100 500'). None List\[str]
`--gc-warning-threshold-secs` The threshold for long GC warning. If a GC takes longer than this, a warning will be logged. Set to 0 to disable. `0.0` Type: float
`--decode-log-interval` The log interval of decode batch. `40` Type: int
`--enable-request-time-stats-logging` Enable per request time stats logging `False` bool flag (set to enable)
`--kv-events-config` Config in json format for NVIDIA dynamo KV event publishing. Publishing will be enabled if this flag is used. Runtime-load publishing for load-aware routers is a separate opt-in; see `--load-publish-endpoint`. Set `replay_endpoint` (e.g. `tcp://*:5558`) to let routers re-fetch dropped batches; `/server_info` advertises its port. None Type: str
`--load-publish-endpoint` Opt in to the runtime-load PUB socket that load-aware routers subscribe to. Off by default (unset or `off`). Use `auto` to reserve the dp\_size ports packed after the `--kv-events-config` range, or a wildcard-host TCP address (e.g. `tcp://*:6000`) to place it explicitly; rank r binds port+r and `/server_info` advertises the base under the `kv_events` block. Requires `--kv-events-config`; startup fails if set without one, not bindable, or overlapping the KV range. `auto` reserves 2\*dp\_size ports from the KV base — space co-hosted engines accordingly. The router-facing update cadence follows `--load-snapshot-publish-interval`, so a large value there also staleness-caps this feed. None Type: str
`--enable-trace` Enable opentelemetry trace `False` bool flag (set to enable)
`--trace-modules` Select the components to trace. Available options are 'request' and 'mooncake'. Format: \,\,...... `request` Type: str
`--otlp-traces-endpoint` Config opentelemetry collector endpoint if --enable-trace is set. format: \:\ `localhost:4317` Type: str
`--grpc-http-sidecar-port` Port for the HTTP sidecar server in gRPC mode (--grpc-mode). Serves Prometheus metrics and profiling endpoints. Defaults to --port + 1. Not used in HTTP mode. `None` Type: int
`--extra-metric-labels` The custom labels for metrics. e.g. '\{"label1": "value1", "label2": "value2"}' `None` Type: str
`--enable-forward-pass-metrics` Enable per-iteration forward pass metrics via ZMQ IPC. External consumers (e.g. Dynamo planner) subscribe to the IPC endpoint exposed in server\_args.forward\_pass\_metrics\_ipc\_name. `False` bool flag (set to enable)
`--forward-pass-metrics-worker-id` — `""` Type: str
`--forward-pass-metrics-ipc-name` — `None` Type: str
## RequestMetricsExporter configuration
Argument Description Defaults Options
`--export-metrics-to-file` Export performance metrics for each request to local file (e.g. for forwarding to external systems). `False` bool flag (set to enable)
`--export-metrics-to-file-dir` Directory path for writing performance metrics files (required when --export-metrics-to-file is enabled). `None` Type: str
## Responses storage Responses storage is now disabled by default, including on standalone servers. Add `--enable-response-store` to retain responses for retrieval, `previous_response_id` chaining, and background requests. Storage is process-local and in-memory, has no TTL or size limit, and is lost on restart. Foreground generation and streaming still accept the API's default `store: true` without retaining responses or message history when server storage is disabled. The response's `store` field echoes the client request; the server flag controls persistence. With storage enabled, `store: false` opts out for the current request while allowing it to read a stored predecessor. Background requests, including streaming requests, require both `--enable-response-store` and `store: true`. Without the flag, chaining, background requests, retrieval, and cancellation return HTTP 400. Cancellation applies to detached background requests; active streams have no stored response until completion. Combining `--enable-response-store` with `--disaggregation-mode=prefill` or `decode` fails at startup. PD clients must send explicit conversation history in foreground requests. Built-in web search and code interpreter calls are also unsupported under PD. ## API related
Argument Description Defaults Options
`--api-key` Set API key of the server. It is also used in the OpenAI API compatible server. `None` Type: str
`--admin-api-key` Set admin API key for administrative/control endpoints (e.g., weights update, cache flush, /server\_info). Endpoints marked as admin-only require Authorization: Bearer \ when this is set. `None` Type: str
`--served-model-name` Override the model name returned by the v1/models endpoint in OpenAI API server. `None` Type: str
`--weight-version` Version identifier for the model weights. Defaults to 'default' if not specified. `default` Type: str
`--chat-template` The builtin chat template name or the path of the chat template file. This is only used for OpenAI-compatible API server. `None` Type: str
`--hf-chat-template-name` When the HuggingFace tokenizer has multiple chat templates (e.g., 'default', 'tool\_use', 'rag'), specify which named template to use. If not set, the first available template is used. `None` Type: str
`--default-chat-template-kwargs` A JSON object used as defaults for chat\_template\_kwargs by Chat Completions, Responses, Anthropic-compatible requests, and chat tokenization. Direct Ollama and embedding paths do not consume these defaults. For ordinary template keys, explicit per-request values take precedence. Keys are specific to the selected model's chat template; reasoning\_effort is normalized separately. `None` Type: JSON / Dict
`--completion-template` The builtin completion template name or the path of the completion template file. This is only used for OpenAI-compatible API server. only for code completion currently. `None` Type: str
`--file-storage-path` The path of the file storage in backend. `sglang_storage` Type: str
`--enable-cache-report` Return number of cached tokens in usage.prompt\_tokens\_details for each openai request. `False` bool flag (set to enable)
`--reasoning-parser` Specify the parser for reasoning models. Use `auto` to detect the parser from the model's chat template. `None` auto, apertus2509, deepseek-r1, deepseek-v3, deepseek-v4, dots, glm45, ling3, hunyuan, gpt-oss, k2\_horizon, kimi, kimi\_k2, kimi\_k3, mimo, muse, poolside\_v1, qwen3, qwen3-thinking, minimax, minimax-append-think, minimax-m3, step3, step3p5, mistral, nemotron\_3, interns1, gemma4, gigachat35, inkling, cohere\_command4
`--tool-call-parser` Specify the parser for handling tool-call interactions. Use `auto` to detect the parser from the model's chat template. `None` auto, apertus2509, cohere\_command4, deepseekv3, deepseekv31, deepseekv32, deepseekv4, dots, glm, glm45, glm47, gpt-oss, k2\_horizon, kimi\_k2, kimi\_k3, lfm2, ling3, llama3, mimo, minicpm5, mistral, muse, poolside\_v1, pythonic, qwen, qwen25, qwen3\_coder, spark25, step3, step3p5, minimax-m2, minimax-m3, trinity, interns1, hermes, hunyuan, gigachat3, gigachat35, gemma4, inkling
`--tool-server` Either 'demo' or a comma-separated list of tool server urls to use for the model. If not specified, no external tool server will be used. Native GPT-OSS `web_search` can still be enabled with `EXA_API_KEY`. `None` Type: str
`--sampling-defaults` Where to get default sampling parameters. 'openai' uses SGLang/OpenAI defaults (temperature=1.0, top\_p=1.0, etc.). 'model' uses the model's generation\_config.json to get the recommended sampling parameters if available. Default is 'model'. `model` openai, model
`--strip-thinking-cache` Skip caching reasoning-model output (thinking + answer) in the radix tree on finish; keep only the prompt prefix. Opt-in: changes cache contents. `False` bool flag (set to enable)
`--enable-strict-thinking` Enable strict token filtering during the thinking phase. Blocks model-specific excluded tokens (e.g., tool call markers) during reasoning. Requires a grammar backend that supports token filtering. See Thinking Budget for per-request thinking caps. `False` bool flag (set to enable)
`--asr-max-buffer-seconds` Maximum seconds of PCM audio the streaming ASR WebSocket handler will accumulate before closing the session with a buffer\_overflow error. Guards against OOM when a client streams audio faster than inference can consume it. Default 60s. `60` Type: int
`--asr-max-concurrent-sessions` Maximum number of concurrent realtime ASR WebSocket sessions served by /v1/realtime. New connections beyond this cap are accepted, sent an error\{code:too\_many\_sessions} frame, and closed. Default 32. `32` Type: int
## Data parallelism
Argument Description Defaults Options
`--data-parallel-size`
`--dp-size`		 The number of data-parallel replicas of the model. `1` Type: int
`--attention-data-parallel-size`
`--attn-dp-size`		 The attention data parallelism size: the number of data-parallel attention groups inside the TP group, one scheduler each, while the FFN stays tensor parallel. `1` Type: int
`--load-balance-method` The load balancing strategy for data parallelism. The `total_tokens` algorithm can only be used when DP attention is applied. This algorithm performs load balancing based on the real-time token load of the DP workers. `auto` auto, round\_robin, follow\_bootstrap\_room, total\_requests, total\_tokens
## Multi-node distributed serving
Argument Description Defaults Options
`--dist-init-addr`
`--nccl-init-addr`		 The host address for initializing distributed backend (e.g., `192.168.0.2:25000`). ` None` Type: str
`--nnodes` The number of nodes. `1` Type: int
`--node-rank` The node rank. `0` Type: int
## Model override args
Argument Description Defaults Options
`--json-model-override-args` A dictionary in JSON string format used to override default model configurations. `{}` Type: str
`--preferred-sampling-params` json-formatted sampling settings that will be returned in /get\_model\_info `None` Type: str
## LoRA
Argument Description Defaults Options
`--enable-lora` Enable LoRA support for the model. This argument is automatically set to `True` if `--lora-paths` is provided for backward compatibility. `False` Bool flag (set to enable)
`--enable-lora-overlap-loading` Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters. `False` Bool flag (set to enable)
`--max-lora-rank` The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in --lora-paths. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup. `None` Type: int
`--lora-target-modules` The union set of all target modules where LoRA should be applied (e.g., q\_proj, k\_proj, gate\_proj). If not specified, it will be automatically inferred from the adapters provided in --lora-paths. You can also set it to all to enable LoRA for all supported modules; note this may introduce minor performance overhead. `None` q\_proj, k\_proj, v\_proj, o\_proj, gate\_proj, up\_proj, down\_proj, qkv\_proj, gate\_up\_proj, all
`--lora-paths` The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: \ | \=\ | JSON with schema \{"lora\_name": str, "lora\_path": str, "pinned": bool}. `None` Type: List\[str] / JSON objects
`--max-loras-per-batch` Maximum number of adapters for a running batch, including base-only requests. `8` Type: int
`--max-loaded-loras` If specified, limits the maximum number of LoRA adapters loaded in CPU memory at a time. Must be ≥ --max-loras-per-batch. `None` Type: int
`--lora-eviction-policy` LoRA adapter eviction policy when the GPU memory pool is full. `lru` lru, fifo
`--lora-backend` Choose the kernel backend for multi-LoRA serving. `csgmv` triton, csgmv, ascend, torch\_native
`--max-lora-chunk-size` Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when --lora-backend is csgmv. Larger values may improve performance. `16` 16, 32, 64, 128
`--lora-drain-wait-threshold` When any LoRA adapter request waits longer than this threshold (in seconds), the scheduler will selectively drain one running adapter to make room. This mitigates extreme tail latency under high or skewed workloads by preventing a small set of adapters from monopolizing batch slots. Set to 0 to disable draining (default). `0.0` Type: float
`--experts-shared-outer-loras` Force shared outer LoRA mode for MoE models. When set, w1/w3 lora\_A and w2 lora\_B are shared across experts (expert\_dim=1). Use --no-experts-shared-outer-loras to force disable. By default this is auto-detected from adapter weights. `None` Type: str
`--lora-use-virtual-experts` Enable virtual expert computation for MoE models. When set, the model will use virtual expert computation. `False` bool flag (set to enable)
`--lora-strict-loading` Enable strict loading for LoRA adapters. When set, mismatched or missing keys in the adapter weights will raise an error. `False` Type: str
## Kernel Backends (Attention, Sampling, Grammar, GEMM)
Argument Description Defaults Options
`--attention-backend` Choose the kernels for attention layers. `None` triton, torch\_native, flex\_attention, dsa, nsa, dsv4, compressed, fa3, fa4, flashinfer, flashmla, trtllm\_mla, cutedsl\_mla, tokenspeed\_mla, trtllm\_mha, aiter, wave, intel\_amx, ascend, intel\_xpu
`--prefill-attention-backend` Choose the kernels for prefill attention layers (have priority over --attention-backend). `None` triton, torch\_native, flex\_attention, dsa, nsa, dsv4, compressed, fa3, fa4, flashinfer, flashmla, trtllm\_mla, cutedsl\_mla, tokenspeed\_mla, trtllm\_mha, aiter, wave, intel\_amx, ascend, intel\_xpu
`--decode-attention-backend` Choose the kernels for decode attention layers (have priority over --attention-backend). `None` triton, torch\_native, flex\_attention, dsa, nsa, dsv4, compressed, fa3, fa4, flashinfer, flashmla, trtllm\_mla, cutedsl\_mla, tokenspeed\_mla, trtllm\_mha, aiter, wave, intel\_amx, ascend, intel\_xpu
`--sampling-backend` Choose the kernels for sampling layers. `None` flashinfer, pytorch, ascend
`--grammar-backend` Choose the backend for grammar-guided decoding. `None` xgrammar, outlines, llguidance, none
`--mm-attention-backend` Set multimodal attention backend. `None` sdpa, fa3, fa4, triton\_attn, ascend\_attn, aiter\_attn, flashinfer\_cudnn, amx\_attn, xpu\_attn
`--dsa-prefill-backend` DSA backend for the prefill stage (overrides `--attention-backend` when running DeepSeek DSA-style attention). Auto (hardware-dependent) flashmla\_sparse, flashmla\_sparse\_q8, triton\_sparse\_mla, flashmla\_kv, flashmla\_auto, fa3, tilelang, aiter, trtllm
`--dsa-triton-union` Union group size for `--dsa-prefill-backend triton_sparse_mla`: adjacent query tokens share one gathered index set (exact; an ownership mask restores each token's own softmax). Requires `num_q_heads * union` to be a power of two in \[16, 32]; forced off under `--enable-deterministic-inference`. `0` 0, 2, 4
`--dsa-decode-backend` DSA backend for the decode stage when running DeepSeek DSA-style attention. Overrides `--attention-backend` for decoding. Auto (hardware-dependent) flashmla\_sparse, flashmla\_kv, flashmla\_auto, fa3, tilelang, aiter, trtllm
`--dsa-topk-backend` Choose the DSA indexer top-k backend for the target model. The `torch` backend currently requires `SGLANG_DSA_FUSE_TOPK=false`. `sgl-kernel` sgl-kernel, torch, flashinfer
`--enable-deepseek-v4-fp4-indexer` Enable the experimental FP4 C4 indexer path for DeepSeek V4. When unset, SGLang keeps the default DeepSeek V4 indexer path. Requires SM100 GPUs with DeepGEMM FP4 indexer support. `False` bool flag (set to enable)
`--fp8-gemm-backend` Choose the runner backend for Blockwise FP8 GEMM operations. For MXFP8 dense GEMM, auto selects flashinfer\_cutedsl when FlashInfer reports support (currently SM100/SM103), and otherwise selects flashinfer\_cutlass on supported Blackwell GPUs. Options also include 'deep\_gemm' (JIT-compiled), 'flashinfer\_trtllm' (FlashInfer TRTLLM backend; SM100/SM103 only), 'flashinfer\_cutlass' (FlashInfer CUTLASS backend), 'flashinfer\_cutedsl' (FlashInfer CuTe DSL MXFP8 backend; SM100/SM103 only), 'flashinfer\_deepgemm' (Hopper SM90 only, uses swapAB optimization for small M dimensions in decoding), 'cutlass' (optimal for SM120 GPUs), 'triton' (fallback, widely compatible), and 'aiter' (ROCm only). `auto` auto, deep\_gemm, flashinfer\_trtllm, flashinfer\_cutlass, flashinfer\_cutedsl, flashinfer\_deepgemm, cutlass, triton, aiter
`--fp4-gemm-backend` Choose the runner backend for NVFP4 GEMM operations. Options: 'auto' (default; selects flashinfer\_cutedsl on SM100, marlin on SM80-SM90, flashinfer\_cutlass otherwise (including SM120)), 'flashinfer\_cutlass' (FlashInfer CUTLASS backend), 'flashinfer\_cudnn' (FlashInfer cuDNN backend, optimal on CUDA 13+ with cuDNN 9.15+), 'flashinfer\_cutedsl' (FlashInfer CuTe DSL backend), 'flashinfer\_trtllm' (FlashInfer TensorRT-LLM backend, requires different weight preparation with shuffling), 'marlin' (weight-only W4A16 fallback for SM80-SM90). Requires FlashInfer to be installed. `auto` auto, flashinfer\_cudnn, flashinfer\_cutedsl, flashinfer\_cutlass, flashinfer\_trtllm, marlin
`--bf16-gemm-backend` Choose the backend for unquantized BF16 GEMM operations. Options: auto (default; selects cutedsl on SM10x GPUs, except deterministic inference selects torch; otherwise uses cuBLAS via torch.nn.functional.linear), cutedsl (SGLang JIT CuTe DSL TGV BF16 GEMM on SM10x; dispatches between the CuTe DSL kernel and cuBLAS), torch (always uses cuBLAS via torch.nn.functional.linear). `auto` auto, cutedsl, torch
`--disable-flashinfer-autotune` Flashinfer autotune is enabled by default. Set this flag to disable the autotune. `False` bool flag (set to enable)
`--flashinfer-autotune-skip-ops` FlashInfer custom-op identifiers to skip during autotuning. See FlashInfer's autotuning documentation. Skipped ops use the heuristic fallback. SGLang temporarily skips mxfp8\_gemm by default due to an IMA. `None` string
`--radix-cache-backend` Name of a radix-cache backend previously registered via register\_radix\_cache\_backend. Omit this flag to use the built-in default cache selection chain. `None` Type: str
## Speculative decoding
Argument Description Defaults Options
`--speculative-algorithm` Speculative algorithm. `None` `EAGLE`, `EAGLE3`, `NEXTN`, `STANDALONE`, `NGRAM`
`--speculative-draft-model-path`
`--speculative-draft-model`		 The path of the draft model weights. This can be a local folder or a Hugging Face repo ID. `None` Type: str
speculative\_boundary\_reduction
Server args field; no CLI flag		 Resolved FFN boundary reduction policy for the speculative draft. Argument processing derives this field from boundary\_reduction and the draft model configuration; an independent value is overwritten by resolution. See FFN boundary reduction. rs+rsv before resolution; model-dependent afterward ar, rs, rsv, rs+rsv
`--speculative-draft-model-revision` The specific draft model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. `None` Type: str
`--speculative-draft-load-format` The format of the draft model weights to load. If not specified, will use the same format as `--load-format`. Use 'dummy' to initialize draft model weights with random values for profiling. `None` auto, pt, safetensors, instanttensor, npcache, dummy, sharded\_state, gguf, bitsandbytes, mistral, layered, flash\_rl, remote, remote\_instance, fastsafetensors, private, runai\_streamer
`--speculative-num-steps` The number of steps sampled from draft model in Speculative Decoding. `None` Type: int
`--speculative-eagle-topk` The number of tokens sampled from the draft model in eagle2 each step. `None` Type: int
`--speculative-num-draft-tokens` The number of tokens sampled from the draft model in Speculative Decoding. `None` Type: int
`--speculative-accept-threshold-single` Accept a draft token if its probability in the target model is greater than this threshold. `1.0` Type: float
`--speculative-accept-threshold-acc` The accept probability of a draft token is raised from its target probability p to min(1, p / threshold\_acc). `1.0` Type: float
`--speculative-use-block-verification` Use block verification for EAGLE/EAGLE3/NEXTN on CUDA or ROCm (requires topk=1). `False` Type: bool
`--speculative-token-map` The path of the draft model's small vocab table. `None` Type: str
`--speculative-attention-mode` Attention backend for speculative decoding operations (both target verify and draft extend). Can be one of 'prefill' (default) or 'decode'. `prefill` prefill, decode
`--speculative-draft-attention-backend` Attention backend for speculative decoding drafting. `None` Same as attention backend options
`--speculative-dsa-topk-backend` Choose the DSA indexer top-k backend for speculative draft workers. The `torch` backend currently requires `SGLANG_DSA_FUSE_TOPK=false`. `sgl-kernel` sgl-kernel, torch, flashinfer
`--speculative-moe-runner-backend` MOE backend for EAGLE speculative decoding, see `--moe-runner-backend` for options. Same as moe runner backend if unset. `None` auto, deep\_gemm, triton, triton\_kernel, flashinfer\_trtllm, experimental\_sgl\_trtllm, flashinfer\_trtllm\_routed, flashinfer\_cutlass, flashinfer\_mxfp4, flashinfer\_cutedsl, cutlass, aiter, marlin
`--speculative-moe-a2a-backend` MOE A2A backend for EAGLE speculative decoding, see `--moe-a2a-backend` for options. Same as moe a2a backend if unset. `None` none, deepep, mooncake, nixl, mori, ascend\_fuseep, flashinfer, megamoe, pplx
`--speculative-enable-w4a4-mxfp4-megamoe` Whether the draft model's MXFP4 MegaMoE layers use the W4A4 `mxf4xmxf4` MMA type. Pass `--no-speculative-enable-w4a4-mxfp4-megamoe` to keep the draft on `fp8xfp4` (W4A8) while the target runs W4A4. Same as `--enable-w4a4-mxfp4-megamoe` if unset. A draft that runs MXFP4 MegaMoE with a different MMA type from the target may allocate an additional MegaMoE symmetric buffer after the KV pool is sized; lower `--mem-fraction-static` if CUDA-graph capture runs out of memory. `None` Type: bool
`--speculative-draft-model-quantization` The quantization method for speculative model. `None` Same as `--quantization` options
`--speculative-dflash-block-size` DFLASH only. Block size (verify window length). Alias of --speculative-num-draft-tokens for DFLASH. `None` Type: int
`--speculative-draft-window-size` Sliding window size for the draft model. Honored by Llama EAGLE-3 (`LlamaForCausalLMEagle3`) and DFLASH only; other EAGLE-3 backends (e.g. MLA-based drafters) silently ignore it. For Llama EAGLE-3, the drafter only attends to the most recent N keys (verifier hidden states + its own outputs); the verifier is unaffected. For DFLASH, the draft worker keeps a recent target-token window in its local KV cache (paged backends may retain up to one extra page on the left for alignment). Default is full attention/context. `None` Type: int
## Ngram speculative decoding
Argument Description Defaults Options
`--speculative-ngram-min-bfs-breadth` The minimum breadth for BFS (Breadth-First Search) in ngram speculative decoding. `1` Type: int
`--speculative-ngram-max-bfs-breadth` The maximum breadth for BFS (Breadth-First Search) in ngram speculative decoding. `10` Type: int
`--speculative-ngram-match-type` Ngram tree-building mode. BFS selects recency-based expansion and PROB selects frequency-based expansion. This setting is forwarded to the ngram cache implementation. `BFS` BFS, PROB
`--speculative-ngram-max-trie-depth` Maximum suffix length stored and matched by the ngram trie. `18` Type: int
`--speculative-ngram-capacity` The cache capacity for ngram speculative decoding. 10000000 Type: int
`--speculative-ngram-external-corpus-path` Path to an external JSONL corpus to pre-load into SAM at startup. Additional corpora can be added at runtime via POST /add\_external\_corpus. `None` Type: str
`--speculative-ngram-external-sam-budget` Number of draft nodes reserved for the external SAM subtree in ngram speculative decoding. `0` Type: int
`--speculative-ngram-external-corpus-max-tokens` Fail startup if the tokenized external ngram corpus exceeds this many tokens. Tune this based on your CPU memory budget. `10000000` Type: int
`--speculative-adaptive` Enable adaptive speculative decoding that dynamically adjusts num\_steps based on acceptance rate. `False` bool flag (set to enable)
`--speculative-adaptive-config` Path to a JSON config file for adaptive speculative decoding tuning knobs. `None` Type: str
`--speculative-skip-dp-mlp-sync` Skip the extra MLP sync that the scheduler performs before merging a new batch when speculative decoding + DP attention are both enabled. `False` bool flag (set to enable)
## Multi-layer Eagle speculative decoding
Argument Description Defaults Options
`--enable-multi-layer-eagle` Enable multi-layer Eagle speculative decoding. `False` bool flag (set to enable)
## MoE
Argument Description Defaults Options
`--expert-parallel-size`
`--ep-size`
`--ep`		 The expert parallelism size. `1` Type: int
`--moe-a2a-backend` Select the backend for all-to-all communication for expert parallelism. `none` none, deepep, mooncake, nixl, mori, ascend\_fuseep, flashinfer, megamoe, pplx
`--moe-runner-backend` Choose the runner backend for MoE. `auto` auto, deep\_gemm, triton, triton\_kernel, flashinfer\_trtllm, experimental\_sgl\_trtllm, flashinfer\_trtllm\_routed, flashinfer\_cutlass, flashinfer\_mxfp4, flashinfer\_cutedsl, cutlass, aiter, marlin
`--flashinfer-mxfp4-moe-precision` Choose the computation precision of flashinfer mxfp4 moe. On SM90, `fp8` selects the Humming-style MXFP4-weight x FP8-activation path introduced by FlashInfer #3738 and requires FlashInfer >= 0.6.18. `default` default, bf16, fp8
`--enable-flashinfer-allreduce-fusion` Enable FlashInfer allreduce fusion with Residual RMSNorm. `False` bool flag (set to enable)
`--enable-aiter-allreduce-fusion` Enable aiter allreduce fusion with Residual RMSNorm. `False` bool flag (set to enable)
`--deepep-mode` Select the mode when enable DeepEP MoE, could be normal, low\_latency or auto. Default is auto, which means low\_latency for decode batch and normal for prefill batch. `auto` normal, low\_latency, auto
`--deepep-dispatcher-output-dtype` Select DeepEP dispatcher output dtype, could be bf16, fp8, int8 (only Ascend A2/A3 Series NPUs), nvfp4 or auto. Default is auto, which follows a priority order (server argument → deprecated env var → input\_global\_scale check → dispatcher\_output\_dtype from quant\_config → flashinfer/cutlass backend → NPU BF16 default → GPU FP8 default). `auto` auto, bf16, fp8, int8, nvfp4
--flashinfer-a2a-dispatch-type Select FlashInfer A2A dispatcher activation dtype. When omitted, it falls back to the SGLANG\_MOE\_NVFP4\_DISPATCH environment variable. Explicit auto selects mxfp8 for --quantization mxfp8, nvfp4 for modelopt FP4 or hybrid NVFP4 MoE checkpoints, and bf16 otherwise. The SGLANG\_MOE\_NVFP4\_DISPATCH environment variable cannot be set with this argument. None auto, bf16, nvfp4, mxfp8
`--ep-num-redundant-experts` Allocate this number of redundant experts in expert parallel. `0` Type: int
`--ep-dispatch-algorithm` The algorithm to choose ranks for redundant experts in expert parallel. None Type: str
`--init-expert-location` Initial location of EP experts. `trivial` Type: str
`--enable-eplb` Enable EPLB algorithm `False` bool flag (set to enable)
`--eplb-algorithm` Chosen EPLB algorithm `auto` Type: str
`--eplb-rebalance-num-iterations` Number of iterations to automatically trigger a EPLB re-balance. `1000` Type: int
`--eplb-rebalance-layers-per-chunk` Number of layers to rebalance per forward pass. None Type: int
`--eplb-min-rebalancing-utilization-threshold` Minimum threshold for GPU average utilization to trigger EPLB rebalancing. Must be in the range \[0.0, 1.0]. `1.0` Type: float
`--expert-distribution-recorder-mode` Mode of expert distribution recorder. ` None` Type: str
`--expert-distribution-recorder-buffer-size` Circular buffer size of expert distribution recorder. Set to -1 to denote infinite buffer. None Type: int
`--expert-balancedness-report-mode` Where to report expert balancedness. off off, server\_log, prometheus, both
`--deepep-config` Tuned DeepEP config suitable for your own cluster. It can be either a string with JSON content or a file path. ` None` Type: str
`--moe-dense-tp-size` TP size for MoE dense MLP layers. This flag is useful when, with large TP size, there are errors caused by weights in MLP layers having dimension smaller than the min dimension GEMM supports. ` none` Type: int
`--elastic-ep-backend` Specify the collective communication backend for elastic EP. Currently supports 'mooncake'. ` None` none, mooncake, nixl
`--enable-elastic-expert-backup` Enable elastic EP backend to backup expert weights in DRAM feature. Currently supports 'mooncake'. `False` bool flag (set to enable)
`--mooncake-ib-device` The InfiniBand devices for Mooncake Backend transfer, accepts multiple comma-separated devices (e.g., --mooncake-ib-device mlx5\_0,mlx5\_1). Default is None, which triggers automatic device detection when Mooncake Backend is enabled. ` None` Type: str
`--enable-waterfill` Enable Waterfill: dispatch the fused shared expert as an extra routed expert slot to the least-loaded EP rank. Supports DeepEP and MegaMOE MoE A2A backends, implicitly enables shared-expert fusion, and supports `--deepep-mode auto`, `normal`, or `low_latency` when used with DeepEP. Use `auto` or `low_latency` for production DeepEP decode so CUDA graph remains enabled. Supported on DeepSeek-V3/R1 with EP >= 2. By default, Waterfill uses the static local-batch path; set `SGLANG_DISABLE_STATIC_WATERFILL=1` to force dynamic Waterfill with runtime EP all-reduce. `False` bool flag (set to enable)
`--flashinfer-allreduce-fusion-backend` Enable FlashInfer allreduce fusion and choose backend. Defaults to auto. 'auto': choose mnnvl on SM90 single-node systems and SM100/SM103 single-node or multi-node systems; choose trtllm otherwise. 'trtllm': available on single-node systems only. 'mnnvl': available on SM90 single-node systems and SM100/SM103 single-node or multi-node systems via MNNVL fabric. Fuses allreduce with Residual + RMSNorm for supported MoE models. `None` auto, trtllm, mnnvl
`--enforce-disable-flashinfer-allreduce-fusion` Enforce disable FlashInfer allreduce fusion. `False` bool flag (set to enable)
## Mamba Cache
Argument Description Defaults Options
`--max-mamba-cache-size` The maximum size of the mamba cache. `None` Type: int
`--mamba-ssm-dtype` The data type of the SSM states in mamba cache. If not set, read from the model config. Auto (from model config) float32, bfloat16, float16
`--enable-mamba-cache-stochastic-rounding` Enable stochastic rounding when writing FP16 Mamba SSM cache states. Requires --mamba-ssm-dtype float16 and CUDA. With --mamba-backend triton, requires SM100. `False` Type: bool
`--mamba-cache-philox-rounds` Number of Philox rounds to use for stochastic rounding of FP16 Mamba SSM cache writes. Triton uses the Triton default when set to 0; FlashInfer uses 10 rounds when set to 0. `0` Type: int
`--mamba-full-memory-ratio` The ratio of mamba state memory to full kv cache memory. `0.9` Type: float
`--mamba-radix-cache-strategy` The strategy to use for mamba scheduler. auto currently defaults to no\_buffer. 1. no\_buffer does not support overlap scheduler due to not allocating extra mamba state buffers. Branching point caching support is feasible but not implemented. 2. extra\_buffer supports overlap schedule by allocating extra mamba state buffers to track mamba state for caching (mamba state usage per running req becomes 2x for non-spec; 1+(1/(2+speculative\_num\_draft\_tokens))x for spec dec (e.g. 1.16x if speculative\_num\_draft\_tokens==4)). 2a. extra\_buffer is strictly better for non-KV-cache-bound cases; for KV-cache-bound cases, the tradeoff depends on whether enabling overlap outweighs reduced max running requests. 2b. mamba caching at radix cache branching point is strictly better than non-branch but requires kernel support, currently only extra\_buffer supports branching. 3. extra\_buffer\_lazy lowers extra\_buffer's slot cost by allocating one track slot per request instead of two; the second slot is allocated on demand at track-interval boundaries (for speculative decoding it is reserved ahead of each verify window and committed only for accepted boundary crossings). Compatible with speculative decoding (EAGLE/NGRAM/DSPARK/DFLASH); not supported under PD disaggregation. `auto` auto, no\_buffer, extra\_buffer, extra\_buffer\_lazy
`--mamba-track-interval` The interval (in tokens) to track the mamba state during decode. Only used when --mamba-radix-cache-strategy is extra\_buffer. Must be divisible by page\_size if set, and must be >= speculative\_num\_draft\_tokens when using speculative decoding. `256` Type: int
`--enable-int8-mamba-checkpoint` Store radix-cached linear-attn (mamba) states in int8 (separate checkpoint pool) for \~2x cached-prefix capacity at fixed memory. `False` bool flag (set to enable)
`--int8-mamba-ckpt-size` Number of int8 mamba checkpoint slots (default: 2x the active mamba pool size). `None` Type: int
`--mamba-backend` Choose the kernel backend for Mamba SSM operations. Default is 'triton'. Options: 'triton' (default), 'flashinfer' (requires FlashInfer with Mamba support). `triton` triton, flashinfer
`--linear-attn-backend` The default kernel backend for linear attention (GDN/KDA). Can be overridden per-mode by --linear-attn-decode-backend and --linear-attn-prefill-backend. `triton` triton, cutedsl, flashinfer
`--linear-attn-decode-backend` Override the kernel backend for linear attention decode. If not set, uses --linear-attn-backend. `None` triton, cutedsl, flashinfer
`--linear-attn-prefill-backend` Override the kernel backend for linear attention prefill/extend. If not set, uses --linear-attn-backend. `None` triton, cutedsl, flashinfer
## Hierarchical cache
Argument Description Defaults Options
`--enable-hierarchical-cache` Enable hierarchical cache `False` bool flag (set to enable)
`--hicache-ratio` The ratio of the size of host KV cache memory pool to the size of device pool. `2.0` Type: float
`--hicache-size` The size of host KV cache memory pool in gigabytes, which will override the hicache\_ratio if set. `0` Type: int
`--hicache-write-policy` The write policy of hierarchical cache. `write_through` write\_back, write\_through, write\_through\_selective
`--hicache-io-backend` The IO backend for KV cache transfer between CPU and GPU `kernel` direct, kernel, kernel\_ascend
`--hicache-mem-layout` The layout of host memory pool for hierarchical cache. `page_first` layer\_first, page\_first, page\_first\_direct, page\_first\_kv\_split, page\_head
`--hicache-storage-backend` The storage backend for hierarchical KV cache. Built-in backends: file, mooncake, hf3fs, nixl, aibrix. For dynamic backend, use --hicache-storage-backend-extra-config to specify: backend\_name (custom name), module\_path (Python module path), class\_name (backend class name). `None` file, mooncake, hf3fs, nixl, aibrix, dynamic, eic, simm
`--hicache-storage-prefetch-policy` Control when prefetching from the storage backend should stop. `timeout` best\_effort, wait\_complete, timeout
`--hicache-storage-backend-extra-config` A dictionary in JSON string format, or a string starting with a `@` followed by a config file in JSON/YAML/TOML format, containing extra configuration for the storage backend. `None` Type: str
## Hierarchical sparse attention
Argument Description Defaults Options
`--hisparse-config`
`--hierarchical-sparse-attention-extra-config`		 A dictionary in JSON string format for hierarchical sparse attention configuration. Required fields: `algorithm` (str), `backend` (str). All other fields are algorithm-specific and passed to the algorithm constructor. `None` Type: str
`--enable-hisparse` Enable hierarchical sparse attention `False` bool flag (set to enable)
## LMCache
Argument Description Defaults Options
`--enable-lmcache` Using LMCache as an alternative hierarchical cache solution `False` bool flag (set to enable)
`--lmcache-config-file` Path to the LMCache YAML configuration file `None` Type: str
## Ktransformers
Argument Description Defaults Options
`--kt-weight-path` \[ktransformers parameter] The path of the quantized expert weights for amx kernel. A local folder. `None` Type: str
`--kt-method` \[ktransformers parameter] Quantization formats for CPU execution. `AMXINT4` Type: str
`--kt-cpuinfer` \[ktransformers parameter] The number of CPUInfer threads. `None` Type: int
`--kt-threadpool-count` \[ktransformers parameter] One-to-one with the number of NUMA nodes (one thread pool per NUMA). `2` Type: int
`--kt-num-gpu-experts` \[ktransformers parameter] The number of GPU experts. `None` Type: int
`--kt-max-deferred-experts-per-token` \[ktransformers parameter] Maximum number of experts deferred to CPU per token. All MoE layers except the final one use this value; the final layer always uses 0. `None` Type: int
## Diffusion LLM
Argument Description Defaults Options
`--dllm-algorithm` The diffusion LLM algorithm, such as LowConfidence. `None` Type: str
`--dllm-algorithm-config` The diffusion LLM algorithm configurations. Must be a YAML file. `None` Type: str
`--dllm-fdfo` First-Done-First-Out (FDFO) scheduling lets completed requests leave the batch immediately instead of waiting for slower requests, eliminating head-of-line blocking. Enabled by default; pass `--no-dllm-fdfo` to fall back to synchronous lockstep scheduling. Works with any dLLM algorithm. `True` Type: bool
## Offloading
Argument Description Defaults Options
`--cpu-offload-gb` How many GBs of RAM to reserve for CPU offloading. `0` Type: int
`--offload-group-size` Number of layers per group in offloading. `-1` Type: int
`--ple-offload-embedding` Keep the Qwen4-Exp PLE n-gram embedding table in host memory instead of device memory; the gather kernel reads it from a host pointer. Enabled by default for BF16 Qwen4-Exp on CUDA; `--no-ple-offload-embedding` disables it. `None` Type: bool
`--ple-offload-backend` Host storage for the offloaded PLE table: `pinned` (CPU pinned memory) or `file` (a sparse file-backed mmap under `--ple-offload-dir`, read directly by the gather kernel). Use `file` on unified-memory devices such as GB10 / DGX Spark, where pinned host memory comes out of the same pool as the weights; it requires a device that reports `cudaDevAttrPageableMemoryAccessUsesHostPageTables`. `pinned` Type: str
`--ple-offload-dir` Directory for the file-backed PLE table when `--ple-offload-backend` is `file`. The file is sparse and rewritten on every startup; keep it on fast local storage (NVMe). Rewriting an existing table can be much slower than filling a fresh sparse file. Before restarting, stop all servers using that directory and remove its `ple_table_*.bin` files to avoid this startup cost. `$SGLANG_CACHE_DIR/ple/` Type: str
`--offload-num-in-group` Number of layers to be offloaded within a group. `1` Type: int
`--offload-prefetch-step` Steps to prefetch in offloading. `1` Type: int
`--offload-mode` Mode of offloading. `cpu` Type: str
## Args for multi-item scoring
Argument Description Defaults Options
`--enable-mis` Enable Multi-Item Scoring optimization. Combines query and multiple items into a single sequence for efficient batch processing. Requires --attention-backend flashinfer; auto-disables CUDA graph, radix cache, and chunked prefill. `False` bool flag (set to enable)
## Optimization/debug options
Argument Description Defaults Options
`--disable-radix-cache` Disable RadixAttention for prefix caching. `False` bool flag (set to enable)
`--cuda-graph-config` Canonical per-phase CUDA graph settings as JSON, e.g. . JSON wins over the per-phase --cuda-graph-\* convenience flags and over the legacy flags. Allowed backends: full, breakable, disabled. `None` Type: JSON (dict-of-dicts)
`--cuda-graph-backend-decode` Backend for the decode phase. Folds into cuda\_graph\_config\[decode].backend. `None` full, breakable, disabled
`--cuda-graph-backend-prefill` Backend for the prefill phase. Folds into cuda\_graph\_config\[prefill].backend. `None` breakable, disabled
`--cuda-graph-max-bs-decode` Maximum batch size captured for the decode CUDA graph. `None` Type: int
`--cuda-graph-max-bs-prefill` Maximum batch size captured for the prefill CUDA graph. `None` Type: int
`--cuda-graph-max-seq-len-prefill` Longest sequence a prefill CUDA graph replay admits; longer batches run eager prefill. Folds into `cuda_graph_config[prefill].max_seq_len`. `None` Type: int
`--cuda-graph-bs-decode` Explicit list of batch sizes to capture for the decode CUDA graph. `None` List\[int]
`--cuda-graph-bs-prefill` Explicit list of batch sizes to capture for the prefill CUDA graph. `None` List\[int]
`--disable-cuda-graph-padding` Disable cuda graph when padding is needed. Still uses cuda graph when padding is not needed. `False` bool flag (set to enable)
`--enable-profile-cuda-graph` Enable profiling of cuda graph capture. `False` bool flag (set to enable)
`--debug-cuda-graph` Eager-mode CUDA graph via the breakable backend: graph breaks let every op run eagerly while still going through the capture/replay path. Useful for debugging capture/replay issues. `False` bool flag (set to enable)
`--enable-cudagraph-gc` Enable garbage collection during CUDA graph capture. If disabled (default), GC is frozen during capture to speed up the process. `False` bool flag (set to enable)
`--enable-layerwise-nvtx-marker` Enable layerwise NVTX profiling annotations for the model. This adds NVTX markers to every layer for detailed per-layer performance analysis with Nsight Systems. `False` bool flag (set to enable)
`--enable-nccl-nvls` Enable NCCL NVLS for prefill heavy requests when available. `False` bool flag (set to enable)
`--enable-symm-mem` Enable NCCL symmetric memory for fast collectives. `False` bool flag (set to enable)
`--disable-flashinfer-cutlass-moe-fp4-allgather` Disables quantize before all-gather for flashinfer cutlass moe. `False` bool flag (set to enable)
`--enable-tokenizer-batch-encode` Enable batch tokenization for improved performance when processing multiple text inputs. Do not use with image inputs, pre-tokenized input\_ids, or input\_embeds. `False` bool flag (set to enable)
`--disable-tokenizer-batch-decode` Disable batch decoding when decoding multiple completions. `False` bool flag (set to enable)
`--disable-outlines-disk-cache` Disable disk cache of outlines to avoid possible crashes related to file system or high concurrency. `False` bool flag (set to enable)
--boundary-reduction Select the optional FFN output reduction strategy. Unsupported reduce-scatter paths fall back to all-reduce followed by token redistribution. See FFN boundary reduction. auto auto, ar, rs, rsv, rs+rsv
`--disable-custom-all-reduce` Disable the custom all-reduce kernel and fall back to NCCL. `False` bool flag (set to enable)
`--enable-mscclpp` Enable using mscclpp for small messages for all-reduce kernel and fall back to NCCL. `False` bool flag (set to enable)
`--enable-torch-symm-mem` Enable using torch symm mem for all-reduce kernel and fall back to NCCL. Only supports CUDA device SM90 and above. SM90 supports world size 4, 6, 8. SM10 supports world size 6, 8. `False` bool flag (set to enable)
`--disable-overlap-schedule` Disable the overlap scheduler, which overlaps the CPU scheduler with GPU model worker. `False` bool flag (set to enable)
`--enable-mixed-chunk` Enabling mixing prefill and decode in a batch when using chunked prefill. `False` bool flag (set to enable)
`--enable-dp-attention` Deprecated. Use --attn-dp-size \ instead of --dp-size \ --enable-dp-attention. False bool flag (set to enable)
`--enable-dp-lm-head` Enable vocabulary parallel across the attention TP group to avoid all-gather across DP groups, optimizing performance under DP attention. `False` bool flag (set to enable)
`--enable-two-batch-overlap` Enabling two micro batches to overlap. `False` bool flag (set to enable)
`--enable-single-batch-overlap` Let computation and communication overlap within one micro batch. `False` bool flag (set to enable)
`--tbo-token-distribution-threshold` The threshold of token distribution between two batches in micro-batch-overlap, determines whether to two-batch-overlap or two-chunk-overlap. Set to 0 denote disable two-chunk-overlap. `0.48` Type: float
`--enable-torch-compile` Optimize the model with torch.compile. Experimental feature. `False` bool flag (set to enable)
`--enable-torch-compile-debug-mode` Enable debug mode for torch compile. `False` bool flag (set to enable)
`--torch-compile-max-bs` Set the maximum batch size when using torch compile. `32` Type: int
`--disable-cuda-graph` Deprecated. Use --cuda-graph-backend-decode=disabled and/or --cuda-graph-backend-prefill=disabled. False bool flag (set to enable)
`--disable-prefill-cuda-graph` Disable the prefill-phase CUDA graph. Convenience for --cuda-graph-backend-prefill=disabled. `False` bool flag (set to enable)
`--disable-decode-cuda-graph` Disable the decode-phase CUDA graph. Convenience for --cuda-graph-backend-decode=disabled. `False` bool flag (set to enable)
`--enable-p2p-check` Enable P2P check for GPU access, otherwise the p2p access is allowed by default. `False` bool flag (set to enable)
`--triton-attention-reduce-in-fp32` Cast the intermediate attention results to fp32 to avoid possible crashes related to fp16. This only affects Triton attention kernels. `False` bool flag (set to enable)
`--triton-attention-num-kv-splits` The number of KV splits in flash decoding Triton kernel. Larger value is better in longer context scenarios. The default value is 8. `8` Type: int
`--triton-attention-split-tile-size` The size of split KV tile in flash decoding Triton kernel. Used for deterministic inference. `None` Type: int
`--num-continuous-decode-steps` Run multiple continuous decoding steps to reduce scheduling overhead. This can potentially increase throughput but may also increase time-to-first-token latency. The default value is 1, meaning only run one decoding step at a time. `1` Type: int
`--delete-ckpt-after-loading` Delete the model checkpoint after loading the model. `False` bool flag (set to enable)
`--enable-memory-saver` Allow saving memory using release\_memory\_occupation and resume\_memory\_occupation `False` bool flag (set to enable)
`--enable-weights-cpu-backup` Save model weights to CPU memory during release\_weights\_occupation and resume\_weights\_occupation `False` bool flag (set to enable)
`--enable-draft-weights-cpu-backup` Save draft model weights to CPU memory during release\_weights\_occupation and resume\_weights\_occupation `False` bool flag (set to enable)
`--allow-auto-truncate` Allow automatically truncating requests that exceed the maximum input length instead of returning an error. `False` bool flag (set to enable)
`--enable-custom-logit-processor` Enable users to pass custom logit processors to the server (disabled by default for security) `False` bool flag (set to enable)
`--flashinfer-mla-disable-ragged` Not using ragged prefill wrapper when running flashinfer mla `False` bool flag (set to enable)
`--disable-shared-experts-fusion` Disable shared experts fusion optimization for deepseek v3/r1. `False` bool flag (set to enable)
`--disable-chunked-prefix-cache` Disable chunked prefix cache feature for deepseek, which should save overhead for short sequences. `False` bool flag (set to enable)
`--image-processor-backend` Image processor backend. `auto` lets Transformers select the best available backend. `auto` `auto`, `torchvision`, `pil`
`--disable-fast-image-processor` Deprecated. Use `--image-processor-backend=pil` instead. `False` bool flag (set to enable)
`--enable-return-hidden-states` Enable returning hidden states with responses. `False` bool flag (set to enable)
`--return-hidden-states-mode` Set the maximum hidden-state return mode supported by the server. `last` allows requests with return\_hidden\_states=False or `last`; `full` also allows return\_hidden\_states=True. `None` last, full
`--enable-return-routed-experts` Enable returning routed experts of each layer with responses. `False` bool flag (set to enable)
`--scheduler-recv-interval` The interval to poll requests in scheduler. Can be set to >1 to reduce the overhead of this. `1` Type: int
`--numa-node` Sets the numa node for the subprocesses. i-th element corresponds to i-th subprocess. `None` List\[int]
`--enable-deterministic-inference` Enable deterministic inference mode with batch invariant ops. `False` bool flag (set to enable)
`--rl-on-policy-target` The training system that SGLang needs to match for true on-policy. `None` fsdp
`--enable-attn-tp-input-scattered` Allow input of attention to be scattered when only using tensor parallelism, to reduce the computational load of operations such as qkv latent. `False` bool flag (set to enable)
`--enable-prefill-cp` Enable context parallelism for the prefill phase. Select the layout with --cp-strategy. `False` bool flag (set to enable)
`--cp-strategy` Sharding strategy for prefill CP. zigzag assigns each rank one early and one late sequence block; interleave assigns token indices modulo the CP size. None zigzag, interleave
`--enable-kv-cache-sharding` Shard at-rest KV cache storage at page granularity across the group that currently replicates it (attention-CP ranks for GQA models under prefill CP; attention-TP ranks for MLA models), multiplying the group's unique KV capacity. Prefill workers of PD disaggregation only. `False` bool flag (set to enable)
`--enable-fused-qk-norm-rope` Enable fused qk normalization and rope rotary embedding. `False` bool flag (set to enable)
`--enable-precise-embedding-interpolation` Enable corner alignment for resize of embeddings grid to ensure more accurate(but slower) evaluation of interpolated embedding values. `False` bool flag (set to enable)
`--kv-canary` KV cache canary mode. 'none' disables the canary (default). 'log' prints them while the server keeps running (production-safe). 'raise' fails the server on the first detected mismatch (CI lane). `none` none, log, raise
`--kv-canary-real-data` Check the real KV-cache in the canary. 'none' (default) disables the feature. 'partial' checks the first 16 bytes of each real-KV slot. 'all' checks the full real-KV slot. `none` Type: str
`--kv-canary-sweep-interval` Every N forward steps, run a full-pool sweep. `0` Type: int
`--pre-warm-nccl` Pre-warm NCCL/RCCL communicators during startup to reduce P99 TTFT cold-start latency. Default: enabled for AMD/HIP (RCCL), disabled for NVIDIA/CUDA (NCCL). `False` bool flag (set to enable)
`--enable-dp-attention-local-control-broadcast` With DP-attention, send control messages to every DP group leader and broadcast within attn\_tp\_group instead of the full tp\_group. Eliminates a costly all-ranks gloo sync on every scheduler iteration. `False` bool flag (set to enable)
`--enforce-shared-experts-fusion` Enforce shared experts fusion even when it would normally be disabled (e.g. under DeepEP). Mutually exclusive with --disable-shared-experts-fusion. `False` bool flag (set to enable)
`--enable-return-indexer-topk` Enable returning indexer topk indices of layers with indexer with responses. `False` bool flag (set to enable)
`--disable-attn-tp-gather` Disable scheduler-side attn\_tp\_gather (the upstream SP path that pads num\_tokens to attn\_tp\_size and pre-allocates a gathered buffer). Use for models that manage SP scatter/gather at the model level (e.g., perform their own all\_gather/reduce\_scatter inside attention) and do not consume the upstream gathered\_buffer. Without this, the cuda graph runner pads num\_tokens to attn\_tp\_size, which can cause kernel autotuners to select wrong-sized variants at small batches. `False` bool flag (set to enable)
`--enable-fused-moe-sum-all-reduce` Enable fused moe triton and sum all reduce. `False` bool flag (set to enable)
`--gc-threshold` Set the garbage collection thresholds (the collection frequency). Accepts 1 to 3 integers. — Type: int (one or more)
### FFN boundary reduction Use `--boundary-reduction` to choose which optional FFN output reductions may combine the sum with token redistribution. The selected policy applies to the dense and MoE FFN stages of decoders built with independently constructed stage boundaries (`declare_ffn` / `append_stages`; see [Layer boundaries](/docs/developer_guide/layer_boundary)). Other models keep their built-in FFN output communication and ignore this option. | Value | Behavior | | - | - | | `auto` | Resolve the model default during server-argument processing, before model construction. | | `ar` | Use all-reduce followed by token redistribution where needed. | | `rs` | Permit fixed-size reduce-scatter when the layout and backend support it; otherwise use `ar`. | | `rsv` | Permit variable-size reduce-scatter for eligible attention-DP paths; otherwise use `ar`. | | `rs+rsv` | Permit both, trying the eligible variable-size path first, then fixed-size, then `ar`. | `auto` selects `ar` for dense Qwen3 language backbones (including Qwen3-VL wrappers), Moss-VL and MiniCPM-V 4.5; `rsv` for BailingMoELinear/BailingMoeV2\_5, LongCat Flash and Step3-VL; and `rs+rsv` otherwise. Qwen3-MoE, Qwen3-Next and Qwen3.5 are not matched by the dense Qwen3 rule and resolve to `rs+rsv`. With `auto`, a draft loaded from a different path resolves its own model default; a draft on the target's model path (the MTP default) inherits the target policy. An explicit value applies to both target and draft. The result is recorded in the `speculative_boundary_reduction` server args field (`no_cli=True`); it has no separate CLI flag or independent override during resolution. This option does not disable required attention or MoE collectives, including expert all-to-all, or select the all-reduce fusion backend. For example, `ar` can still use fused all-reduce plus residual normalization; use `--enforce-disable-flashinfer-allreduce-fusion` to disable that FlashInfer fusion. ## Dynamic batch tokenizer
Argument Description Defaults Options
`--enable-dynamic-batch-tokenizer` Enable async dynamic batch tokenizer for improved performance when multiple requests arrive concurrently. `False` bool flag (set to enable)
`--dynamic-batch-tokenizer-batch-size` \[Only used if --enable-dynamic-batch-tokenizer is set] Maximum batch size for dynamic batch tokenizer. `32` Type: int
`--dynamic-batch-tokenizer-batch-timeout` \[Only used if --enable-dynamic-batch-tokenizer is set] Timeout in seconds for batching tokenization requests. `0.002` Type: float
## Debug tensor dumps
Argument Description Defaults Options
`--debug-tensor-dump-output-folder` The output folder for dumping tensors. `None` Type: str
`--debug-tensor-dump-layers` The layer ids to dump. Dump all layers if not specified. `None` Type: JSON list
`--debug-tensor-dump-input-file` The input filename for dumping tensors `None` Type: str
## PD disaggregation
Argument Description Defaults Options
`--disaggregation-mode` Only used for PD disaggregation. "prefill" for prefill-only server, and "decode" for decode-only server. If not specified, it is not PD disaggregated `null` null, prefill, decode
`--disaggregation-transfer-backend` The backend for disaggregation transfer. Default is mooncake. `mooncake` mooncake, nixl, ascend, fake, mori, mooncake\_tcp
`--disaggregation-bootstrap-port` Bootstrap server port on the prefill server. Default is 8998. `8998` Type: int
`--disaggregation-ib-device` The InfiniBand devices for disaggregation transfer, accepts single device (e.g., --disaggregation-ib-device mlx5\_0) or multiple comma-separated devices (e.g., --disaggregation-ib-device mlx5\_0,mlx5\_1). Default is None, which triggers automatic device detection when mooncake backend is enabled. None Type: str
`--disaggregation-decode-allocation-policy` When to reserve per-request decode KV slots. Set the same policy on prefill and decode. See requirements and tradeoffs. `early` early, prefill\_complete
`--disaggregation-decode-enable-offload-kvcache` Enable async KV cache offloading on decode server (PD mode). `False` bool flag (set to enable)
`--num-reserved-decode-tokens` Number of decode tokens that will have memory reserved when adding new request to the running batch. `512` Type: int
`--disaggregation-decode-polling-interval` The interval to poll requests in decode server. Can be set to >1 to reduce the overhead of this. `1` Type: int
`--disaggregation-decode-enable-radix-cache` Enable radix cache on decode server (PD mode). Caches KV prefixes to avoid redundant transfers. Incompatible with --enable-hisparse, speculative decoding, and --disaggregation-transfer-backend fake. `False` bool flag (set to enable)
`--optimistic-prefill-attempts` Number of optimistic prefill forward passes that skip the bootstrap wait. `0` Type: int
## Encode prefill disaggregation
Argument Description Defaults Options
`--encoder-only` For MLLM with an encoder, launch an encoder-only server `False` bool flag (set to enable)
`--language-only` For VLM, load weights for the language model only. `False` bool flag (set to enable)
`--language-model-only` Skip the multimodal encoder entirely: its weights are never loaded and the tower is never built, freeing that GPU memory for KV cache. Multimodal requests are rejected. Unlike --language-only this is a standalone mode, not part of encoder/decoder disaggregation. `False` bool flag (set to enable)
`--encoder-transfer-backend` The backend for encoder disaggregation transfer. Default is zmq\_to\_scheduler. `zmq_to_scheduler` zmq\_to\_scheduler, zmq\_to\_tokenizer, mooncake
`--encoder-urls` List of encoder server urls. `[]` Type: JSON list
`--encoder-bootstrap-port` Port for the EncoderBootstrapServer that runs in the language-only tokenizer manager process. Encoders register here, and language-only receivers fetch the current URL list from here. `8997` Type: int
`--encoder-register-urls` One or more EncoderBootstrapServer URLs to register this encoder with on startup, for dynamic encoder discovery. Example: --encoder-register-urls [http://prefill0:8997](http://prefill0:8997) [http://prefill1:8997](http://prefill1:8997). Used with --encoder-only servers. `[]` Type: str (one or more)
`--enable-adaptive-dispatch-to-encoder` When enabled, adaptively dispatch: multi-image requests go to encoder in language\_only epd mode, single-image requests are processed locally. `False` bool flag (set to enable)
## Custom weight loader
Argument Description Defaults Options
`--custom-weight-loader` The custom dataloader which used to update the model. Should be set with a valid import path, such as my\_package.weight\_load\_func None List\[str]
`--weight-loader-disable-mmap` Disable mmap while loading weight using safetensors. `False` bool flag (set to enable)
`--weight-loader-prefetch-checkpoints` Prefetch checkpoint files into OS page cache before loading. Each rank prefetches a fraction of the shards in a background thread, reducing total network I/O on shared filesystems (NFS/Lustre) from N\*checkpoint to 1\*checkpoint. Recommended for models on network storage. When enabled, multi-threaded safetensors loading is disabled by default to avoid I/O oversubscription with the prefetch threads; set `enable_multithread_load=true` in `--model-loader-extra-config` to keep multi-threaded loading (e.g. on local NVMe where prefetch is a no-op). `False` bool flag (set to enable)
`--weight-loader-prefetch-num-threads` Number of threads per rank for checkpoint prefetching. `4` Type: int
`--remote-instance-weight-loader-seed-instance-ip` The ip of the seed instance for loading weights from remote instance. None Type: str
`--remote-instance-weight-loader-seed-instance-service-port` The service port of the seed instance for loading weights from remote instance. None Type: int
`--remote-instance-weight-loader-send-weights-group-ports` The communication group ports for loading weights from remote instance. None Type: JSON list
`--remote-instance-weight-loader-backend` The backend for loading weights from remote instance. Can be 'transfer\_engine', 'nccl', or 'modelexpress'. Default is 'nccl'. `nccl` transfer\_engine, nccl, modelexpress
`--remote-instance-weight-loader-start-seed-via-transfer-engine` Start seed server via transfer engine backend for remote instance weight loader. `False` bool flag (set to enable)
`--weight-loader-drop-cache-after-load` Call posix\_fadvise(DONTNEED) on each safetensors shard after loading it. `False` bool flag (set to enable)
`--engine-info-bootstrap-port` Port for the engine info bootstrap server. Default is 6789. Must be set explicitly when running multiple instances on the same node. `6789` Type: int
`--modelexpress-config` JSON config for ModelExpress P2P weight loading. Keys: "url" (optional gRPC host:port override), "transport" ("nixl" or "transfer\_engine"). Example: '\{"url": "localhost:8001", "transport": "nixl"}' `None` Type: str
## For PD-Multiplexing
Argument Description Defaults Options
`--enable-pdmux` Enable PD-Multiplexing, PD running on greenctx stream. `False` bool flag (set to enable)
`--pdmux-config-path` The path of the PD-Multiplexing config file. `None` Type: str
`--sm-group-num` Number of sm partition groups. `8` Type: int
## Configuration file support
Argument Description Defaults Options
`--config` Read CLI options from a config file. Must be a YAML file with configuration options. `None` Type: str
## For Multi-Modal
Argument Description Defaults Options
`--enable-broadcast-mm-inputs-process` Enable broadcast mm-inputs process in scheduler. `False` bool flag (set to enable)
`--mm-process-config` Multimodal preprocessing config, a json config contains keys: image, video, audio. \{} Type: JSON / Dict
`--allowed-media-domains` Restrict client-supplied HTTP(S) media URLs and redirect destinations to these exact hostnames. Unrestricted Space-separated hostnames
`--media-url-max-file-size-mb` Maximum streamed size in MiB for one remote media download. Set to 0 to disable the limit. `64` Type: int
`--mm-enable-dp-encoder` Enabling data parallelism for mm encoder. The dp size will be set to the tp size automatically. `False` bool flag (set to enable)
`--limit-mm-data-per-request` Limit the number of multimodal inputs per request. e.g. '\{"image": 1, "video": 1, "audio": 1}' `None` Type: JSON / Dict
`--enable-mm-global-cache` Enable Mooncake-backed global multimodal embedding cache on encoder servers so repeated images can reuse cached ViT embeddings instead of recomputing them. `False` bool flag (set to enable)
## For checkpoint decryption
Argument Description Defaults Options
`--decrypted-config-file` The path of the decrypted config file. `None` Type: str
`--decrypted-draft-config-file` The path of the decrypted draft config file. `None` Type: str
`--enable-prefix-mm-cache` Enable prefix multimodal cache. Currently only supports mm-only. `False` bool flag (set to enable)
## Forward hooks
Argument Description Defaults Options
`--forward-hooks` JSON-formatted list of forward hook specifications. Each element must include `target_modules` (list of glob patterns matched against `model.named_modules()` names) and `hook_factory` (Python import path to a factory, e.g. `my_package.hooks:make_hook`). An optional `name` field is used for logging, and an optional `config` object is passed as a `dict` to the factory. `None` Type: JSON list
## For MindStudio-probe(msProbe) dump
Argument Description Defaults Options
`--msprobe-dump-config` The path of the JSON configuration file for msProbe. If specified, enables msProbe dump. `None` Type: str
# Session-Aware Radix Cache Source: https://docs.sglang.io/docs/advanced_features/session_radix_cache Session-aware radix caching improves cache hits for long-lived, multi-turn workloads under memory pressure. It registers reusable KV to a session and evicts unreferenced KV before KV still referenced by an active session. Session references are soft protection, not memory pins. Referenced KV can still be evicted when reclaiming unreferenced KV is insufficient. ## Enable the cache This feature is implemented only by `UnifiedRadixCache`. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path MODEL_PATH \ --enable-session-radix-cache ``` ## Pass and close the session Your application must pass the same top-level `session_id` on every request in a session. The ID labels cache references only; it does not append or reconstruct conversation context, so each request must still contain the intended prompt. ```bash Command theme={null} curl http://localhost:30000/generate \ -H "Content-Type: application/json" \ -d '{ "text": "FULL_PROMPT_FOR_THIS_TURN", "sampling_params": {"max_new_tokens": 128}, "session_id": "agent-42" }' ``` When a request finishes, SGLang automatically registers its reusable cache leaves under the `session_id`. This reference-only workflow does not require an `/open_session` call. Call `/close_session` when the application session ends, including error and cancellation paths: ```bash Command theme={null} curl -X POST http://localhost:30000/close_session \ -H "Content-Type: application/json" \ -d '{"session_id": "agent-42"}' ``` Closing removes the session's references but does not immediately free its KV. The KV remains reusable and returns to the normal eviction order. ## Eviction behavior The cache tracks references independently for each UnifiedRadixCache component. Device and host eviction use the same session preference. | Component | Referenced data | Eviction order | | - | - | - | | Full attention | The reusable prefix path from the registered leaf to the root | Unreferenced nodes first, then referenced nodes with fewer session references, then the configured policy such as LRU | | Sliding-window attention (SWA) | The reusable tail covering the sliding window plus page-alignment allowance | Two LRU passes: unreferenced nodes first, then referenced nodes if more space is required | | Mamba | The reusable state on the registered leaf | Two LRU passes: unreferenced nodes first, then referenced nodes if more space is required | UnifiedRadixCache still applies component cascade rules. Evicting an internal Full node also evicts its SWA and Mamba data; evicting SWA also evicts Mamba data; evicting Mamba affects only Mamba. Evicting a leaf removes all component data on that leaf. # SGLang Model Gateway Source: https://docs.sglang.io/docs/advanced_features/sgl_model_gateway SGLang Model Gateway is a high-performance model-routing gateway for large-scale LLM deployments. It centralizes worker lifecycle management, balances traffic across heterogeneous protocols (HTTP, gRPC, OpenAI-compatible), and provides enterprise-ready control over history storage, MCP tooling, and privacy-sensitive workflows. The gateway is deeply optimized for the SGLang serving runtime, but can route to any OpenAI-compatible backend. *** ## Table of Contents 1. [Overview](#overview) 2. [Architecture](#architecture) * [Control Plane](#control-plane) * [Data Plane](#data-plane) * [Storage and Privacy](#storage-and-privacy) 3. [Installation](#installation) 4. [Quick Start](#quick-start) 5. [Deployment Modes](#deployment-modes) * [Co-launch Router and Workers](#co-launch-router-and-workers) * [Separate Launch (HTTP)](#separate-launch-http) * [gRPC Launch](#grpc-launch) * [Prefill-Decode Disaggregation](#prefill-decode-disaggregation) * [OpenAI Backend Proxy](#openai-backend-proxy) * [Multi-Model Inference Gateway](#multi-model-inference-gateway) 6. [API Reference](#api-reference) * [Inference Endpoints](#inference-endpoints) * [Tokenization Endpoints](#tokenization-endpoints) * [Parser Endpoints](#parser-endpoints) * [Classification API](#classification-api) * [Conversation and Response APIs](#conversation-and-response-apis) * [Worker Management APIs](#worker-management-apis) * [Admin and Health Endpoints](#admin-and-health-endpoints) 7. [Load Balancing Policies](#load-balancing-policies) 8. [Reliability and Flow Control](#reliability-and-flow-control) * [Retries](#retries) * [Circuit Breaker](#circuit-breaker) * [Rate Limiting and Queuing](#rate-limiting-and-queuing) * [Health Checks](#health-checks) 9. [Reasoning Parser Integration](#reasoning-parser-integration) 10. [Tool Call Parsing](#tool-call-parsing) 11. [Tokenizer Management](#tokenizer-management) 12. [MCP Integration](#mcp-integration) 13. [Service Discovery (Kubernetes)](#service-discovery-kubernetes) 14. [History and Data Connectors](#history-and-data-connectors) 15. [WASM Middleware](#wasm-middleware) 16. [Language Bindings](#language-bindings) 17. [Security and Authentication](#security-and-authentication) * [TLS (HTTPS) for Gateway Server](#tls-https-for-gateway-server) * [mTLS for Worker Communication](#mtls-for-worker-communication) 18. [Observability](#observability) * [Prometheus Metrics](#prometheus-metrics) * [OpenTelemetry Tracing](#opentelemetry-tracing) * [Logging](#logging) 19. [Production Recommendations](#production-recommendations) * [Security Best Practices](#security-best-practices) * [High Availability](#high-availability) * [Performance](#performance) * [Kubernetes Deployment](#kubernetes-deployment) * [Monitoring with PromQL](#monitoring-with-promql) 20. [Configuration Reference](#configuration-reference) 21. [Troubleshooting](#troubleshooting) *** ## Overview * **Unified control plane** for registering, monitoring, and orchestrating regular, prefill, and decode workers across heterogeneous model fleets. * **Multi-protocol data plane** that routes traffic across HTTP, PD (prefill/decode), gRPC, and OpenAI-compatible backends with shared reliability primitives. * **Industry-first gRPC pipeline** with native Rust tokenization, reasoning parsers, and tool-call execution for high-throughput, OpenAI-compatible serving; supports both single-stage and PD topologies. * **Inference Gateway Mode (`--enable-igw`)** dynamically instantiates multiple router stacks (HTTP regular/PD, gRPC) and applies per-model policies for multi-tenant deployments. * **Conversation & responses connectors** centralize chat history inside the router so the same context can be reused across models and MCP loops without leaking data to upstream vendors (memory, none, Oracle ATP, PostgreSQL). * **Enterprise privacy**: agentic multi-turn `/v1/responses`, native MCP client (STDIO/HTTP/SSE/Streamable), and history storage all operate within the router boundary. * **Reliability core**: retries with jitter, worker-scoped circuit breakers, token-bucket rate limiting with queuing, background health checks, and cache-aware load monitoring. * **Comprehensive observability**: 40+ Prometheus metrics, OpenTelemetry distributed tracing, structured logging, and request ID propagation. *** ## Architecture ### Control Plane * **Worker Manager** discovers capabilities (`/get_server_info`, `/get_model_info`), tracks load, and registers/removes workers in the shared registry. * **Job Queue** serializes add/remove requests and exposes status (`/workers/{worker_id}`) so clients can track onboarding progress. * **Load Monitor** feeds cache-aware and power-of-two policies with live worker load statistics. * **Health Checker** continuously probes workers and updates readiness, circuit breaker state, and router metrics. * **Tokenizer Registry** manages dynamically registered tokenizers with async loading from HuggingFace or local paths. ### Data Plane * **HTTP routers** (regular & PD) implement `/generate`, `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/rerank`, `/v1/classify`, `/v1/tokenize`, `/v1/detokenize`, and associated admin endpoints. * **gRPC router** streams tokenized requests directly to SRT gRPC workers, running fully in Rust—tokenizer, reasoning parser, and tool parser all reside in-process. Supports both single-stage and PD routing, including embeddings and classification. * **OpenAI router** proxies OpenAI-compatible endpoints to external vendors (OpenAI, xAI, etc.) while keeping chat history and multi-turn orchestration local. ### Storage and Privacy * Conversation and response history is stored at the router tier (memory, none, Oracle ATP, or PostgreSQL). The same history can power multiple models or MCP loops without sending data to upstream vendors. * `/v1/responses` agentic flows, MCP sessions, and conversation APIs share the same storage layer, enabling compliance for regulated workloads. *** ## Installation ### Docker Pre-built Docker images are available on Docker Hub with multi-architecture support (x86\_64 and ARM64): ```bash Command theme={null} docker pull lmsysorg/sgl-model-gateway:latest ``` ### Prerequisites * **Rust and Cargo** ```bash Command theme={null} curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source "$HOME/.cargo/env" rustc --version cargo --version ``` * **Python** with `pip` and virtualenv tooling available. ### Rust Binary ```bash Command theme={null} cd sgl-model-gateway cargo build --release ``` ### Python Package ```bash Command theme={null} pip install maturin # Fast development mode cd sgl-model-gateway/bindings/python maturin develop # Production build maturin build --release --out dist --features vendored-openssl pip install --force-reinstall dist/*.whl ``` *** ## Quick Start ### Regular HTTP Routing ```bash Command theme={null} # Rust binary ./target/release/sgl-model-gateway \ --worker-urls http://worker1:8000 http://worker2:8000 \ --policy cache_aware # Python launcher python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 http://worker2:8000 \ --policy cache_aware ``` ### gRPC Routing ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls grpc://127.0.0.1:20000 \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --reasoning-parser deepseek-r1 \ --tool-call-parser json \ --host 0.0.0.0 --port 8080 ``` *** ## Deployment Modes ### Co-launch Router and Workers Launch the router and a fleet of SGLang workers in one process: ```bash Command theme={null} python -m sglang_router.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --dp-size 4 \ --host 0.0.0.0 \ --port 30000 ``` Comprehensive example with router arguments (prefixed with `--router-`): ```bash Command theme={null} python -m sglang_router.launch_server \ --host 0.0.0.0 \ --port 8080 \ --model meta-llama/Llama-3.1-8B-Instruct \ --tp-size 1 \ --dp-size 8 \ --grpc-mode \ --log-level debug \ --router-prometheus-port 10001 \ --router-tool-call-parser llama \ --router-model-path meta-llama/Llama-3.1-8B-Instruct \ --router-policy round_robin \ --router-log-level debug ``` ### Separate Launch (HTTP) Run workers independently and point the router at their HTTP endpoints: ```bash Command theme={null} # Worker nodes python -m sglang.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --port 8000 python -m sglang.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --port 8001 # Router node python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 http://worker2:8001 \ --policy cache_aware \ --host 0.0.0.0 --port 30000 ``` ### gRPC Launch Use SRT gRPC workers to unlock the highest throughput and access native reasoning/tool pipelines: ```bash Command theme={null} # Workers expose gRPC endpoints python -m sglang.launch_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --grpc-mode \ --port 20000 # Router python -m sglang_router.launch_router \ --worker-urls grpc://127.0.0.1:20000 \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --reasoning-parser deepseek-r1 \ --tool-call-parser json \ --host 0.0.0.0 --port 8080 ``` The gRPC router supports both regular HTTP-equivalent serving and PD (prefill/decode) serving. Provide `--tokenizer-path` or `--model-path` (HuggingFace ID or local directory) whenever connection mode resolves to gRPC. ### Prefill-Decode Disaggregation Split prefill and decode workers for PD-aware caching and balancing: ```bash Command theme={null} python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://prefill1:30001 9001 \ --decode http://decode1:30011 \ --prefill-policy cache_aware \ --decode-policy power_of_two ``` Prefill entries accept an optional bootstrap port. PD mode merges prefill metadata with decode outputs and streams results back to the client. ### OpenAI Backend Proxy Proxy OpenAI-compatible endpoints while keeping history and MCP sessions local: ```bash Command theme={null} python -m sglang_router.launch_router \ --backend openai \ --worker-urls https://api.openai.com \ --history-backend memory ``` OpenAI backend mode expects exactly one `--worker-urls` entry per router instance. ### Multi-Model Inference Gateway Enable IGW mode to route multiple models through a single router: ```bash Command theme={null} ./target/release/sgl-model-gateway \ --enable-igw \ --policy cache_aware \ --max-concurrent-requests 512 # Register workers dynamically curl -X POST http://localhost:30000/workers \ -H "Content-Type: application/json" \ -d '{ "url": "http://worker-a:8000", "model_id": "mistral", "priority": 10, "labels": {"tier": "gold"} }' ``` *** ## API Reference ### Inference Endpoints
Method Path Description
`POST` `/generate` SGLang generate API
`POST` `/v1/chat/completions` OpenAI-compatible chat completions (streaming/tool calls)
`POST` `/v1/completions` OpenAI-compatible text completions
`POST` `/v1/embeddings` Embedding generation (HTTP and gRPC)
`POST` `/v1/rerank`, `/rerank` Reranking requests
`POST` `/v1/classify` Text classification
### Tokenization Endpoints The gateway provides HTTP endpoints for text tokenization with batch support, designed to mirror the SGLang Python tokenization API.
Method Path Description
`POST` `/v1/tokenize` Tokenize text to token IDs (single or batch)
`POST` `/v1/detokenize` Convert token IDs back to text (single or batch)
`POST` `/v1/tokenizers` Register a new tokenizer (async, returns job status)
`GET` `/v1/tokenizers` List all registered tokenizers
`GET` `/v1/tokenizers/{id}` Get tokenizer info by UUID
`GET` `/v1/tokenizers/{id}/status` Check async tokenizer loading status
`DELETE` `/v1/tokenizers/{id}` Remove a tokenizer from the registry
#### Tokenize Request ```json Config theme={null} { "model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "Hello, world!" } ``` #### Batch Tokenize Request ```json Config theme={null} { "model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": ["Hello", "World", "How are you?"] } ``` #### Tokenize Response ```json Config theme={null} { "tokens": [15339, 11, 1917, 0], "count": 4, "char_count": 13 } ``` #### Detokenize Request ```json Config theme={null} { "model": "meta-llama/Llama-3.1-8B-Instruct", "tokens": [15339, 11, 1917, 0], "skip_special_tokens": true } ``` #### Detokenize Response ```json Config theme={null} { "text": "Hello, world!" } ``` #### Add Tokenizer (Async) ```bash Command theme={null} curl -X POST http://localhost:30000/v1/tokenizers \ -H "Content-Type: application/json" \ -d '{"name": "llama3", "source": "meta-llama/Llama-3.1-8B-Instruct"}' ``` Response: ```json Config theme={null} { "id": "550e8400-e29b-41d4-a716-446655440000", "status": "pending", "message": "Tokenizer registration queued" } ``` Check status: ```bash Command theme={null} curl http://localhost:30000/v1/tokenizers/550e8400-e29b-41d4-a716-446655440000/status ``` ### Parser Endpoints The gateway provides admin endpoints for parsing reasoning content and function calls from LLM outputs.
Method Path Description
`POST` `/parse/reasoning` Separate reasoning (`<think>`) from normal text
`POST` `/parse/function_call` Parse function/tool calls from text
#### Separate Reasoning Request ```json Config theme={null} { "text": "<think>Let me analyze this step by step...</think>The answer is 42.", "parser": "deepseek-r1" } ``` #### Response ```json Config theme={null} { "normal_text": "The answer is 42.", "reasoning_text": "Let me analyze this step by step..." } ``` #### Function Call Parsing ```json Config theme={null} { "text": "{\"name\": \"get_weather\", \"arguments\": {\"city\": \"NYC\"}}", "parser": "json" } ``` ### Classification API The `/v1/classify` endpoint provides text classification using sequence classification models (e.g., `Qwen2ForSequenceClassification`, `BertForSequenceClassification`). #### Request ```bash Command theme={null} curl http://localhost:30000/v1/classify \ -H "Content-Type: application/json" \ -d '{ "model": "jason9693/Qwen2.5-1.5B-apeach", "input": "I love this product!" }' ``` #### Response ```json Config theme={null} { "id": "classify-a1b2c3d4-5678-90ab-cdef-1234567890ab", "object": "list", "created": 1767034308, "model": "jason9693/Qwen2.5-1.5B-apeach", "data": [ { "index": 0, "label": "positive", "probs": [0.12, 0.88], "num_classes": 2 } ], "usage": { "prompt_tokens": 6, "completion_tokens": 0, "total_tokens": 6 } } ``` #### Response Fields
Field Description
`label` Predicted class label (from model's `id2label` config, or `LABEL_N` fallback)
`probs` Probability distribution over all classes (softmax of logits)
`num_classes` Number of classification classes
#### Notes * Classification reuses the embedding backend—the scheduler returns logits which are converted to probabilities via softmax * Labels come from the model's HuggingFace config (`id2label` field); models without this mapping use generic labels (`LABEL_0`, `LABEL_1`, etc.) * Both HTTP and gRPC routers support classification ### Conversation and Response APIs
Method Path Description
`POST` `/v1/responses` Create background responses (agentic loops)
`GET` `/v1/responses/{id}` Retrieve stored response
`POST` `/v1/responses/{id}/cancel` Cancel background response
`DELETE` `/v1/responses/{id}` Delete response
`GET` `/v1/responses/{id}/input_items` List response input items
`POST` `/v1/conversations` Create conversation
`GET` `/v1/conversations/{id}` Get conversation
`POST` `/v1/conversations/{id}` Update conversation
`DELETE` `/v1/conversations/{id}` Delete conversation
`GET` `/v1/conversations/{id}/items` List conversation items
`POST` `/v1/conversations/{id}/items` Add items to conversation
`GET` `/v1/conversations/{id}/items/{item_id}` Get conversation item
`DELETE` `/v1/conversations/{id}/items/{item_id}` Delete conversation item
### Worker Management APIs
Method Path Description
`POST` `/workers` Queue worker registration (returns 202 Accepted)
`GET` `/workers` List workers with health, load, and policy metadata
`GET` `/workers/{worker_id}` Inspect specific worker or job queue entry
`PUT` `/workers/{worker_id}` Queue worker update
`DELETE` `/workers/{worker_id}` Queue worker removal
#### Add Worker ```bash Command theme={null} curl -X POST http://localhost:30000/workers \ -H "Content-Type: application/json" \ -d '{"url":"grpc://0.0.0.0:31000","worker_type":"regular"}' ``` #### List Workers ```bash Command theme={null} curl http://localhost:30000/workers ``` Response: ```json Config theme={null} { "workers": [ { "id": "2f3a0c3e-3a7b-4c3f-8c70-1b7d4c3a6e1f", "url": "http://0.0.0.0:31378", "model_id": "mistral", "priority": 50, "cost": 1.0, "worker_type": "regular", "is_healthy": true, "load": 0, "connection_mode": "Http" } ], "total": 1, "stats": { "prefill_count": 0, "decode_count": 0, "regular_count": 1 } } ``` ### Admin and Health Endpoints
Method Path Description
`GET` `/liveness` Health check (always returns OK)
`GET` `/readiness` Readiness check (checks healthy worker availability)
`GET` `/health` Alias for liveness
`GET` `/health_generate` Health generate test
`GET` `/engine_metrics` Engine-level metrics from workers
`GET` `/v1/models` List available models
`GET` `/get_model_info` Get model information
`GET` `/get_server_info` Get server information
`POST` `/flush_cache` Clear all caches
`GET` `/get_loads` Get all worker loads
`POST` `/wasm` Upload WASM module
`GET` `/wasm` List WASM modules
`DELETE` `/wasm/{module_uuid}` Remove WASM module
*** ## Load Balancing Policies
Policy Description Usage
`random` Uniform random selection `--policy random`
`round_robin` Cycles through workers in order `--policy round_robin`
`power_of_two` Samples two workers and picks the lighter one `--policy power_of_two`
`cache_aware` Combines cache locality with load balancing (default) `--policy cache_aware`
`bucket` Divides workers into load buckets with dynamic boundaries `--policy bucket`
### Cache-Aware Policy Tuning ```bash Command theme={null} --cache-threshold 0.5 \ --balance-abs-threshold 32 \ --balance-rel-threshold 1.5 \ --eviction-interval-secs 120 \ --max-tree-size 67108864 ```
Parameter Default Description
`--cache-threshold` 0.3 Minimum prefix match ratio for cache hit
`--balance-abs-threshold` 64 Absolute load difference before rebalancing
`--balance-rel-threshold` 1.5 Relative load ratio before rebalancing
`--eviction-interval-secs` 120 Cache eviction cadence in seconds
`--max-tree-size` 67108864 Maximum nodes in cache tree
*** ## Reliability and Flow Control ### HTTP Client Configure upstream HTTP client connection settings:
Parameter Default Description
`--pool-idle-timeout-secs` 50 Idle timeout in seconds for pooled upstream HTTP connections. Can also be set with `SMG_POOL_IDLE_TIMEOUT_SECS`.
`--connect-timeout-secs` 10 Timeout in seconds for new upstream HTTP connections. Can also be set with `SMG_CONNECT_TIMEOUT_SECS`.
`--pool-max-idle-per-host` 500 Maximum idle upstream HTTP connections to keep per host. Can also be set with `SMG_POOL_MAX_IDLE_PER_HOST`.
`--tcp-keepalive-secs` 30 TCP keepalive idle time in seconds for upstream HTTP connections. Can also be set with `SMG_TCP_KEEPALIVE_SECS`.
### Retries Configure exponential backoff retries: ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 http://worker2:8001 \ --retry-max-retries 5 \ --retry-initial-backoff-ms 50 \ --retry-max-backoff-ms 30000 \ --retry-backoff-multiplier 1.5 \ --retry-jitter-factor 0.2 ```
Parameter Default Description
`--retry-max-retries` 5 Maximum retry attempts
`--retry-initial-backoff-ms` 50 Initial backoff duration (ms)
`--retry-max-backoff-ms` 5000 Maximum backoff duration (ms)
`--retry-backoff-multiplier` 2.0 Exponential backoff multiplier
`--retry-jitter-factor` 0.1 Random jitter factor (0.0-1.0)
`--disable-retries` false Disable retries entirely
**Retryable Status Codes:** 408, 429, 500, 502, 503, 504 ### Circuit Breaker Per-worker circuit breakers prevent cascading failures: ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 http://worker2:8001 \ --cb-failure-threshold 5 \ --cb-success-threshold 2 \ --cb-timeout-duration-secs 30 \ --cb-window-duration-secs 60 ```
Parameter Default Description
`--cb-failure-threshold` 5 Consecutive failures to open circuit
`--cb-success-threshold` 2 Successes to close from half-open
`--cb-timeout-duration-secs` 30 Time before half-open attempt
`--cb-window-duration-secs` 60 Failure counting window
`--disable-circuit-breaker` false Disable circuit breaker
**Circuit Breaker States:** * **Closed**: Normal operation, requests allowed * **Open**: Failing, requests rejected immediately * **Half-Open**: Testing recovery, limited requests allowed ### Rate Limiting and Queuing ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 http://worker2:8001 \ --max-concurrent-requests 256 \ --rate-limit-tokens-per-second 512 \ --queue-size 128 \ --queue-timeout-secs 30 ``` Requests beyond the concurrency limit wait in a FIFO queue. Returns: * `429 Too Many Requests` when queue is full * `408 Request Timeout` when queue timeout expires ### Health Checks ```bash Command theme={null} --health-check-interval-secs 30 \ --health-check-timeout-secs 10 \ --health-success-threshold 2 \ --health-failure-threshold 3 \ --health-check-endpoint /health ``` *** ## Reasoning Parser Integration The gateway includes built-in reasoning parsers for models that use Chain-of-Thought (CoT) reasoning with explicit thinking blocks. ### Supported Parsers
Parser ID Model Family Think Tokens
`deepseek-r1` DeepSeek-R1 `<think>...</think>` (initial reasoning)
`qwen3` Qwen-3 `<think>...</think>`
`qwen3-thinking` Qwen-3 Thinking `<think>...</think>` (initial reasoning)
`kimi` Kimi K2 Unicode think tokens
`glm45` GLM-4.5/4.6/4.7 `<think>...</think>`
`step3` Step-3 `<think>...</think>`
`minimax` MiniMax `<think>...</think>`
### Usage ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls grpc://127.0.0.1:20000 \ --model-path deepseek-ai/DeepSeek-R1 \ --reasoning-parser deepseek-r1 ``` The gRPC router automatically: 1. Detects reasoning blocks in streaming output 2. Separates reasoning content from normal text 3. Applies incremental streaming parsing with buffer management 4. Handles partial token detection for correct streaming behavior *** ## Tool Call Parsing The gateway supports parsing function/tool calls from LLM outputs in multiple formats. ### Supported Formats
Parser Format Description
`json` JSON Standard JSON tool calls
`python` Pythonic Python function call syntax
`xml` XML XML-formatted tool calls
### Usage ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls grpc://127.0.0.1:20000 \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --tool-call-parser json ``` *** ## Tokenizer Management ### Tokenizer Sources The gateway supports multiple tokenizer backends: * **HuggingFace**: Load from HuggingFace Hub by model ID * **Local**: Load from local `tokenizer.json` or directory * **Tiktoken**: Auto-detect OpenAI GPT models (gpt-4, davinci, etc.) ### Configuration ```bash Command theme={null} # HuggingFace model --model-path meta-llama/Llama-3.1-8B-Instruct # Local tokenizer --tokenizer-path /path/to/tokenizer.json # With chat template override --chat-template /path/to/template.jinja ``` ### Tokenizer Caching Two-level caching for optimal performance:
Cache Type Description
L0 Exact match Whole-string caching for repeated prompts
L1 Prefix match Prefix boundary matching for incremental prompts
```bash Command theme={null} --enable-l0-cache \ --l0-max-entries 10000 \ --enable-l1-cache \ --l1-max-memory 52428800 # 50MB ``` *** ## MCP Integration The gateway provides native Model Context Protocol (MCP) client integration for tool execution. ### Supported Transports
Transport Description
STDIO Local process execution
SSE Server-Sent Events (HTTP)
Streamable Bidirectional streaming
### Configuration ```bash Command theme={null} python -m sglang_router.launch_router \ --mcp-config-path /path/to/mcp-config.yaml \ --worker-urls http://worker1:8000 ``` ### MCP Configuration File ```yaml Config theme={null} servers: - name: "filesystem" command: "npx" args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] protocol: "stdio" required: false - name: "github" url: "https://api.github.com/mcp" token: "ghp_xxxxx" protocol: "sse" required: false - name: "custom-tools" url: "https://tools.example.com/mcp" protocol: "streamable" required: true pool: max_connections: 100 idle_timeout: 300 proxy: http: "http://proxy.internal:8080" https: "https://proxy.internal:8443" no_proxy: "localhost,127.0.0.1,*.internal" inventory: enable_refresh: true tool_ttl: 300 refresh_interval: 300 ``` *** ## Service Discovery (Kubernetes) Enable automatic worker discovery via Kubernetes pod selectors: ```bash Command theme={null} python -m sglang_router.launch_router \ --service-discovery \ --selector app=sglang-worker role=inference \ --service-discovery-namespace production \ --service-discovery-port 8000 ``` ### PD Mode Discovery ```bash Command theme={null} --pd-disaggregation \ --prefill-selector app=sglang component=prefill \ --decode-selector app=sglang component=decode \ --service-discovery ``` Prefill pods can expose bootstrap ports via the `sglang.ai/bootstrap-port` annotation. RBAC must allow `get`, `list`, and `watch` on pods. *** ## History and Data Connectors
Backend Description Usage
`memory` In-memory storage (default) `--history-backend memory`
`none` No persistence `--history-backend none`
`oracle` Oracle Autonomous Database `--history-backend oracle`
`postgres` PostgreSQL Database `--history-backend postgres`
`redis` Redis `--history-backend redis`
### Oracle Configuration ```bash Command theme={null} # Connection descriptor export ATP_DSN="(description=(address=(protocol=tcps)(port=1522)(host=adb.region.oraclecloud.com))(connect_data=(service_name=service_name)))" # Or TNS alias (requires wallet) export ATP_TNS_ALIAS="sglroutertestatp_high" export ATP_WALLET_PATH="/path/to/wallet" # Credentials export ATP_USER="admin" export ATP_PASSWORD="secret" export ATP_POOL_MIN=4 export ATP_POOL_MAX=32 python -m sglang_router.launch_router \ --backend openai \ --worker-urls https://api.openai.com \ --history-backend oracle ``` ### PostgreSQL Configuration ```bash Command theme={null} export POSTGRES_DB_URL="postgres://user:password@host:5432/dbname" python -m sglang_router.launch_router \ --backend openai \ --worker-urls https://api.openai.com \ --history-backend postgres ``` ### Redis Configuration ```bash Command theme={null} export REDIS_URL="redis://localhost:6379" export REDIS_POOL_MAX=16 export REDIS_RETENTION_DAYS=30 python -m sglang_router.launch_router \ --backend openai \ --worker-urls https://api.openai.com \ --history-backend redis \ --redis-retention-days 30 ``` Use `--redis-retention-days -1` for persistent storage (default is 30 days). *** ## WASM Middleware The gateway supports WebAssembly (WASM) middleware modules for custom request/response processing. This enables organization-specific logic for authentication, rate limiting, billing, logging, and more—without modifying or recompiling the gateway. ### Overview WASM middleware runs in a sandboxed environment with memory isolation, no network/filesystem access, and configurable resource limits.
Attach Point When Executed Use Cases
`OnRequest` Before forwarding to workers Auth, rate limiting, request modification
`OnResponse` After receiving worker response Logging, response modification, error handling
Action Description
`Continue` Proceed without modification
`Reject(status)` Reject request with HTTP status code
`Modify(...)` Modify headers, body, or status
### Examples Complete working examples are available in `examples/wasm/`:
Example Description
`auth/` API key authentication for protected routes
`rate_limit/` Per-client rate limiting (requests/minute)
`logging/` Request tracking headers and response modification
The interface definition is located at `src/wasm/interface`. ### Building Modules ```bash Command theme={null} # Prerequisites rustup target add wasm32-wasip2 cargo install wasm-tools # Build cargo build --target wasm32-wasip2 --release # Convert to component format wasm-tools component new \ target/wasm32-wasip2/release/my_middleware.wasm \ -o my_middleware.component.wasm ``` ### Deploying Modules ```bash Command theme={null} # Enable WASM support python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 \ --enable-wasm # Upload module curl -X POST http://localhost:30000/wasm \ -H "Content-Type: application/json" \ -d '{ "modules": [{ "name": "auth-middleware", "file_path": "/absolute/path/to/auth.component.wasm", "module_type": "Middleware", "attach_points": [{"Middleware": "OnRequest"}] }] }' # List modules curl http://localhost:30000/wasm # Remove module curl -X DELETE http://localhost:30000/wasm/{module_uuid} ``` ### Runtime Configuration
Parameter Default Description
`max_memory_pages` 1024 (64MB) Maximum WASM memory
`max_execution_time_ms` 1000 Execution timeout
`max_stack_size` 1MB Stack size limit
`module_cache_size` 10 Cached modules per worker
**Note:** Rate limiting state is per-worker thread and not shared across gateway replicas. For production, consider implementing rate limiting at a shared layer (e.g., Redis) *** ## Language Bindings SGLang Model Gateway provides official language bindings for Python and Go, enabling integration with different technology stacks and organizational requirements. ### Python Bindings The Python bindings provide a PyO3-based wrapper around the Rust gateway library. This is a straightforward binding that calls the gateway server startup from Python. #### Installation ```bash Command theme={null} # From PyPI pip install sglang-router # Development build cd sgl-model-gateway/bindings/python pip install maturin && maturin develop --features vendored-openssl ``` #### Usage The Python bindings are used throughout this documentation. See the [Quick Start](#quick-start) and [Deployment Modes](#deployment-modes) sections for detailed examples. Key components: * `RouterArgs` dataclass with 50+ configuration options * `Router.from_args()` for programmatic startup * CLI commands: `smg launch`, `smg server`, `python -m sglang_router.launch_router` ### Go Bindings The Go bindings provide a high-performance gRPC client library for organizations with Go-based infrastructure. This is ideal for: * Integration with internal Go services and tooling * High-performance client applications * Building custom OpenAI-compatible proxy servers #### Architecture ```text Output theme={null} +-------------------------------------------+ | High-Level Go API | | (client.go - OpenAI-style interface) | +-------------------------------------------+ | gRPC Layer | +-------------------------------------------+ | Rust FFI Layer | | (Tokenization, Parsing, Conversion) | +-------------------------------------------+ ``` **Key Features:** * Native Rust tokenization via FFI (thread-safe, lock-free) * Full streaming support with context cancellation * Configurable channel buffer sizes for high concurrency * Built-in tool call parsing and chat template application #### Installation ```bash Command theme={null} # Build the FFI library first cd sgl-model-gateway/bindings/golang make build && make lib # Then use in your Go project go get github.com/sgl-project/sgl-go-sdk ``` **Requirements:** Go 1.24+, Rust toolchain #### Examples Complete working examples are available in `bindings/golang/examples/`:
Example Description
`simple/` Non-streaming chat completion
`streaming/` Streaming chat completion with SSE
`oai_server/` Full OpenAI-compatible HTTP server
```bash Command theme={null} # Run examples cd sgl-model-gateway/bindings/golang/examples/simple && ./run.sh cd sgl-model-gateway/bindings/golang/examples/streaming && ./run.sh cd sgl-model-gateway/bindings/golang/examples/oai_server && ./run.sh ``` #### Testing ```bash Command theme={null} cd sgl-model-gateway/bindings/golang # Unit tests go test -v ./... # Integration tests (requires running SGLang server) export SGL_GRPC_ENDPOINT=grpc://localhost:20000 export SGL_TOKENIZER_PATH=/path/to/tokenizer go test -tags=integration -v ./... ``` ### Comparison
Feature Python Go
**Primary Use** Gateway server launcher gRPC client library
**CLI Support** Full CLI (smg, sglang-router) Library only
**K8s Discovery** Native support N/A (client library)
**PD Mode** Built-in N/A (client library)
**When to Use Python:** Launching and managing the gateway server, service discovery, PD disaggregation. **When to Use Go:** Building custom client applications, integration with Go microservices, OpenAI-compatible proxy servers *** ## Security and Authentication ### Router API Key ```bash Command theme={null} python -m sglang_router.launch_router \ --api-key "your-router-api-key" \ --worker-urls http://worker1:8000 ``` Clients must supply `Authorization: Bearer ` for protected endpoints. ### Worker API Keys ```bash Command theme={null} # Add worker with explicit key curl -H "Authorization: Bearer router-key" \ -X POST http://localhost:8080/workers \ -H "Content-Type: application/json" \ -d '{"url":"http://worker:8000","api_key":"worker-key"}' ``` ### Security Configurations 1. **No Authentication** (default): Use only in trusted environments 2. **Router-only Authentication**: Clients authenticate to router 3. **Worker-only Authentication**: Router open, workers require keys 4. **Full Authentication**: Both router and workers protected ### TLS (HTTPS) for Gateway Server Enable TLS to serve the gateway over HTTPS: ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 \ --tls-cert-path /path/to/server.crt \ --tls-key-path /path/to/server.key ```
Parameter Description
`--tls-cert-path` Path to server certificate (PEM format)
`--tls-key-path` Path to server private key (PEM format)
Both parameters must be provided together. The gateway uses rustls with the ring crypto provider for TLS termination. If TLS is not configured, the gateway falls back to plain HTTP. ### mTLS for Worker Communication Enable mutual TLS (mTLS) for secure communication with workers in HTTP mode: ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls https://worker1:8443 https://worker2:8443 \ --client-cert-path /path/to/client.crt \ --client-key-path /path/to/client.key \ --ca-cert-path /path/to/ca.crt ```
Parameter Description
`--client-cert-path` Path to client certificate for mTLS (PEM format)
`--client-key-path` Path to client private key for mTLS (PEM format)
`--ca-cert-path` Path to CA certificate for verifying worker TLS (PEM format, repeatable)
**Key Points:** * Client certificate and key must be provided together * Multiple CA certificates can be added with multiple `--ca-cert-path` flags * Uses rustls backend when TLS is configured * Single HTTP client is created for all workers (assumes single security domain) * TCP keepalive (30 seconds) is enabled for long-lived connections ### Full TLS Configuration Example Gateway HTTPS + Worker mTLS + API Key authentication: ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls https://worker1:8443 https://worker2:8443 \ --tls-cert-path /etc/certs/server.crt \ --tls-key-path /etc/certs/server.key \ --client-cert-path /etc/certs/client.crt \ --client-key-path /etc/certs/client.key \ --ca-cert-path /etc/certs/ca.crt \ --api-key "secure-api-key" \ --policy cache_aware ``` *** ## Observability ### Prometheus Metrics Enable with `--prometheus-host`/`--prometheus-port` (defaults to `0.0.0.0:29000`). #### Metric Categories (40+ metrics)
Layer Prefix Metrics
HTTP `smg_http_*` `requests_total`, `request_duration_seconds`, `responses_total`, `connections_active`, `rate_limit_total`
Router `smg_router_*` `requests_total`, `request_duration_seconds`, `request_errors_total`, `stage_duration_seconds`, `upstream_responses_total`
Inference `smg_router_*` `ttft_seconds`, `tpot_seconds`, `tokens_total`, `generation_duration_seconds`
Worker `smg_worker_*` `pool_size`, `connections_active`, `requests_active`, `health_checks_total`, `selection_total`, `errors_total`
Circuit Breaker `smg_worker_cb_*` `state`, `transitions_total`, `outcomes_total`, `consecutive_failures`, `consecutive_successes`
Retry `smg_worker_*` `retries_total`, `retries_exhausted_total`, `retry_backoff_seconds`
Discovery `smg_discovery_*` `registrations_total`, `deregistrations_total`, `sync_duration_seconds`, `workers_discovered`
MCP `smg_mcp_*` `tool_calls_total`, `tool_duration_seconds`, `servers_active`, `tool_iterations_total`
Database `smg_db_*` `operations_total`, `operation_duration_seconds`, `connections_active`, `items_stored`
#### Key Inference Metrics (gRPC mode)
Metric Type Description
`smg_router_ttft_seconds` Histogram Time to first token
`smg_router_tpot_seconds` Histogram Time per output token
`smg_router_tokens_total` Counter Total tokens (input/output)
`smg_router_generation_duration_seconds` Histogram End-to-end generation time
#### Duration Buckets 1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s, 15s, 30s, 45s, 60s, 90s, 120s, 180s, 240s ### OpenTelemetry Tracing Enable distributed tracing with OTLP export: ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 \ --enable-trace \ --otlp-traces-endpoint localhost:4317 ``` #### Features * OTLP/gRPC exporter (default port 4317) * W3C Trace Context propagation for HTTP and gRPC * Batch span processing (500ms delay, 64 span batch size) * Custom filtering to reduce noise * Trace context injection into upstream worker requests * Service name: `sgl-router` ### Logging ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 \ --log-level debug \ --log-dir ./router_logs ``` Structured tracing with optional file sink. Log levels: `debug`, `info`, `warn`, `error`. ### Request ID Propagation ```bash Command theme={null} --request-id-headers x-request-id x-trace-id x-correlation-id ``` Responses include `x-request-id` header for correlation. *** ## Production Recommendations This section provides guidance for deploying SGLang Model Gateway in production environments. ### Security Best Practices **Always enable TLS in production:** ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls https://worker1:8443 https://worker2:8443 \ --tls-cert-path /etc/certs/server.crt \ --tls-key-path /etc/certs/server.key \ --client-cert-path /etc/certs/client.crt \ --client-key-path /etc/certs/client.key \ --ca-cert-path /etc/certs/ca.crt \ --api-key "${ROUTER_API_KEY}" ``` **Security Checklist:** * Enable TLS for gateway HTTPS termination * Enable mTLS for worker communication when workers are on untrusted networks * Set `--api-key` to protect router endpoints * Use Kubernetes Secrets or a secrets manager for credentials * Rotate certificates and API keys periodically * Restrict network access with firewalls or network policies ### High Availability **Scaling Strategy:** The gateway supports running multiple replicas behind a load balancer for high availability. However, there are important considerations:
Component Shared Across Replicas Impact
Worker Registry No (independent) Each replica discovers workers independently
Radix Cache Tree No (independent) Cache hits may decrease by 10-20%
Circuit Breaker State No (independent) Each replica tracks failures independently
Rate Limiting No (independent) Limits apply per-replica, not globally
**Recommendations:** 1. **Prefer horizontal scaling over vertical scaling**: Deploy multiple smaller gateway replicas rather than one large instance with excessive CPU and memory. This provides: * Better fault tolerance (single replica failure doesn't take down the gateway) * More predictable resource usage * Easier capacity planning 2. **Use Kubernetes Service Discovery**: Let the gateway automatically discover and manage workers: ```bash Command theme={null} python -m sglang_router.launch_router \ --service-discovery \ --selector app=sglang-worker \ --service-discovery-namespace production ``` 3. **Accept cache efficiency trade-off**: With multiple replicas, the cache-aware routing policy's radix tree is not synchronized across replicas. This means: * Each replica builds its own cache tree * Requests from the same user may hit different replicas * Expected cache hit rate reduction: **10-20%** * This is often acceptable given the HA benefits 4. **Configure session affinity (optional)**: If cache efficiency is critical, configure your load balancer for session affinity based on a consistent hash of the request (e.g., user ID or API key). **Example HA Architecture:** ```text Output theme={null} +-------------------+ | Load Balancer | | (L4/L7) | +---------+---------+ | +-------------------+-------------------+ | | | v v v +-----------+ +-----------+ +-----------+ | Gateway | | Gateway | | Gateway | | Replica 1 | | Replica 2 | | Replica 3 | +-----+-----+ +-----+-----+ +-----+-----+ | | | +-------------------+-------------------+ | +-------------------+-------------------+ | | | v v v +-----------+ +-----------+ +-----------+ | Worker | | Worker | | Worker | | Pod 1 | | Pod 2 | | Pod N | +-----------+ +-----------+ +-----------+ ``` ### Performance **Use gRPC mode for high throughput:** gRPC mode provides the highest performance for SGLang workers: ```bash Command theme={null} # Start workers in gRPC mode python -m sglang.launch_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --grpc-mode \ --port 20000 # Configure gateway for gRPC python -m sglang_router.launch_router \ --worker-urls grpc://worker1:20000 grpc://worker2:20000 \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --policy cache_aware ``` **Performance Benefits of gRPC:** * Native Rust tokenization (no Python overhead) * Streaming with lower latency * Built-in reasoning parser execution * Tool call parsing in the gateway * Reduced serialization overhead **Tuning Recommendations:**
Parameter Recommendation Reason
`--policy` `cache_aware` Best for repeated prompts, \~30% latency reduction
`--max-concurrent-requests` 2-4x worker count Prevent overload while maximizing throughput
`--queue-size` 2x max-concurrent Buffer for burst traffic
`--request-timeout-secs` Based on max generation length Prevent stuck requests
### Kubernetes Deployment **Pod Labeling for Service Discovery:** For the gateway to discover workers automatically, label your worker pods consistently: ```yaml Config theme={null} # Worker Deployment (Regular Mode) apiVersion: apps/v1 kind: Deployment metadata: name: sglang-worker namespace: production spec: replicas: 4 selector: matchLabels: app: sglang-worker component: inference template: metadata: labels: app: sglang-worker component: inference model: llama-3-8b spec: containers: - name: worker image: lmsysorg/sglang:latest ports: - containerPort: 8000 name: http - containerPort: 20000 name: grpc ``` **Gateway configuration for discovery:** ```bash Command theme={null} python -m sglang_router.launch_router \ --service-discovery \ --selector app=sglang-worker component=inference \ --service-discovery-namespace production \ --service-discovery-port 8000 ``` **PD (Prefill/Decode) Mode Labeling:** ```yaml Config theme={null} # Prefill Worker metadata: labels: app: sglang-worker component: prefill annotations: sglang.ai/bootstrap-port: "9001" # Decode Worker metadata: labels: app: sglang-worker component: decode ``` **Gateway configuration for PD discovery:** ```bash Command theme={null} python -m sglang_router.launch_router \ --service-discovery \ --pd-disaggregation \ --prefill-selector app=sglang-worker component=prefill \ --decode-selector app=sglang-worker component=decode \ --service-discovery-namespace production ``` **RBAC Requirements:** The gateway needs permissions to watch pods: ```yaml Config theme={null} apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: sglang-gateway namespace: production rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] *** apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: sglang-gateway namespace: production subjects: - kind: ServiceAccount name: sglang-gateway namespace: production roleRef: kind: Role name: sglang-gateway apiGroup: rbac.authorization.k8s.io ``` ### Monitoring with PromQL Configure Prometheus to scrape the gateway metrics endpoint (default: `:29000/metrics`). **Essential Dashboards:** **1. Request Rate and Latency:** ```sql Example theme={null} # Request rate by endpoint sum(rate(smg_http_requests_total[5m])) by (path, method) # P50 latency histogram_quantile(0.50, sum(rate(smg_http_request_duration_seconds_bucket[5m])) by (le)) # P99 latency histogram_quantile(0.99, sum(rate(smg_http_request_duration_seconds_bucket[5m])) by (le)) # Error rate sum(rate(smg_http_responses_total{status=~"5.."}[5m])) / sum(rate(smg_http_responses_total[5m])) ``` **2. Worker Health:** ```sql Example theme={null} # Healthy workers sum(smg_worker_pool_size) # Active connections per worker smg_worker_connections_active # Worker health check failures sum(rate(smg_worker_health_checks_total{result="failure"}[5m])) by (worker_id) ``` **3. Circuit Breaker Status:** ```sql Example theme={null} # Circuit breaker states (0=closed, 1=open, 2=half-open) smg_worker_cb_state # Circuit breaker transitions sum(rate(smg_worker_cb_transitions_total[5m])) by (worker_id, from_state, to_state) # Workers with open circuits count(smg_worker_cb_state == 1) ``` **4. Inference Performance (gRPC mode):** ```sql Example theme={null} # Time to first token (P50) histogram_quantile(0.50, sum(rate(smg_router_ttft_seconds_bucket[5m])) by (le, model)) # Time per output token (P99) histogram_quantile(0.99, sum(rate(smg_router_tpot_seconds_bucket[5m])) by (le, model)) # Token throughput sum(rate(smg_router_tokens_total[5m])) by (model, direction) # Generation duration P95 histogram_quantile(0.95, sum(rate(smg_router_generation_duration_seconds_bucket[5m])) by (le)) ``` **5. Rate Limiting and Queuing:** ```sql Example theme={null} # Rate limit rejections sum(rate(smg_http_rate_limit_total{decision="rejected"}[5m])) # Queue depth (if using concurrency limiting) smg_worker_requests_active # Retry attempts sum(rate(smg_worker_retries_total[5m])) by (worker_id) # Exhausted retries (failures after all retries) sum(rate(smg_worker_retries_exhausted_total[5m])) ``` **6. MCP Tool Execution:** ```sql Example theme={null} # Tool call rate sum(rate(smg_mcp_tool_calls_total[5m])) by (server, tool) # Tool latency P95 histogram_quantile(0.95, sum(rate(smg_mcp_tool_duration_seconds_bucket[5m])) by (le, tool)) # Active MCP server connections smg_mcp_servers_active ``` **Alerting Rules Example:** ```yaml Config theme={null} groups: - name: sglang-gateway rules: - alert: HighErrorRate expr: | sum(rate(smg_http_responses_total{status=~"5.."}[5m])) / sum(rate(smg_http_responses_total[5m])) > 0.05 for: 5m labels: severity: critical annotations: summary: "High error rate on SGLang Gateway" - alert: CircuitBreakerOpen expr: count(smg_worker_cb_state == 1) > 0 for: 2m labels: severity: warning annotations: summary: "Worker circuit breaker is open" - alert: HighLatency expr: | histogram_quantile(0.99, sum(rate(smg_http_request_duration_seconds_bucket[5m])) by (le)) > 30 for: 5m labels: severity: warning annotations: summary: "P99 latency exceeds 30 seconds" - alert: NoHealthyWorkers expr: sum(smg_worker_pool_size) == 0 for: 1m labels: severity: critical annotations: summary: "No healthy workers available" ``` *** ## Configuration Reference ### Core Settings
Parameter Type Default Description
`--host` str 127.0.0.1 Router host
`--port` int 30000 Router port
`--worker-urls` list \[] Worker URLs (HTTP or gRPC)
`--policy` str cache\_aware Routing policy
`--max-concurrent-requests` int -1 Concurrency limit (-1 disables)
`--request-timeout-secs` int 600 Request timeout
`--max-payload-size` int 256MB Maximum request payload
### Prefill/Decode
Parameter Type Default Description
`--pd-disaggregation` flag false Enable PD mode
`--prefill` list \[] Prefill URLs + optional bootstrap ports
`--decode` list \[] Decode URLs
`--prefill-policy` str None Override policy for prefill nodes
`--decode-policy` str None Override policy for decode nodes
`--worker-startup-timeout-secs` int 600 Worker init timeout
### Kubernetes Discovery
Parameter Type Description
`--service-discovery` flag Enable discovery
`--selector` list Label selectors (key=value)
`--prefill-selector` / `--decode-selector` list PD mode selectors
`--service-discovery-namespace` str Namespace to watch
`--service-discovery-port` int Worker port (default 80)
`--bootstrap-port-annotation` str Annotation for bootstrap ports
### TLS Configuration
Parameter Type Description
`--tls-cert-path` str Server certificate for gateway HTTPS (PEM)
`--tls-key-path` str Server private key for gateway HTTPS (PEM)
`--client-cert-path` str Client certificate for worker mTLS (PEM)
`--client-key-path` str Client private key for worker mTLS (PEM)
`--ca-cert-path` str CA certificate for verifying workers (PEM, repeatable)
*** ## Troubleshooting ### Workers Never Ready Increase `--worker-startup-timeout-secs` or ensure health probes respond before router startup. ### Load Imbalance / Hot Workers Inspect `smg_router_requests_total` by worker and tune cache-aware thresholds (`--balance-*`, `--cache-threshold`). ### Circuit Breaker Flapping Increase `--cb-failure-threshold` or extend the timeout/window durations. Consider temporarily disabling retries. ### Queue Overflow (429) Increase `--queue-size` or reduce client concurrency. Ensure `--max-concurrent-requests` matches downstream capacity. ### Memory Growth Reduce `--max-tree-size` or lower `--eviction-interval-secs` for more aggressive cache pruning. ### Debugging ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 \ --log-level debug \ --log-dir ./router_logs ``` ### gRPC Connection Issues Ensure workers are started with `--grpc-mode` and verify `--model-path` or `--tokenizer-path` is provided to the router. ### Tokenizer Loading Failures Check HuggingFace Hub credentials (`HF_TOKEN` environment variable) for private models. Verify local paths are accessible. *** SGLang Model Gateway continues to evolve alongside the SGLang runtime. Keep CLI flags, integrations, and documentation aligned when adopting new features or contributing improvements. # SGLang for RL Systems Source: https://docs.sglang.io/docs/advanced_features/sglang_for_rl This document is a practical guide for infrastructure teams integrating SGLang into RL and post-training systems. It focuses on the operational pain points in the loop (rollout, evaluation, training, weight sync) and maps them to concrete SGLang APIs, flags, and integration patterns. The focus is on maximizing rollout efficiency, accuracy and stability while keeping rollout-serving behavior aligned in production environments. ## Why SGLang for RL Lifecycle? Let's embrace a guiding principle from early DeepMind's RL engineering: **Be a library, not a framework.** This philosophy empowers innovation by providing SGLang as flexible tools, not rigid structures. Here are five reasons to use SGLang for your RL lifecycle: * **Fine-Grained Engine Sleep and Wake Up**: facilitate maximum-powered rollout and training * **Open-To-Use Refit Functionality**: diverse methods for co-location or disaggregation * **Easy To Postpone Generation**: enable partial rollout and dedicated rollout control * **Deterministic Inference**: achieve deterministic inference to enable zero training-inference mismatch * **Load Balancing Router**: cache-aware load-balancing for high-throughput rollout The following sections cover these aspects in detail. ## Fine-Grained Engine Sleep and Wake Up Rollout and training are both memory-intensive, and co-locating them on the same GPUs often leads to memory pressure and slow handoffs. SGLang provides a memory-aware sleep/wake mechanism that releases KV cache and weights while keeping the server process alive, then resumes them for rollout without a full restart. This avoids repeated disk I/O and CUDA graph recapture during each RL step. Under the hood, the RL team uses CUDA-graph-aware weight offload via [torch\_memory\_saver](https://github.com/fzyzcjy/torch_memory_saver) to preserve virtual memory addresses for graph replay. For details, see: [Efficient RL Training - Optimizing Memory Usage in verl](https://hebiao064.github.io/rl-memory-management). ### Server flag Enable memory saver support when launching the server: ```text Output theme={null} --enable-memory-saver ``` ### Release Memory **Endpoint:** `POST /release_memory_occupation` **Request body:**
Field Description Defaults Options
`tags` Which memory regions to release. If omitted, all are released. `None` Type: list\[str], values: `kv_cache`, `weights`
**Behavior notes:** * This call asserts there are no ongoing requests. Ensure the engine is idle before calling it. * If `kv_cache` is released, SGLang flushes cache; subsequent requests will rebuild KV cache as needed. ### Resume Memory **Endpoint:** `POST /resume_memory_occupation` **Request body:**
Field Description Defaults Options
`tags` Which memory regions to resume. If omitted, all are resumed. `None` Type: list\[str], values: `kv_cache`, `weights`
## Open-To-Use Refit Functionality After training completes each step, rollout engines must be refit with new weights. SGLang supports three refit strategies so you can match your infrastructure style (co-located vs disaggregated) and scaling needs. Each strategy maps to a concrete API with clear request schemas. For a deeper dive into SGLang's weight update utilities, see [RL System Deep Thinking: Weight Update Mechanisms](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/sys-design/readme-1-EN.md). **How to choose:** * **From disk** is simplest and best for elastic rollout scaling and checkpointing. * **From tensor** is best for co-located training/rollout when you can pass in-memory tensors. * **From distributed** is best for disaggregated training/rollout with dedicated communication groups (NCCL/IB). ### Update Weights from Disk **When to use:** * Save checkpoint to disk and update weights from disk * Dynamic scaling (new rollout instances can load from the same checkpoint) **Why it works well:** This path trades some I/O overhead for simplicity and flexibility. It integrates naturally with checkpointing and makes it trivial to add new rollout engines: point them at the same checkpoint and call the API. It is also the safest option for high availability because the checkpoint itself is the source of truth. **Endpoint:** `POST /update_weights_from_disk` **Request body:**
Field Description Defaults Options
`model_path` The model path with the new weights. Required Type: str
`load_format` The format to load the weights. `None` Type: str
`abort_all_requests` Abort all running requests before update. `False` Type: bool
`weight_version` Optional weight version label tracked by the server. `None` Type: str
`is_async` Perform weight load asynchronously. `False` Type: bool
`torch_empty_cache` Empty torch cache. `False` Type: bool
`keep_pause` Keep scheduler paused after update. `False` Type: bool
`recapture_cuda_graph` Recapture CUDA graphs after update. `False` Type: bool
`token_step` Trainer step id for rollout bookkeeping. `0` Type: int
`flush_cache` Flush KV cache after update. `True` Type: bool
**Response body:**
Field Description Defaults Options
`success` Whether the update succeeded. - Type: bool
`message` Status / error message. - Type: str
`num_paused_requests` Number of paused requests during update. `0` Type: int
**Python Engine API:** `engine.update_weights_from_disk(model_path, load_format=None)` **Diffusion engine (SGLang-Diffusion):** The diffusion engine exposes the same `POST /update_weights_from_disk` endpoint with the following behavior: * **All-or-nothing with rollback:** if any module fails to load, all previously updated modules are rolled back to the original weights by reloading from the original model path. No partial updates are left behind. If rollback itself fails, the exception propagates so the caller knows the model is in an inconsistent state. * **Offload-aware:** when layerwise offload (`--dit-layerwise-offload`) is enabled, the diffusion offload manager replaces GPU parameters with small `torch.empty((1,))` placeholders while real weights live in consolidated pinned CPU buffers. A naive `param.data.copy_()` would fail with a shape mismatch. Instead, the updater dynamically detects active offload managers and writes new weights directly into their CPU buffers, bypassing the placeholders entirely. For any layer that happens to be prefetched on GPU at update time, the live GPU tensor is also updated so the change takes effect immediately. This requires no extra GPU memory and does not disturb the offload state. * **DTensor-aware:** parameters distributed via `torch.distributed.tensor` (tensor parallelism) are updated through `distribute_tensor` so that each shard is correctly placed on the right device mesh. **Request body:**
Field Description Defaults Options
model\_path The model path with the new weights. Required Type: str
flush\_cache Flush TeaCache state after update. True Type: bool
target\_modules List of module names to update (e.g. \["transformer"]). If omitted, all nn.Module components are updated. None Type: list\[str]
**Response body:**
Field Description Defaults Options
success Whether the update succeeded. - Type: bool
message Status / error message. - Type: str
> **Note:** The diffusion engine (SGLang-Diffusion) does not currently support hot refit (updating weights while inference is in progress). The diffusion scheduler processes one request at a time and completes the entire inference before handling the next request, so weight updates and inference never run concurrently. ### Weight Update Session `update_weights_from_tensor` and `update_weights_from_distributed` must run inside a weight-update session. The update calls only write checkpoint-format values; `end_weight_update` runs every quantized layer's `process_weights_after_loading` once, turning them into kernel layout (Marlin repack, MXFP8 shuffles, ...), and runs `post_load_weights` if no weights were loaded through the engine during the session (P2P/RDMA writes). `begin_weight_update` first restores the few schemes whose kernel layout changes parameter shapes (today compressed-tensors W4A16 MoE on Marlin) so the checkpoint-format values fit. One session can span many update calls, so the finalize cost is paid once per refit rather than once per bucket. A call outside a session, a second `begin_weight_update`, or an `end_weight_update` without one is rejected with `success: false` (HTTP 400) and leaves the weights untouched. **Endpoints:** `POST /begin_weight_update`, `POST /end_weight_update` `begin_weight_update` accepts `selector`: `"target"`, `"draft"`, or `"all"` (default). Under speculative decoding the same selector on the update endpoints picks which model runners receive the weights; `end_weight_update` finalizes the runners the session opened. ```python theme={null} engine.begin_weight_update() for bucket in buckets: engine.update_weights_from_tensor(bucket) engine.end_weight_update() ``` ### Update Weights from Tensor **When to use:** * Co-located training and rollout, where training can provide tensors directly * Fast in-memory updates **Important constraints:** This strategy requires the training process and rollout engine to share access to the tensors. Co-located setups must keep the model on GPU; moving tensors to CPU will break the update path. For high-performance MoE or specialized attention kernels, co-location may limit some optimizations compared to disaggregated rollouts. **Endpoint:** `POST /update_weights_from_tensor` **Request body:**
Field Description Defaults Options
serialized\_named\_tensors Per-TP serialized tensor payloads. Required Type: list\[str|bytes]
load\_format Optional load format selector. None None, direct, flattened\_bucket, or a custom loader path string
flush\_cache Flush KV cache after update. True Type: bool
abort\_all\_requests Abort all running requests before update. False Type: bool
weight\_version Optional version label tracked by the server. None Type: str
**Note:** The serialized tensor payloads must be created with `MultiprocessingSerializer.serialize(...)` and should be base64-safe strings. **Python Engine API:** `engine.update_weights_from_tensor(named_tensors, load_format=None, flush_cache=True)`, inside `begin_weight_update()` / `end_weight_update()`. ### Update Weights from Distributed Group **When to use:** * Disaggregated training and rollout * NCCL or IB-backed weight broadcast from training workers to rollout workers **How it works:** Training workers gather weights (typically on TP rank 0), broadcast them to the rollout group, and each rollout TP shard loads the parameters it needs. This avoids disk I/O and keeps training and rollout decoupled, at the cost of managing a dedicated communication group. **Initialize weight update group** **Endpoint:** `POST /init_weights_update_group` **Request body:**
Field Description Defaults Options
`master_address` Group master address. Required Type: str
`master_port` Group master port. Required Type: int
`rank_offset` Offset for local rank mapping. Required Type: int
`world_size` Total world size. Required Type: int
`group_name` Group name. `weight_update_group` Type: str
`backend` Communication backend. `nccl` Type: str
**Update weight** **Endpoint:** `POST /update_weights_from_distributed` **Request body:**
Field Description Defaults Options
`names` Parameter names to update. Required Type: list\[str]
`dtypes` Dtype strings for each parameter. Required Type: list\[str]
`shapes` Tensor shapes. Required Type: list\[list\[int]]
`group_name` Group name. `weight_update_group` Type: str
`flush_cache` Flush KV cache after update. `True` Type: bool
`abort_all_requests` Abort all running requests before update. `False` Type: bool
`weight_version` Optional version label. `None` Type: str
`load_format` Optional format selector. `None` `None` or `flattened_bucket`
**Destroy weights update group** **Endpoint:** `POST /destroy_weights_update_group` **Request body:**
Field Description Defaults Options
`group_name` Group name. `weight_update_group` Type: str
**Python Engine APIs:** * `engine.init_weights_update_group(...)` * `engine.begin_weight_update(selector="all")` * `engine.update_weights_from_distributed(names, dtypes, shapes, ...)` * `engine.end_weight_update()` * `engine.destroy_weights_update_group(group_name)` ### Per-Token Weight Version Attribution A request can outlive a version change: the RL flow retracts it with `pause_generation`, refits, and continues it, or `POST /update_weight_version` relabels while it is still generating. `meta_info` then reports which tokens came from which weights: ```json Output theme={null} "weight_version": "42", "weight_versions": [ {"version": "41", "start": 0, "end": 57}, {"version": "42", "start": 57, "end": 128} ] ``` * Ranges are half-open `[start, end)` over output-token indices, prompt excluded. The usual case is a single span. ## Easy To Postpone Generation Multi-turn RL rollouts often suffer from long-tail requests that block the entire batch. A small number of slow interactions can stall all GPUs, and the long-tail behavior makes profiling and monitoring difficult. SGLang exposes explicit pause/resume APIs so you can pause slow requests and continue them later. This pattern matches systems like [APRIL](https://arxiv.org/abs/2509.18521), terminate once enough responses are collected, and recycle incomplete responses in the next step. The result is higher GPU utilization without discarding partial work. `pause_generation` --- update weights --- `continue_generation` is the correct execution flow when updating weights from training. An update can only happen when SGLang is not actively processing inference tasks. ### Pause Generation **Endpoint:** `POST /pause_generation` **Request body:**
Field Description Defaults Options
`mode` Pause mode. `abort` `abort`, `retract`, `in_place`
**Modes:** * `abort`: Default behavior, identical to `abort` endpoint with `abort_all` set. Pending requests from `waiting_queue` and `running_queue` will be returned immediately to the caller. * `retract`: Put engine in "paused" state. Move running requests back to waiting queue. KV cache can be flushed and recomputed later. * `in_place`: Put engine in "paused" state without changing states of the requests. Running requests rely on availability of KV caches to continue, so any subsequent `flush_cache` call will be unsuccessful. ### Continue Generation **Endpoint:** `POST /continue_generation` ## Deterministic Inference In many RL stacks, rollout and training are implemented with different kernels or batching behavior. Even when weights are identical, token probabilities can drift, silently breaking the on-policy assumption. This is the training–inference mismatch problem. SGLang supports a deterministic inference mode that reduces non-determinism across batch shapes. This mitigates variance introduced by runtime batching and kernel selection. To further achieve true on-policy training, you need to modify the training engine to use the same deterministic kernels. For implementation details, see these miles examples: [True On-Policy](https://github.com/radixark/miles/tree/main/examples/true_on_policy) and [True On-Policy for VLM](https://github.com/radixark/miles/tree/main/examples/true_on_policy_vlm). For additional context, see the blog post [Let Speed Be With Stability: All-In-One Solution to Training-Inference Mismatch with Miles](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/slime/mismatch/blog-en.md). **Server flag:** ```text Output theme={null} --enable-deterministic-inference ``` For more details, see [Deterministic Inference](./deterministic_inference) ## Load Balancing Router SGLang Model Gateway is the recommended control plane for large‑scale RL rollouts. It provides async, non‑blocking request handling, cache‑aware load balancing, and fault‑tolerant routing across rollout and reward servers. This lets you keep GPUs saturated while avoiding long‑tail stalls and brittle, engine‑local concurrency logic. It has been deployed in the training of GLM 4.5+ models and proven to be highly efficient in production-level large-scale RL workloads. Key benefits for RL infrastructure: * **Async non-blocking efficiency**: SGLang’s native async server/router architecture (HTTPS/gRPC) manages concurrency automatically. This guarantees maximum GPU saturation and effective continuous batching without requiring complex, manual implementation by engineers. * **Elasticity and fault tolerance**: By encapsulating the reward model and rollout as independent servers, SGLang decouples them logically and physically. This architecture provides robust disaster recovery for large-scale distributed training; if a server fails, the router automatically redirects traffic to healthy nodes, ensuring the training process continues without interruption. * **Training–Inference alignment**: Using the SGLang Model Gateway for both training and inference ensures "What You See Is What You Get." This eliminates score discrepancies and the painful backend alignment issues often caused by using different engines for training versus deployment. * **Dynamic load balancing and long-tail mitigation**: Unlike static partitioning, the SGLang Model Gateway enables request-level dynamic dispatching for multi-turn RL. It can distribute different turns of a conversation across different servers to balance workloads and eliminate long-tail latency caused by varying sequence lengths. For deployment and configuration, see: [SGLang Model Gateway](./sgl_model_gateway) # SGLang Simulator Source: https://docs.sglang.io/docs/advanced_features/sglang_simulator SGLang Simulator reuses SGLang's scheduler, request lifecycle, and KV-cache implementation while replacing model forward execution with a latency predictor. Use it to compare scheduling and cache configurations on timestamped or synthetic workloads without loading model weights. ## Supported scope SGLang Simulator tracks the current `main` branch and recent SGLang releases. The current integration is validated with `v0.5.16`, `v0.5.17`, `v0.5.18`, and `main`. The initial upstream scope uses one simulated worker with `tp_size=1`, `ep_size=1`, `dp_size=1`, and `pp_size=1`. A simulator configuration can describe a larger target system for latency prediction, but the SGLang runtime process topology remains single-worker. The simulator supports: * synthetic request rates, ShareGPT workloads, and timestamped Autobench traces; * OFFLINE logical-time simulation and BLOCKING wall-clock replay; * AIConfigurator, ML, and replay latency predictors; * SGLang prefix caching and [HiCache](/docs/advanced_features/hicache); and * serving-compatible TTFT, TPOT, ITL, throughput, and cache-hit metrics. ## Install from the SGLang repository Use the simulator and SGLang source from the same monorepo checkout: ```bash theme={null} python3 -m pip install -e tools/sglang-simulator export PYTHONPATH="$PWD/tools/sglang-simulator/src:$PWD/python" ``` AIConfigurator is optional. Install the validated extra only when you use an AIConfigurator predictor: ```bash theme={null} python3 -m pip install -e "tools/sglang-simulator[aic]" ``` ## Start a simulator server Choose a fresh output directory for every run. The server owns the simulation mode and writes metrics to this directory. ```bash theme={null} export SGLANG_USE_CPU_ENGINE=1 export CUDA_VISIBLE_DEVICES="" export SGLANG_SIMULATOR_OUTPUT_MODE=OFFLINE export SGLANG_SIMULATOR_OUTPUT_DIR=/tmp/sglang-simulator-quickstart python3 -m sglang_simulator.simulation.sglang.launch_server \ --model-path tools/sglang-simulator/test/assets/qwen3-8b \ --tokenizer-path tools/sglang-simulator/examples/assets/tokenizer \ --sim-config-path tools/sglang-simulator/examples/sim_configs/replay.json \ --port 30000 ``` `OFFLINE` advances the simulator's logical clock without sleeping. `BLOCKING` also sleeps for predicted forward and cache-load latency, which is useful when a client must observe simulated wall-clock pacing. ## Send a workload In another terminal, export the same output directory and run the simulator-aware serving benchmark from the repository root: ```bash theme={null} export PYTHONPATH="$PWD/tools/sglang-simulator/src:$PWD/python" export SGLANG_SIMULATOR_OUTPUT_DIR=/tmp/sglang-simulator-quickstart python3 benchmark/simulator/bench_serving.py \ --simulator-mode offline \ --backend sglang \ --base-url http://127.0.0.1:30000 \ --model tools/sglang-simulator/test/assets/qwen3-8b \ --tokenizer tools/sglang-simulator/examples/assets/tokenizer \ --dataset-name sharegpt \ --dataset-path tools/sglang-simulator/examples/workloads/sharegpt-example.json \ --sharegpt-output-len 4 \ --num-prompts 3 \ --output-file /tmp/sglang-simulator-quickstart/benchmark.json ``` The benchmark injects logical arrival metadata into each request and displays the server-side simulator metrics. For timestamped traffic, use the simulator-owned Autobench JSONL format and add `--use-trace-timestamps`. ## Read the results The output directory contains: * `metrics.json`: aggregate latency, throughput, and cache metrics; * `request.jsonl`: per-request timing and cache information; and * `iteration.jsonl`: scheduler batch composition and predicted iteration latency. Use a unique output directory for each run so metrics from separate experiments are not mixed. See the [SGLang Simulator source README](https://github.com/sgl-project/sglang/tree/main/tools/sglang-simulator) for simulator configuration fields, predictor examples, and maintained tests. # Speculative Decoding Source: https://docs.sglang.io/docs/advanced_features/speculative_decoding SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3, MTP, UNO, DFLASH, classic draft-model decoding, and an NGRAM-based variant. Our implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines. ## Summary ### Jump to sections * [EAGLE Decoding](#eagle-decoding) * [EAGLE-2 Decoding](#eagle-2-decoding) * [EAGLE-2 Decoding with torch.compile](#eagle-2-decoding-with-torch-compile) * [EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling](#eagle-2-decoding-via-frequency-ranked-speculative-sampling) * [EAGLE-3 Decoding](#eagle-3-decoding) * [Block verification](#block-verification) * [Multi Token Prediction](#multi-token-prediction) * [UNO decoding](#uno-decoding) * [DFlash Decoding](#dflash-decoding) * [Standalone Speculative Decoding (Small Draft Model)](#standalone-speculative-decoding-small-draft-model) * [Speculative Decoding V2 (Overlap Scheduler)](#speculative-decoding-v2-overlap-scheduler) * [Ngram Speculative Decoding](#ngram-speculative-decoding) * [Full Parameter Reference](#full-parameter-reference) * [OOM Troubleshooting](#oom-troubleshooting) * [References](#references) ### Quick guidance * **Best speed/quality (recommended)**: Use **EAGLE-3** with `--speculative-algorithm EAGLE3`. * **Strong default / broad compatibility**: Use **EAGLE-2** with `--speculative-algorithm EAGLE`. * **Workload acceptance changes over time**: Use [**Adaptive speculative decoding**](./adaptive_speculative_decoding) on top of **EAGLE** with `--speculative-eagle-topk 1`. * **Lower `lm_head` overhead for EAGLE-2**: Enable **FR-Spec** with `--speculative-token-map`. * **Model is MTP-enabled**: Use **MTP via speculative decoding** (often with small `speculative_num_steps/topk/num_draft_tokens`, see the example section). * **You have a UNO adapter for the target model**: Use **UNO** with `--speculative-algorithm UNO` and `--uno-lora-path ...`. * **You have a DFlash draft checkpoint**: Use **DFLASH** with `--speculative-algorithm DFLASH` and `--speculative-draft-model-path ...`. * **You have a smaller draft LLM**: Use **STANDALONE** (`--speculative-algorithm STANDALONE`). * **No extra model available**: Use **NGRAM** (`--speculative-algorithm NGRAM`, CUDA-only). ### Method comparison (mini table)
Method Draft source Separate draft model? How to enable Notes / constraints
EAGLE-2 EAGLE draft model (feature drafting + tree) Typically yes --speculative-algorithm EAGLE + --speculative-draft-model-path ... Tune --speculative-num-steps, --speculative-eagle-topk, --speculative-num-draft-tokens
EAGLE-2 + torch.compile Same as EAGLE-2 Typically yes Add --enable-torch-compile (optionally --torch-compile-max-bs) Benefit varies by hardware/model; benchmark to verify
EAGLE-2 + FR-Spec Same as EAGLE-2 + token subset Typically yes Add --speculative-token-map ... Reduces lm\_head overhead with high-frequency token vocab
EAGLE-3 EAGLE3 draft model Yes --speculative-algorithm EAGLE3 + --speculative-draft-model-path ... Best throughput in the benchmark below
MTP Built-in multi-token heads (model-specific) Often no See Multi Token Prediction section Uses speculative workflow; draft path may be auto-handled for some models
UNO Target model with a trained UNO LoRA (linear chain or tree) No --speculative-algorithm UNO + --uno-lora-path ... CUDA and FA3; currently requires TP=PP=1
DFLASH DFlash draft model (linear block verification) Yes --speculative-algorithm DFLASH + --speculative-draft-model-path ... No DP attention (--attn-dp-size > 1); pp\_size == 1; disables overlap scheduler & mixed chunked prefill
STANDALONE Smaller draft LLM (token-level) Yes --speculative-algorithm STANDALONE + --speculative-draft-model-path ... Does not support DP attention (--attn-dp-size > 1)
NGRAM Ngram cache from previous tokens No --speculative-algorithm NGRAM CUDA-only; no DP attention (--attn-dp-size > 1); disables overlap scheduler & mixed chunked prefill
### Performance Highlights Please see below for the huge improvements on throughput for LLaMA-Instruct 3.1 8B tested on MT bench that can be achieved via EAGLE3 decoding. For further details please see the [EAGLE3 paper](https://arxiv.org/pdf/2503.01840).
Method Throughput (tokens/s)
SGLang (w/o speculative, 1x H100) 158.34 tokens/s
SGLang + EAGLE-2 (1x H100) 244.10 tokens/s
SGLang + EAGLE-3 (1x H100) 373.25 tokens/s
*** ## EAGLE Decoding To enable EAGLE speculative decoding the following parameters are relevant:
Parameter Description Default
--speculative-draft-model-path Draft model path/weights. Typically required for EAGLE/EAGLE3 and STANDALONE. For some MTP-enabled models, this can be omitted. None
--speculative-num-steps Depth of autoregressive drafting. Increases speculation range but risks rejection cascades. Auto (5 for Llama/Grok; 3 for many other models)
--speculative-eagle-topk Branching factor per step. Improves candidate diversity and acceptance rate, but increases memory/compute consumption. Auto (4 for Llama/Grok; 1 for many other models)
--speculative-num-draft-tokens Maximum parallel verification capacity. Allows deeper tree evaluation but increases GPU memory usage. Auto (8 for Llama/Grok; 4 for many other models). If topk=1, it is adjusted to num\_steps + 1.
--speculative-accept-threshold-single Acceptance threshold for single-token verification. Lower values accept more aggressively. 1.0
--speculative-accept-threshold-acc Accumulated acceptance threshold across steps. 1.0
--speculative-attention-mode Attention mode for speculative operations (prefill or decode), affecting both target verification and draft extension. "prefill"
--speculative-draft-attention-backend Override attention backend for the draft model. None (same as target)
--speculative-dsa-topk-backend Select the DSA indexer top-k backend for speculative draft workers independently of --dsa-topk-backend. Options are sgl-kernel, torch, and flashinfer; torch requires SGLANG\_DSA\_FUSE\_TOPK=false. sgl-kernel
--speculative-draft-model-quantization Quantization method for the draft model. Use "unquant" to force no quantization even when the target model is quantized. Same as target model
--speculative-draft-model-revision Specific revision/commit of the draft model to load. None (auto-set to "main" when --speculative-draft-model-path is set and revision is omitted)
--speculative-draft-load-format Load format for the draft model weights. None
These parameters are mostly the same for EAGLE-2 and EAGLE-3. `--speculative-token-map` is ignored for EAGLE-3 models. For `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`: leave all three unset to use auto-tuning, or set all three explicitly when tuning. If you use EAGLE with `--speculative-eagle-topk 1` and your acceptance rate varies across requests, see [Adaptive Speculative Decoding](./adaptive_speculative_decoding). You can find the best combinations of these parameters with [bench\_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). ### EAGLE-2 Decoding You can enable EAGLE-2 Decoding by setting `--speculative-algorithm EAGLE` and choosing an appropriate model. **Launch the server:** ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Llama-2-7b-chat-hf \ --speculative-algorithm EAGLE \ --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \ --speculative-num-steps 3 \ --speculative-eagle-topk 4 \ --speculative-num-draft-tokens 16 \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="meta-llama/Llama-2-7b-chat-hf", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ### EAGLE-2 Decoding with `torch.compile` You can optionally enable `torch.compile` to apply kernel-level optimizations (operator fusion, autotune) to the draft model. The actual speedup depends on your hardware, model architecture, and batch size. In some configurations (e.g., small draft models on H100 where cuBLAS is already optimal and CUDA graphs are enabled), the benefit may be negligible. We recommend benchmarking with and without this flag on your specific setup to verify whether it helps. To enable it, add `--enable-torch-compile` and optionally set `--torch-compile-max-bs`: ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Llama-2-7b-chat-hf \ --speculative-algorithm EAGLE \ --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \ --speculative-num-steps 3 \ --speculative-eagle-topk 4 \ --speculative-num-draft-tokens 16 \ --mem-fraction-static 0.7 \ --enable-torch-compile \ --torch-compile-max-bs 8 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="meta-llama/Llama-2-7b-chat-hf", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ### EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling By employing a truncated high-frequency token vocabulary in the draft model, EAGLE speculative decoding reduces `lm_head` computational overhead while accelerating the pipeline without quality degradation. For more details, check out [the paper](https://arxiv.org/pdf/2502.14856). In our implementation, set `--speculative-token-map` to enable the optimization. You can get the high-frequency tokens in FR-Spec from [this model](https://huggingface.co/thunlp/LLaMA3-Instruct-8B-FR-Spec). Or you can obtain high-frequency tokens by directly downloading these tokens from [this repo](https://github.com/thunlp/FR-Spec/tree/main?tab=readme-ov-file#prepare-fr-spec-vocabulary-subset). Thanks for the contribution from [Weilin Zhao](https://github.com/Achazwl) and [Zhousx](https://github.com/Zhou-sx). ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3-8B-Instruct \ --speculative-algorithm EAGLE \ --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B \ --speculative-num-steps 3 \ --speculative-eagle-topk 4 \ --speculative-num-draft-tokens 16 \ --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --dtype float16 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="meta-llama/Meta-Llama-3-8B-Instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ### EAGLE-3 Decoding You can enable EAGLE-3 decoding by setting `--speculative-algorithm EAGLE3` and choosing an appropriate model. ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B \ --speculative-num-steps 3 \ --speculative-eagle-topk 4 \ --speculative-num-draft-tokens 16 \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --dtype float16 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ## Block verification For stochastic EAGLE, EAGLE3, or NEXTN decoding on NVIDIA CUDA or AMD ROCm, add `--speculative-use-block-verification` with `--speculative-eagle-topk 1`. The draft model must produce a full target-vocabulary proposal distribution, as required by the existing rejection sampler. The same flag applies to the DFlash family (`DFLASH` and `DSPARK`) on CUDA or ROCm when the draft samples its proposals: DSpark drafts, DFlash checkpoints with a candidate selector, and [LiLiCorr](#lilicorr-reranked-dflash-drafts) drafts with `SGLANG_ENABLE_LILICORR_SAMPLING=1`. These paths already publish their proposal distribution, so `--speculative-use-rejection-sampling` is not needed. A DFlash draft that proposes greedy tokens is a point mass, where block and token-wise verification accept the same prefixes; the server warns that the flag has no effect. [Block verification](https://arxiv.org/abs/2403.10444) checks every draft prefix and keeps the longest accepted prefix, including when an earlier prefix was rejected. It uses a corresponding correction distribution to preserve target sampling. Use `--speculative-use-rejection-sampling` alone for token-wise verification, which stops at the first rejected token. Greedy requests keep their existing verification behavior. Block verification adds vocabulary reductions during verification. Benchmark your model and workload to measure the effect on latency and throughput. ## Multi Token Prediction We support [MTP (Multi-Token Prediction)](https://arxiv.org/pdf/2404.19737) in SGLang by using speculative decoding. We use `XiaomiMiMo/MiMo-7B-RL` as an example here (for DeepSeek MTP usage, refer to [DeepSeek-V3.2 cookbook §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2#4-2-3-multi-token-prediction-eagle-speculative-decoding)). ```bash Command theme={null} python3 -m sglang.launch_server \ --model XiaomiMiMo/MiMo-7B-RL \ --host 0.0.0.0 \ --trust-remote-code \ --speculative-algorithm EAGLE \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --log-level warning ``` For DSA-based MTP, draft workers default to the `sgl-kernel` top-k backend. Use `--speculative-dsa-topk-backend` to override the draft independently of `--dsa-topk-backend` for the target model. **Send a request:** ```python Example theme={null} import requests url = "http://localhost:30000/v1/chat/completions" data = { "model": "XiaomiMiMo/MiMo-7B-RL", "messages": [{"role": "user", "content": "What is the capital of France?"}], } response = requests.post(url, json=data) print(response.json()) ``` *** ## UNO decoding UNO reuses the target transformer for both passes of each speculative decode cycle instead of loading a separate draft model. During the draft forward, the first row of each request uses the base weights and the remaining `B - 1` rows use a trained UNO LoRA. Target verification is entirely base-only. UNO provides two sampling modes: * **Linear** constructs and verifies one chain of draft tokens, similar to DFlash's proposal layout. * **Tree** expands multiple candidates at each draft depth and uses SGLang's EAGLE tree-verification path. For UNO modes, the three numbers are `B/K/V`: draft-forward width, candidates kept per expansion, and verification width. Thus, `Linear 8/1/8` uses an 8-token draft width, one candidate per expansion, and an 8-token verification width. UNO's `K` controls proposal-tree breadth and is unrelated to the request sampling parameter `top_k`. The command-line mapping depends on the mode: * In linear mode, `B` and `V` are both `--speculative-num-draft-tokens`; set `--speculative-num-steps 1` and `--speculative-eagle-topk 1`. * In tree mode, `B` is `--speculative-num-steps + 1`, `K` is `--speculative-eagle-topk`, and `V` is `--speculative-num-draft-tokens`. Use an adapter trained for the exact target-model checkpoint. The current integration has been validated with `Qwen/Qwen3-8B`; unsupported LoRA target layers are rejected at startup. Set the adapter's local path or Hugging Face repository before starting the server: ### Linear 8/1/8 ```bash Command theme={null} export UNO_LORA_PATH="s-sahoo/uno-qwen3-8B" sglang serve \ --model-path Qwen/Qwen3-8B \ --speculative-algorithm UNO \ --uno-lora-path "$UNO_LORA_PATH" \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 8 \ --attention-backend fa3 \ --tp 1 \ --host 0.0.0.0 \ --port 30000 ``` ### Tree 8/32/8 The following command selects tree mode because `K` is greater than one: ```bash Command theme={null} export UNO_LORA_PATH="s-sahoo/uno-qwen3-8B" sglang serve \ --model-path Qwen/Qwen3-8B \ --speculative-algorithm UNO \ --uno-lora-path "$UNO_LORA_PATH" \ --speculative-num-steps 7 \ --speculative-eagle-topk 32 \ --speculative-num-draft-tokens 8 \ --attention-backend fa3 \ --tp 1 \ --host 0.0.0.0 \ --port 30000 ``` Send requests through the standard OpenAI-compatible API: ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=[{"role": "user", "content": "Explain speculative decoding."}], max_tokens=128, ) print(response.choices[0].message.content) ``` ### Key requirements and limitations * UNO requires CUDA with FA3 for prefill and decode, tensor and pipeline parallel sizes of 1, and no DP attention or context parallelism. * `--uno-lora-path` loads UNO's fixed internal adapter and cannot be combined with request-selectable Multi-LoRA serving. * UNO manages its own stochastic verification; do not set `--speculative-use-rejection-sampling`. * Grammar decoding, returned logprobs or hidden states, sampling penalties, `min_p`, logit bias, custom logit processors, strict thinking, and deterministic inference are not yet supported. * In tree mode, both speculative acceptance thresholds must remain at 1.0, `V` must be at least `B` and at most 128, and `V * K` must not exceed 2048. SGLang validates the remaining tree-capacity and EAGLE parent-representation constraints at startup. * Mixed chunked prefill is disabled for UNO. * Ordinary overlap scheduling is supported. Tree mode does not yet support PDMux or the separate `--enable-two-batch-overlap` feature. *** ## DFlash Decoding SGLang also supports **DFLASH** speculative decoding using a dedicated draft model checkpoint. Compared with EAGLE-style tree verification, DFLASH verifies a linear draft block and is configured around a block size / draft window. This path is useful when the target model has a matching DFlash draft checkpoint, such as `meta-llama/Llama-3.1-8B-Instruct` with `z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat`. Relevant parameters:
Parameter Description Default
--speculative-draft-model-path Required DFlash draft model path/weights. None
--speculative-num-draft-tokens DFlash verify block size. Inferred from draft config, otherwise 16
--speculative-dflash-block-size Alias of --speculative-num-draft-tokens for DFlash. None
--speculative-draft-window-size Draft KV sliding-window size. Must be >= speculative-num-draft-tokens when set. None
```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --speculative-algorithm DFLASH \ --speculative-draft-model-path z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[ {"role": "user", "content": "Write a quicksort implementation in Python."}, ], temperature=0, max_tokens=128, ) print(response.choices[0].message.content) ``` ### LiLiCorr: reranked DFlash drafts A DFlash drafter is trained on per-position marginals rather than on the joint block distribution, so its drafted tokens are individually plausible yet jointly incoherent. **LiLiCorr** ([arXiv:2608.20530](https://arxiv.org/abs/2608.20530)) keeps the top-`k` candidates per block position, scores the whole lattice jointly with a small two-layer transformer, and commits a path through it greedily instead of taking the per-slot argmax. Verify is unchanged, so outputs remain distributionally identical to the target model's and only acceptance length moves. LiLiCorr is **not a separate `--speculative-algorithm`**: it is a property of the draft checkpoint, the same way the DFlash candidate selector is. Serve it with `--speculative-algorithm DFLASH` and a checkpoint whose config declares `architectures: ["LiLiCorrDraftModel"]` and carries the `lilicorr_*` geometry fields inside `dflash_config`. The head is then built, loaded and folded into the draft CUDA graph automatically. No LiLiCorr drafter is published yet, so the draft path below is a placeholder. ```bash Command theme={null} python3 -m sglang.launch_server \ --model Qwen/Qwen3-8B \ --attention-backend fa3 \ --speculative-algorithm DFLASH \ --speculative-draft-model-path ``` **Use a backend that opts out of the host `seq_lens` mirror.** `AttentionBackend.needs_cpu_seq_lens` defaults to `True`; `fa3`, `triton`, `trtllm_mha`, `trtllm_mla`, `flashmla` and `dsa` set it `False`, while `FlashInferAttnBackend` inherits the default. Under a backend that does not opt out the worker publishes `seq_lens_cpu` every step, which is a device-to-host sync per block. The flag is OR-ed over the target and draft runners, so both have to opt out. This is a property of DFlash decoding rather than of LiLiCorr, but it dominates the throughput of a 16-slot block. Backends also differ in attention numerics, so acceptance measured on different backends does not belong in one table. A checkpoint carrying head weights but declaring `architectures: ["DFlashDraftModel"]` loads as plain DFlash and ignores the head, which reads as a small acceptance loss rather than an error. The reverse, declaring `LiLiCorrDraftModel` without complete head weights, is refused at load. #### Sampling the committed path Above temperature 0 the target samples, but a deterministic draft is paid `p(argmax)` per slot because verify is handed a point mass. `SGLANG_ENABLE_LILICORR_SAMPLING=1` draws each slot from `softmax(scores / T)` over that slot's candidates at the request's own temperature and publishes the proposal it drew from, so rejection sampling is paid the overlap of the two distributions instead. Verify stays exact. Greedy requests sharing the batch take the argmax and report a one-hot proposal. Off by default; with it unset the captured body is `select` itself, so temperature-0 output is unchanged bit for bit. On Qwen3-8B it is worth about 6% acceptance and 6% end-to-end at temperature 1, at under 1% of the per-block wall. **Only enable it on the folded path.** The eager fallback that serves the steps the draft CUDA graph cannot is correct but costs a large fraction of throughput, and because acceptance is identical either way that regression shows up only in tokens/s. `build_lilicorr_draft_sampler` warns with a reason when it declines to fold, and the worker warns once if a decode step lands on the eager head. #### Drafters with grouped convolutions LiLiCorr composes with the grouped dynamic convolutions that wrap each DFlash sublayer, and serving one needs no extra flag: the convolution belongs to the DFlash backbone, so `LiLiCorrDraftModel` inherits the builder and picks it up from `conv_kernel_size` and `conv_group_size` in `dflash_config`. Both keys default to `0`, so a checkpoint whose config lost them builds no convolution modules and the backbone loader then drops every convolution tensor it cannot resolve. That is correct for rotary caches and the worst case here, because the draft would serve as its convolution-free parent at a lower but entirely believable acceptance length. The loader therefore refuses any checkpoint whose convolution tensors and built modules disagree, in either direction. *** ## Standalone Speculative Decoding (Small Draft Model) Besides EAGLE/MTP, SGLang also supports **token-level speculative decoding** using a smaller **draft model**. Enable it with `--speculative-algorithm STANDALONE` and provide a draft model via `--speculative-draft-model-path`. Relevant parameters:
Parameter Description Default
--speculative-draft-model-path Draft model weights (smaller than the target model). None
--speculative-num-steps Draft depth (how many steps the draft model runs autoregressively). 3 (auto default for STANDALONE)
--speculative-eagle-topk Branching factor (token candidates per step). 1 (auto default for STANDALONE)
--speculative-num-draft-tokens Verification capacity. 4 (auto default for STANDALONE)
--speculative-draft-model-quantization Quantization for the draft model. Use "unquant" to disable quantization on the draft even when the target is quantized. Same as target
> **Note:** Standalone speculative decoding currently **does not support** DP attention (`--attn-dp-size` > 1). ```bash Command theme={null} python3 -m sglang.launch_server \ --model Qwen/Qwen2.5-7B-Instruct \ --speculative-algorithm STANDALONE \ --speculative-draft-model-path Qwen/Qwen2.5-1.5B-Instruct \ --speculative-num-steps 4 \ --speculative-eagle-topk 2 \ --speculative-num-draft-tokens 7 \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ## Speculative Decoding V2 (Overlap Scheduler) Speculative decoding runs the V2 speculative workers (e.g. `StandaloneWorkerV2`, `EAGLEWorkerV2`) with the overlap scheduler enabled by default. Pass `--disable-overlap-schedule` to fall back to the synchronous (non-overlap) path. Notes: * The overlap scheduler currently only supports `--speculative-eagle-topk 1`; **set `--speculative-eagle-topk 1` explicitly**. * If you explicitly set `--speculative-eagle-topk > 1`, the server will error. * If you omit `--speculative-eagle-topk`, auto-tuning may pick `topk > 1` for some models (e.g. Llama). This is incompatible with the overlap scheduler and may not always trigger an immediate config error, so set `--speculative-eagle-topk 1` explicitly. ```bash Command theme={null} python3 -m sglang.launch_server \ --model Qwen/Qwen2.5-7B-Instruct \ --speculative-algorithm STANDALONE \ --speculative-draft-model-path Qwen/Qwen2.5-1.5B-Instruct \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ## Ngram Speculative Decoding SGLang also supports **ngram-based speculative decoding** (no separate draft model). It retrieves draft tokens from an ngram cache built from previously generated tokens, and then verifies them with the target model. Enable it with: * `--speculative-algorithm NGRAM` ### Ngram-specific parameters
Parameter Description Default
--speculative-num-draft-tokens Number of draft tokens verified per step. If omitted, defaults to min(--speculative-ngram-max-trie-depth, 12). 12 (with default ngram settings)
--speculative-ngram-min-bfs-breadth Minimum BFS breadth. 1
--speculative-ngram-max-bfs-breadth Maximum BFS breadth. 10
--speculative-ngram-match-type Ngram tree-building mode: "BFS" for recency-based expansion or "PROB" for frequency-based expansion. "BFS"
--speculative-ngram-max-trie-depth Maximum suffix length stored and matched by the ngram trie. 18
--speculative-ngram-capacity Cache capacity (number of entries). 10,000,000
Notes: * Ngram speculative decoding **only supports CUDA**. * It currently **does not support** DP attention (`--attn-dp-size` > 1). * It disables the overlap scheduler and mixed chunked prefill. * If `--speculative-ngram-max-bfs-breadth > 1` (thus `speculative_eagle_topk > 1`) and `page_size > 1`, use `--attention-backend flashinfer`; otherwise the server will error. * Optional: set `SGLANG_NGRAM_FORCE_GREEDY_VERIFY=True` to force greedy verification. ```bash Command theme={null} python3 -m sglang.launch_server \ --model Qwen/Qwen2.5-7B-Instruct \ --speculative-algorithm NGRAM \ --speculative-num-draft-tokens 16 \ --speculative-ngram-max-bfs-breadth 10 \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ## Full Parameter Reference Below is a comprehensive list of all speculative decoding parameters available in SGLang: ### Core parameters
Parameter Type Default Description
--speculative-algorithm str None Algorithm to use: UNO, DFLASH, EAGLE, EAGLE3, STANDALONE, NGRAM, NEXTN (alias of EAGLE)
--speculative-draft-model-path str None Path to the draft model weights
--uno-lora-path str None UNO-only path or Hugging Face repository for the draft LoRA checkpoint
--speculative-draft-model-revision str None Specific revision/commit of the draft model ("main" is auto-used when draft path is set and revision is omitted)
--speculative-draft-load-format str None Load format for draft model weights
--speculative-num-steps int None (auto-chosen when omitted) Autoregressive drafting depth. Some algorithms auto-tune it; tree UNO requires it and uses B = speculative\_num\_steps + 1.
--speculative-eagle-topk int None (auto-chosen when omitted) Branching factor per drafting step. Some algorithms auto-tune it; UNO resolves an omitted value to 1, and values greater than one select UNO tree mode.
--speculative-num-draft-tokens int None (auto-chosen when omitted) Maximum number of draft tokens for verification. Some algorithms auto-tune it; UNO requires it and uses it as linear width or tree verification width V.
--speculative-dflash-block-size int None DFlash-only alias of --speculative-num-draft-tokens
--speculative-draft-window-size int None DFlash-only draft KV sliding-window size
--speculative-accept-threshold-single float 1.0 Single-token acceptance threshold
--speculative-accept-threshold-acc float 1.0 Accumulated acceptance threshold
--speculative-token-map str None Path to FR-Spec high-frequency token map
--speculative-attention-mode str "prefill" Attention mode for speculative operations ("prefill" or "decode")
--speculative-draft-attention-backend str None Override attention backend for the draft model
--speculative-dsa-topk-backend str sgl-kernel DSA indexer top-k backend for speculative draft workers, independent of --dsa-topk-backend (sgl-kernel, torch, or flashinfer)
--speculative-moe-runner-backend str None MoE runner backend for the draft model
--speculative-moe-a2a-backend str None MoE all-to-all backend for the draft model
--speculative-draft-model-quantization str Same as target Quantization for the draft model ("unquant" to disable)
### Ngram-specific parameters
Parameter Type Default Description
--speculative-ngram-min-bfs-breadth int 1 Minimum BFS breadth
--speculative-ngram-max-bfs-breadth int 10 Maximum BFS breadth
--speculative-ngram-match-type str "BFS" Ngram tree-building mode: "BFS" for recency-based expansion or "PROB" for frequency-based expansion
--speculative-ngram-max-trie-depth int 18 Maximum suffix length stored and matched by the ngram trie
--speculative-ngram-capacity int 10,000,000 Cache capacity
### Environment variables
Variable Default Description
SGLANG\_NGRAM\_FORCE\_GREEDY\_VERIFY False Force greedy verification for ngram decoding
### Other related flags
Parameter Description
--enable-multi-layer-eagle Enable multi-layer EAGLE (auto-enabled for MiMoV2 and Step3p5 models)
--enable-torch-compile Enable torch.compile for kernel-level optimizations
--torch-compile-max-bs Maximum batch size for torch.compile
*** ## OOM Troubleshooting > \[!WARNING] > **Out of Memory (OOM)?** Speculative decoding may increase GPU memory usage because the draft tree, CUDA graphs, and verification-related buffers consume additional VRAM. If you encounter OOM errors, try the following adjustments. ### Step 1: Lower static memory fraction (most effective) ```bash Command theme={null} --mem-fraction-static 0.5 # when omitted, this value is auto-computed ``` * `--mem-fraction-static` controls the memory budget for model weights + KV cache pool. * Lowering it directly increases dynamic headroom for activations and CUDA graph buffers. * If omitted, SGLang auto-estimates this value from other settings, and those auto settings can still be too aggressive for some workloads. ### Step 2: Reduce CUDA graph batch size ```bash Command theme={null} # Fewer CUDA graph captures = less memory reserved --cuda-graph-max-bs-decode 4 # or even 2 for tight memory situations ``` * If omitted, `--cuda-graph-max-bs-decode` is auto-selected based on GPU memory and TP size, and can be much larger on high-memory GPUs. ### Step 3: Reduce draft tree size These three parameters directly control how much memory the draft tree consumes: ```bash Command theme={null} # Before (aggressive, high memory) --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 # After (conservative, lower memory) --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 ``` ### Step 4: Limit concurrent requests ```bash Command theme={null} # Fewer concurrent requests lowers in-flight load and can reduce OOM risk --max-running-requests 4 ``` ### Quick OOM recovery recipe If you're hitting OOM and just want something that works, start with this minimal configuration and scale up: ```bash Command theme={null} python3 -m sglang.launch_server \ --model \ --speculative-algorithm EAGLE \ --speculative-draft-model-path \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --cuda-graph-max-bs-decode 2 \ --mem-fraction-static 0.5 \ --max-running-requests 4 \ --log-level warning ``` Then gradually increase `--speculative-num-draft-tokens`, `--speculative-eagle-topk`, and `--cuda-graph-max-bs-decode`. Increase `--mem-fraction-static` last, only after the run is stable. *** ## References EAGLE process is as follows: * Within EAGLE the draft model predicts the next feature vector, i.e. the last hidden state of the original LLM, using the feature sequence $(f_1, ..., f_k)$ and the token sequence $(t_2, ..., t_{k+1})$. * The next token is then sampled from $p_{k+2}=\text{LMHead}(f_{k+1})$. Afterwards, the two sequences are extended in a tree style—branching out multiple potential continuations, with the branching factor per step controlled by the `speculative_eagle_topk` parameter—to ensure a more coherent connection of context, and are given as input again. * In SGLang's EAGLE-2 implementation, the draft tree is expanded for the configured steps and then reranked to select the top `speculative_num_draft_tokens` final nodes as draft tokens. * EAGLE-3 removes the feature prediction objective, incorporates low and mid-layer features, and is trained in an on-policy manner. This enhances drafting accuracy by operating on features instead of tokens for more regular inputs and by additionally passing tokens from the next timestep to reduce sampling randomness. For more details, see the [EAGLE-2](https://arxiv.org/abs/2406.16858) and [EAGLE-3](https://arxiv.org/abs/2503.01840) papers. For guidance on how to train your own EAGLE model please see the [EAGLE repo](https://github.com/SafeAILab/EAGLE/tree/main?tab=readme-ov-file#train). For EAGLE-3 training specifically, check out [SpecForge](https://github.com/sgl-project/SpecForge), the SGLang team's training framework designed for EAGLE-3 speculative decoding models with seamless porting to SGLang serving. See the [SpecForge documentation](https://docs.sglang.ai/SpecForge/) and [blog post](https://lmsys.org/blog/2025-07-25-spec-forge) for details. # Structured Outputs Source: https://docs.sglang.io/docs/advanced_features/structured_outputs You can specify a JSON schema, [regular expression](https://en.wikipedia.org/wiki/Regular_expression) or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request. SGLang supports three grammar backends: * [XGrammar](https://github.com/mlc-ai/xgrammar)(default): Supports JSON schema, regular expression, and EBNF constraints. * [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints. * [Llguidance](https://github.com/guidance-ai/llguidance): Supports JSON schema, regular expression, and EBNF constraints. We suggest using XGrammar for its better performance and utility. XGrammar currently uses the [GGML BNF format](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md). For more details, see [XGrammar technical overview](https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar). To use Outlines, simply add `--grammar-backend outlines` when launching the server. To use llguidance, add `--grammar-backend llguidance` when launching the server. If no backend is specified, XGrammar will be used as the default. For better output quality, **It's advisable to explicitly include instructions in the prompt to guide the model to generate the desired format.** For example, you can specify, 'Please generate the output in the following JSON format: ...'. ## OpenAI Compatible API ```python Example theme={null} import openai import os from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process os.environ["TOKENIZERS_PARALLELISM"] = "false" server_process, port = launch_server_cmd( "python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 0.0.0.0 --log-level warning" ) wait_for_server(f"http://localhost:{port}") client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") ``` ### JSON you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response. **Using Pydantic** ```python Example theme={null} from pydantic import BaseModel, Field # Define the schema using Pydantic class CapitalInfo(BaseModel): name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city") population: int = Field(..., description="Population of the capital city") response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[ { "role": "user", "content": "Please generate the information of the capital of France in the JSON format.", }, ], temperature=0, max_tokens=128, response_format={ "type": "json_schema", "json_schema": { "name": "foo", # convert the pydantic model to json schema "schema": CapitalInfo.model_json_schema(), }, }, ) response_content = response.choices[0].message.content # validate the JSON response by the pydantic model capital_info = CapitalInfo.model_validate_json(response_content) print_highlight(f"Validated response: {capital_info.model_dump_json()}") ``` **JSON Schema Directly** ```python Example theme={null} import json json_schema = json.dumps( { "type": "object", "properties": { "name": {"type": "string", "pattern": "^[\\w]+$"}, "population": {"type": "integer"}, }, "required": ["name", "population"], } ) response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[ { "role": "user", "content": "Give me the information of the capital of France in the JSON format.", }, ], temperature=0, max_tokens=128, response_format={ "type": "json_schema", "json_schema": {"name": "foo", "schema": json.loads(json_schema)}, }, ) print_highlight(response.choices[0].message.content) ``` ### EBNF ```python Example theme={null} ebnf_grammar = """ root ::= city | description city ::= "London" | "Paris" | "Berlin" | "Rome" description ::= city " is " status status ::= "the capital of " country country ::= "England" | "France" | "Germany" | "Italy" """ response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[ {"role": "system", "content": "You are a helpful geography bot."}, { "role": "user", "content": "Give me the information of the capital of France.", }, ], temperature=0, max_tokens=32, extra_body={"ebnf": ebnf_grammar}, ) print_highlight(response.choices[0].message.content) ``` ### Regular expression ```python Example theme={null} response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[ {"role": "user", "content": "What is the capital of France?"}, ], temperature=0, max_tokens=128, extra_body={"regex": "(Paris|London)"}, ) print_highlight(response.choices[0].message.content) ``` ### Structural Tag ```python Example theme={null} tool_get_current_weather = { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": "The city to find the weather for, e.g. 'San Francisco'", }, "state": { "type": "string", "description": "the two-letter abbreviation for the state that the city is" " in, e.g. 'CA' which would mean 'California'", }, "unit": { "type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"], }, }, "required": ["city", "state", "unit"], }, }, } tool_get_current_date = { "type": "function", "function": { "name": "get_current_date", "description": "Get the current date and time for a given timezone", "parameters": { "type": "object", "properties": { "timezone": { "type": "string", "description": "The timezone to fetch the current date and time for, e.g. 'America/New_York'", } }, "required": ["timezone"], }, }, } schema_get_current_weather = tool_get_current_weather["function"]["parameters"] schema_get_current_date = tool_get_current_date["function"]["parameters"] def get_messages(): return [ { "role": "system", "content": f""" # Tool Instructions - Always execute python code in messages that you share. - When looking for real time information use relevant functions if available else fallback to brave_search You have access to the following functions: Use the function 'get_current_weather' to: Get the current weather in a given location {tool_get_current_weather["function"]} Use the function 'get_current_date' to: Get the current date and time for a given timezone {tool_get_current_date["function"]} If a you choose to call a function ONLY reply in the following format: <{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}} where start_tag => ` a JSON dict with the function argument name as key and function argument value as value. end_tag => `` Here is an example, {{"example_name": "example_value"}} Reminder: - Function calls MUST follow the specified format - Required parameters MUST be specified - Only call one function at a time - Put the entire function call reply on one line - Always add your sources when using search results to answer the user query You are a helpful assistant.""", }, { "role": "user", "content": "You are in New York. Please get the current date and time, and the weather.", }, ] messages = get_messages() response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=messages, response_format={ "type": "structural_tag", "structures": [ { "begin": "", "schema": schema_get_current_weather, "end": "", }, { "begin": "", "schema": schema_get_current_date, "end": "", }, ], "triggers": ["", "content": { "type": "json_schema", "json_schema": schema_get_current_weather, }, "end": "", }, { "begin": "", "content": { "type": "json_schema", "json_schema": schema_get_current_date, }, "end": "", }, ], "at_least_one": False, "stop_after_first": False, }, }, ) print_highlight(response.choices[0].message.content) ``` ## Native API and SGLang Runtime (SRT) ### JSON **Using Pydantic** ```python Example theme={null} import requests import json from pydantic import BaseModel, Field from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct") # Define the schema using Pydantic class CapitalInfo(BaseModel): name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city") population: int = Field(..., description="Population of the capital city") # Make API request messages = [ { "role": "user", "content": "Here is the information of the capital of France in the JSON format.\n", } ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) response = requests.post( f"http://localhost:{port}/generate", json={ "text": text, "sampling_params": { "temperature": 0, "max_new_tokens": 64, "json_schema": json.dumps(CapitalInfo.model_json_schema()), }, }, ) print_highlight(response.json()) response_data = json.loads(response.json()["text"]) # validate the response by the pydantic model capital_info = CapitalInfo.model_validate(response_data) print_highlight(f"Validated response: {capital_info.model_dump_json()}") ``` **JSON Schema Directly** ```python Example theme={null} json_schema = json.dumps( { "type": "object", "properties": { "name": {"type": "string", "pattern": "^[\\w]+$"}, "population": {"type": "integer"}, }, "required": ["name", "population"], } ) # JSON response = requests.post( f"http://localhost:{port}/generate", json={ "text": text, "sampling_params": { "temperature": 0, "max_new_tokens": 64, "json_schema": json_schema, }, }, ) print_highlight(response.json()) ``` ### EBNF ```python Example theme={null} messages = [ { "role": "user", "content": "Give me the information of the capital of France.", } ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) response = requests.post( f"http://localhost:{port}/generate", json={ "text": text, "sampling_params": { "max_new_tokens": 128, "temperature": 0, "n": 3, "ebnf": ( "root ::= city | description\n" 'city ::= "London" | "Paris" | "Berlin" | "Rome"\n' 'description ::= city " is " status\n' 'status ::= "the capital of " country\n' 'country ::= "England" | "France" | "Germany" | "Italy"' ), }, "stream": False, "return_logprob": False, }, ) print_highlight(response.json()) ``` ### Regular expression ```python Example theme={null} messages = [ { "role": "user", "content": "Paris is the capital of", } ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) response = requests.post( f"http://localhost:{port}/generate", json={ "text": text, "sampling_params": { "temperature": 0, "max_new_tokens": 64, "regex": "(France|England)", }, }, ) print_highlight(response.json()) ``` ### Structural Tag ```python Example theme={null} from transformers import AutoTokenizer # generate an answer tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct") text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) payload = { "text": text, "sampling_params": { "structural_tag": json.dumps( { "type": "structural_tag", "structures": [ { "begin": "", "schema": schema_get_current_weather, "end": "", }, { "begin": "", "schema": schema_get_current_date, "end": "", }, ], "triggers": ["", "content": { "type": "json_schema", "json_schema": schema_get_current_weather, }, "end": "", }, { "begin": "", "content": { "type": "json_schema", "json_schema": schema_get_current_date, }, "end": "", }, ], "at_least_one": False, "stop_after_first": False, }, } ) }, } # Send POST request to the API endpoint response = requests.post(f"http://localhost:{port}/generate", json=payload) print_highlight(response.json()) ``` ```python Example theme={null} terminate_process(server_process) ``` ## Offline Engine API ```python Example theme={null} import sglang as sgl llm = sgl.Engine( model_path="meta-llama/Meta-Llama-3.1-8B-Instruct", grammar_backend="xgrammar" ) ``` ### JSON **Using Pydantic** ```python Example theme={null} import json from pydantic import BaseModel, Field prompts = [ "Give me the information of the capital of China in the JSON format.", "Give me the information of the capital of France in the JSON format.", "Give me the information of the capital of Ireland in the JSON format.", ] # Define the schema using Pydantic class CapitalInfo(BaseModel): name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city") population: int = Field(..., description="Population of the capital city") sampling_params = { "temperature": 0.1, "top_p": 0.95, "json_schema": json.dumps(CapitalInfo.model_json_schema()), } outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print_highlight("===============================") print_highlight(f"Prompt: {prompt}") # validate the output by the pydantic model capital_info = CapitalInfo.model_validate_json(output["text"]) print_highlight(f"Validated output: {capital_info.model_dump_json()}") ``` **JSON Schema Directly** ```python Example theme={null} prompts = [ "Give me the information of the capital of China in the JSON format.", "Give me the information of the capital of France in the JSON format.", "Give me the information of the capital of Ireland in the JSON format.", ] json_schema = json.dumps( { "type": "object", "properties": { "name": {"type": "string", "pattern": "^[\\w]+$"}, "population": {"type": "integer"}, }, "required": ["name", "population"], } ) sampling_params = {"temperature": 0.1, "top_p": 0.95, "json_schema": json_schema} outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print_highlight("===============================") print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}") ``` ### EBNF ```python Example theme={null} prompts = [ "Give me the information of the capital of France.", "Give me the information of the capital of Germany.", "Give me the information of the capital of Italy.", ] sampling_params = { "temperature": 0.8, "top_p": 0.95, "ebnf": ( "root ::= city | description\n" 'city ::= "London" | "Paris" | "Berlin" | "Rome"\n' 'description ::= city " is " status\n' 'status ::= "the capital of " country\n' 'country ::= "England" | "France" | "Germany" | "Italy"' ), } outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print_highlight("===============================") print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}") ``` ### Regular expression ```python Example theme={null} prompts = [ "Please provide information about London as a major global city:", "Please provide information about Paris as a major global city:", ] sampling_params = {"temperature": 0.8, "top_p": 0.95, "regex": "(France|England)"} outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print_highlight("===============================") print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}") ``` ### Structural Tag ```python Example theme={null} text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) prompts = [text] sampling_params = { "temperature": 0.8, "top_p": 0.95, "structural_tag": json.dumps( { "type": "structural_tag", "structures": [ { "begin": "", "schema": schema_get_current_weather, "end": "", }, { "begin": "", "schema": schema_get_current_date, "end": "", }, ], "triggers": ["", "content": { "type": "json_schema", "json_schema": schema_get_current_weather, }, "end": "", }, { "begin": "", "content": { "type": "json_schema", "json_schema": schema_get_current_date, }, "end": "", }, ], "at_least_one": False, "stop_after_first": False, }, } ), } # Send POST request to the API endpoint outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print_highlight("===============================") print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}") ``` ```python Example theme={null} llm.shutdown() ``` # Structured Outputs For Reasoning Models Source: https://docs.sglang.io/docs/advanced_features/structured_outputs_for_reasoning_models When working with reasoning models that use special tokens like `...` to denote reasoning sections, you might want to allow free-form text within these sections while still enforcing grammar constraints on the rest of the output. SGLang provides a feature to disable grammar restrictions within reasoning sections. This is particularly useful for models that need to perform complex reasoning steps before providing a structured output. To enable this feature, use the `--reasoning-parser` flag which decide the think\_end\_token, such as ``, when launching the server. You can also specify the reasoning parser using the `--reasoning-parser` flag. ## Supported Models Currently, SGLang supports the following reasoning models: * [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d): The reasoning content is wrapped with `` and `` tags. * [QwQ](https://huggingface.co/Qwen/QwQ-32B): The reasoning content is wrapped with `` and `` tags. ## Usage ## OpenAI Compatible API Specify the `--grammar-backend`, `--reasoning-parser` option. ```python Example theme={null} import openai import os from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process os.environ["TOKENIZERS_PARALLELISM"] = "false" server_process, port = launch_server_cmd( "python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning" ) wait_for_server(f"http://localhost:{port}") client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") ``` ### JSON you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response. **Using Pydantic** ```python Example theme={null} from pydantic import BaseModel, Field # Define the schema using Pydantic class CapitalInfo(BaseModel): name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city") population: int = Field(..., description="Population of the capital city") response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", messages=[ { "role": "assistant", "content": "Give me the information and population of the capital of France in the JSON format.", }, ], temperature=0, max_tokens=2048, response_format={ "type": "json_schema", "json_schema": { "name": "foo", # convert the pydantic model to json schema "schema": CapitalInfo.model_json_schema(), }, }, ) print_highlight( f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}" ) ``` **JSON Schema Directly** ```python Example theme={null} import json json_schema = json.dumps( { "type": "object", "properties": { "name": {"type": "string", "pattern": "^[\\w]+$"}, "population": {"type": "integer"}, }, "required": ["name", "population"], } ) response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", messages=[ { "role": "assistant", "content": "Give me the information and population of the capital of France in the JSON format.", }, ], temperature=0, max_tokens=2048, response_format={ "type": "json_schema", "json_schema": {"name": "foo", "schema": json.loads(json_schema)}, }, ) print_highlight( f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}" ) ``` ### EBNF ```python Example theme={null} ebnf_grammar = """ root ::= city | description city ::= "London" | "Paris" | "Berlin" | "Rome" description ::= city " is " status status ::= "the capital of " country country ::= "England" | "France" | "Germany" | "Italy" """ response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", messages=[ {"role": "system", "content": "You are a helpful geography bot."}, { "role": "assistant", "content": "Give me the information and population of the capital of France in the JSON format.", }, ], temperature=0, max_tokens=2048, extra_body={"ebnf": ebnf_grammar}, ) print_highlight( f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}" ) ``` ### Regular expression ```python Example theme={null} response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", messages=[ {"role": "assistant", "content": "What is the capital of France?"}, ], temperature=0, max_tokens=2048, extra_body={"regex": "(Paris|London)"}, ) print_highlight( f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}" ) ``` ### Structural Tag ```python Example theme={null} tool_get_current_weather = { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": "The city to find the weather for, e.g. 'San Francisco'", }, "state": { "type": "string", "description": "the two-letter abbreviation for the state that the city is" " in, e.g. 'CA' which would mean 'California'", }, "unit": { "type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"], }, }, "required": ["city", "state", "unit"], }, }, } tool_get_current_date = { "type": "function", "function": { "name": "get_current_date", "description": "Get the current date and time for a given timezone", "parameters": { "type": "object", "properties": { "timezone": { "type": "string", "description": "The timezone to fetch the current date and time for, e.g. 'America/New_York'", } }, "required": ["timezone"], }, }, } schema_get_current_weather = tool_get_current_weather["function"]["parameters"] schema_get_current_date = tool_get_current_date["function"]["parameters"] def get_messages(): return [ { "role": "system", "content": f""" # Tool Instructions - Always execute python code in messages that you share. - When looking for real time information use relevant functions if available else fallback to brave_search You have access to the following functions: Use the function 'get_current_weather' to: Get the current weather in a given location {tool_get_current_weather["function"]} Use the function 'get_current_date' to: Get the current date and time for a given timezone {tool_get_current_date["function"]} If a you choose to call a function ONLY reply in the following format: <{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}} where start_tag => ` a JSON dict with the function argument name as key and function argument value as value. end_tag => `` Here is an example, {{"example_name": "example_value"}} Reminder: - Function calls MUST follow the specified format - Required parameters MUST be specified - Only call one function at a time - Put the entire function call reply on one line - Always add your sources when using search results to answer the user query You are a helpful assistant.""", }, { "role": "assistant", "content": "You are in New York. Please get the current date and time, and the weather.", }, ] messages = get_messages() response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", messages=messages, response_format={ "type": "structural_tag", "max_new_tokens": 2048, "structures": [ { "begin": "", "schema": schema_get_current_weather, "end": "", }, { "begin": "", "schema": schema_get_current_date, "end": "", }, ], "triggers": [" Note: For native API, as a work-around, you need to set `require_reasoning` argument to `True` to ensure the model will think before generating the structured output. It's not required for chat-completion API. ### JSON **Using Pydantic** ```python Example theme={null} import requests from pydantic import BaseModel, Field from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B") # Define the schema using Pydantic class CapitalInfo(BaseModel): name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city") population: int = Field(..., description="Population of the capital city") messages = [ { "role": "assistant", "content": "Give me the information and population of the capital of France in the JSON format.", }, ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) # Make API request response = requests.post( f"http://localhost:{port}/generate", json={ "text": text, "require_reasoning": True, "sampling_params": { "temperature": 0, "max_new_tokens": 2048, "json_schema": json.dumps(CapitalInfo.model_json_schema()), }, }, ) print(response.json()) reasoing_content = response.json()["text"].split("")[0] content = response.json()["text"].split("")[1] print_highlight(f"reasoing_content: {reasoing_content}\n\ncontent: {content}") ``` **JSON Schema Directly** ```python Example theme={null} json_schema = json.dumps( { "type": "object", "properties": { "name": {"type": "string", "pattern": "^[\\w]+$"}, "population": {"type": "integer"}, }, "required": ["name", "population"], } ) # JSON text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) response = requests.post( f"http://localhost:{port}/generate", json={ "text": text, "require_reasoning": True, "sampling_params": { "temperature": 0, "max_new_tokens": 2048, "json_schema": json_schema, }, }, ) print_highlight(response.json()) ``` ### EBNF ```python Example theme={null} response = requests.post( f"http://localhost:{port}/generate", json={ "text": "Give me the information of the capital of France.", "require_reasoning": True, "sampling_params": { "max_new_tokens": 2048, "temperature": 0, "n": 3, "ebnf": ( "root ::= city | description\n" 'city ::= "London" | "Paris" | "Berlin" | "Rome"\n' 'description ::= city " is " status\n' 'status ::= "the capital of " country\n' 'country ::= "England" | "France" | "Germany" | "Italy"' ), }, "stream": False, "return_logprob": False, }, ) print(response.json()) ``` ### Regular expression ```python Example theme={null} response = requests.post( f"http://localhost:{port}/generate", json={ "text": "Paris is the capital of", "require_reasoning": True, "sampling_params": { "temperature": 0, "max_new_tokens": 2048, "regex": "(France|England)", }, }, ) print(response.json()) ``` ### Structural Tag ```python Example theme={null} text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) payload = { "text": text, "require_reasoning": True, "sampling_params": { "max_new_tokens": 2048, "structural_tag": json.dumps( { "type": "structural_tag", "structures": [ { "begin": "", "schema": schema_get_current_weather, "end": "", }, { "begin": "", "schema": schema_get_current_date, "end": "", }, ], "triggers": ["", "schema": schema_get_current_weather, "end": "", }, { "begin": "", "schema": schema_get_current_date, "end": "", }, ], "triggers": ["` token sequence, so the cap is exact. The model then produces the final answer. * With strict thinking enabled and no budget set, behavior is unchanged except that model-specific excluded tokens (for Qwen3, ``, ``, `<|im_end|>`, `<|endoftext|>`) are blocked during the thinking phase. * To apply a server-wide cap to every request, set the environment variable `SGLANG_MAX_THINK_TOKENS` (default `-1`, no cap). ### Native `/generate` API Set `max_thinking_tokens` per request, and always pair it with `require_reasoning: true`: ```python /generate with a thinking budget theme={null} import requests response = requests.post( f"http://localhost:{port}/generate", json={ "text": prompt, "require_reasoning": True, "max_thinking_tokens": 512, "sampling_params": { "temperature": 0, "max_new_tokens": 2048, }, }, ) ``` Without `require_reasoning: true`, the grammar object starts in the generation state and `max_thinking_tokens` is silently ignored. Sending `max_thinking_tokens` to a server launched without `--enable-strict-thinking` returns a 400 error. ### OpenAI chat completions API There is no `max_thinking_tokens` field on chat completions. Pass the budget through `custom_params` instead; the runtime picks up the `thinking_budget` key the same way. The chat path sets `require_reasoning` for you based on the thinking mode: ```python Chat completions with a thinking budget theme={null} response = client.chat.completions.create( model="Qwen/Qwen3-32B", messages=[{"role": "user", "content": "What is 1+3?"}], extra_body={"custom_params": {"thinking_budget": 512}}, ) print(response.choices[0].message.reasoning_content) print(response.choices[0].message.content) ``` ## Hard budget with a custom logit processor When the grammar backend cannot be used, a custom logit processor can enforce the budget instead. Launch the server with `--enable-custom-logit-processor`: ```bash Launch with custom logit processors theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3-32B \ --host 0.0.0.0 \ --reasoning-parser qwen3 \ --enable-custom-logit-processor ``` Pass a serialized processor class and the budget on each request. This works on both `/generate` and chat completions: ```python Logit processor with a thinking budget theme={null} from sglang.srt.sampling.custom_logit_processor import ( Qwen3ThinkingBudgetLogitProcessor, ) response = client.chat.completions.create( model="Qwen/Qwen3-32B", messages=[{"role": "user", "content": "What is 1+3?"}], extra_body={ "custom_logit_processor": Qwen3ThinkingBudgetLogitProcessor.to_str(), "custom_params": {"thinking_budget": 512}, }, ) ``` The processor counts the tokens generated after the `` start token. Once the budget is reached, it first forces a newline token, then forces ``. The model may emit its own `` right after the forced one; this is harmless. ### Built-in processors SGLang ships processors in `python/sglang/srt/sampling/custom_logit_processor.py`, each with hardcoded thinking start / end / newline token ids: | Processor | Model family | | - | - | | `Qwen3ThinkingBudgetLogitProcessor` | Qwen3 | | `Glm4MoeThinkingBudgetLogitProcessor` | GLM-4.5 / GLM-4.6 | | `DeepSeekR1ThinkingBudgetLogitProcessor` | DeepSeek-R1 | | `InklingThinkingBudgetLogitProcessor` | Inkling | These only work for models whose tokenizer produces those exact ids. For any other model, subclass `ThinkingBudgetLogitProcessor` with the model's own ids: ```python Custom thinking-budget processor theme={null} from sglang.srt.sampling.custom_logit_processor import ( ThinkingBudgetLogitProcessor, ) # Find the ids with your model's tokenizer, e.g.: # tokenizer.encode("", add_special_tokens=False) class MyModelThinkingBudgetLogitProcessor(ThinkingBudgetLogitProcessor): THINKING_START_TOKEN_ID = 12345 THINKING_END_TOKEN_ID = 12346 NEW_LINE_TOKEN_ID = 198 ``` ## Verify the budget applies Budget knobs can be silently ignored, so confirm the cap with a deterministic comparison: 1. Run a prompt with `temperature=0` and no budget. Record the thinking length. 2. Run the same prompt with the budget set, again at `temperature=0`. A working cap ends the thinking exactly at the budget, usually mid-sentence. An ignored knob reproduces the baseline output token for token. Common silent-failure modes: * Missing `require_reasoning: true` on the native `/generate` API. * `chat_template_kwargs.thinking_budget` on a model whose chat template does not define the variable. * A built-in logit processor whose hardcoded token ids do not match the model's tokenizer. # Tool Parser Source: https://docs.sglang.io/docs/advanced_features/tool_parser This guide demonstrates how to use SGLang’s [Function calling](https://platform.openai.com/docs/guides/function-calling) functionality. ## Currently supported parsers:
Parser Supported Models Notes
`apertus2509` Apertus 2509 (e.g., `swiss-ai/Apertus-{8,70}B-Instruct-2509`) Tool calls are emitted as a JSON list of single-key objects: `<|tools_prefix|>[{"tool": {...}}]<|tools_suffix|>`.
`deepseekv3` DeepSeek-v3 (e.g., `deepseek-ai/DeepSeek-V3-0324`) Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja` to launch command.
`deepseekv31` DeepSeek-V3.1 and DeepSeek-V3.2-Exp (e.g. `deepseek-ai/DeepSeek-V3.1`, `deepseek-ai/DeepSeek-V3.2-Exp`) Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja` (Or ..deepseekv32.jinja for DeepSeek-V3.2) to launch command.
`deepseekv32` DeepSeek-V3.2 (`deepseek-ai/DeepSeek-V3.2`)
`glm` GLM series (e.g. `zai-org/GLM-4.6`)
`gpt-oss` GPT-OSS (e.g., `openai/gpt-oss-120b`, `openai/gpt-oss-20b`, `lmsys/gpt-oss-120b-bf16`, `lmsys/gpt-oss-20b-bf16`) The gpt-oss tool parser filters out analysis channel events and only preserves normal text. This can cause the content to be empty when explanations are in the analysis channel. To work around this, complete the tool round by returning tool results as `role="tool"` messages, which enables the model to generate the final content.
`kimi_k2` `moonshotai/Kimi-K2-Instruct`
`llama3` Llama 3.1 / 3.2 / 3.3 (e.g. `meta-llama/Llama-3.1-8B-Instruct`, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.3-70B-Instruct`)
`llama4` Llama 4 (e.g. `meta-llama/Llama-4-Scout-17B-16E-Instruct`)
`mistral` Mistral (e.g. `mistralai/Mistral-7B-Instruct-v0.3`, `mistralai/Mistral-Nemo-Instruct-2407`, `mistralai/Mistral-7B-v0.3`)
`pythonic` Llama-3.2 / Llama-3.3 / Llama-4 Model outputs function calls as Python code. Requires `--tool-call-parser pythonic` and is recommended to use with a specific chat template.
`qwen` Qwen series (e.g. `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-VL-30B-A3B-Thinking`) except Qwen3-Coder
`qwen3_coder` Qwen3-Coder (e.g. `Qwen/Qwen3-Coder-30B-A3B-Instruct`)
`step3` Step-3
## OpenAI Compatible API ### Launching the Server ```python Example theme={null} import json from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process from openai import OpenAI server_process, port = launch_server_cmd( "python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0 --log-level warning" # qwen25 ) wait_for_server(f"http://localhost:{port}") ``` Note that `--tool-call-parser` defines the parser used to interpret responses. ### Define Tools for Function Call Below is a Python snippet that shows how to define a tool as a dictionary. The dictionary includes a tool name, a description, and property defined Parameters. ```python Example theme={null} # Define tools tools = [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": "The city to find the weather for, e.g. 'San Francisco'", }, "state": { "type": "string", "description": "the two-letter abbreviation for the state that the city is" " in, e.g. 'CA' which would mean 'California'", }, "unit": { "type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"], }, }, "required": ["city", "state", "unit"], }, }, } ] ``` ### Define Messages ```python Example theme={null} def get_messages(): return [ { "role": "user", "content": "What's the weather like in Boston today? Output a reasoning before act, then use the tools to help you.", } ] messages = get_messages() ``` ### Initialize the Client ```python Example theme={null} # Initialize OpenAI-like client client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:{port}/v1") model_name = client.models.list().data[0].id ``` ### Non-Streaming Request ```python Example theme={null} # Non-streaming mode test response_non_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0, top_p=0.95, max_tokens=1024, stream=False, # Non-streaming tools=tools, ) print_highlight("Non-stream response:") print_highlight(response_non_stream) print_highlight("==== content ====") print_highlight(response_non_stream.choices[0].message.content) print_highlight("==== tool_calls ====") print_highlight(response_non_stream.choices[0].message.tool_calls) ``` #### Handle Tools When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly. ```python Example theme={null} name_non_stream = response_non_stream.choices[0].message.tool_calls[0].function.name arguments_non_stream = ( response_non_stream.choices[0].message.tool_calls[0].function.arguments ) print_highlight(f"Final streamed function call name: {name_non_stream}") print_highlight(f"Final streamed function call arguments: {arguments_non_stream}") ``` ### Streaming Request ```python Example theme={null} # Streaming mode test print_highlight("Streaming response:") response_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0, top_p=0.95, max_tokens=1024, stream=True, # Enable streaming tools=tools, ) texts = "" tool_calls = [] name = "" arguments = "" for chunk in response_stream: if chunk.choices[0].delta.content: texts += chunk.choices[0].delta.content if chunk.choices[0].delta.tool_calls: tool_calls.append(chunk.choices[0].delta.tool_calls[0]) print_highlight("==== Text ====") print_highlight(texts) print_highlight("==== Tool Call ====") for tool_call in tool_calls: print_highlight(tool_call) ``` #### Handle Tools When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly. ```python Example theme={null} # Parse and combine function call arguments arguments = [] for tool_call in tool_calls: if tool_call.function.name: print_highlight(f"Streamed function call name: {tool_call.function.name}") if tool_call.function.arguments: arguments.append(tool_call.function.arguments) # Combine all fragments into a single JSON string full_arguments = "".join(arguments) print_highlight(f"streamed function call arguments: {full_arguments}") ``` ### Define a Tool Function ```python Example theme={null} # This is a demonstration, define real function according to your usage. def get_current_weather(city: str, state: str, unit: "str"): return ( f"The weather in {city}, {state} is 85 degrees {unit}. It is " "partly cloudly, with highs in the 90's." ) available_tools = {"get_current_weather": get_current_weather} ``` ### Execute the Tool ```python Example theme={null} messages.append(response_non_stream.choices[0].message) # Call the corresponding tool function tool_call = messages[-1].tool_calls[0] tool_name = tool_call.function.name tool_to_call = available_tools[tool_name] result = tool_to_call(**(json.loads(tool_call.function.arguments))) print_highlight(f"Function call result: {result}") # messages.append({"role": "tool", "content": result, "name": tool_name}) messages.append( { "role": "tool", "tool_call_id": tool_call.id, "content": str(result), "name": tool_name, } ) print_highlight(f"Updated message history: {messages}") ``` ### Send Results Back to Model ```python Example theme={null} final_response = client.chat.completions.create( model=model_name, messages=messages, temperature=0, top_p=0.95, stream=False, tools=tools, ) print_highlight("Non-stream response:") print_highlight(final_response) print_highlight("==== Text ====") print_highlight(final_response.choices[0].message.content) ``` ## Native API and SGLang Runtime (SRT) ```python Example theme={null} from transformers import AutoTokenizer import requests # generate an answer tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct") messages = get_messages() input = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, tools=tools, return_dict=False ) gen_url = f"http://localhost:{port}/generate" gen_data = { "text": input, "sampling_params": { "skip_special_tokens": False, "max_new_tokens": 1024, "temperature": 0, "top_p": 0.95, }, } gen_response = requests.post(gen_url, json=gen_data).json()["text"] print_highlight("==== Response ====") print_highlight(gen_response) # parse the response parse_url = f"http://localhost:{port}/parse_function_call" function_call_input = { "text": gen_response, "tool_call_parser": "qwen25", "tools": tools, } function_call_response = requests.post(parse_url, json=function_call_input) function_call_response_json = function_call_response.json() print_highlight("==== Text ====") print(function_call_response_json["normal_text"]) print_highlight("==== Calls ====") print("function name: ", function_call_response_json["calls"][0]["name"]) print("function arguments: ", function_call_response_json["calls"][0]["parameters"]) ``` ```python Example theme={null} terminate_process(server_process) ``` ## Offline Engine API ```python Example theme={null} import sglang as sgl from sglang.srt.function_call.function_call_parser import FunctionCallParser from sglang.srt.managers.io_struct import Tool, Function llm = sgl.Engine(model_path="Qwen/Qwen2.5-7B-Instruct") tokenizer = llm.tokenizer_manager.tokenizer input_ids = tokenizer.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, tools=tools, return_dict=False ) # Note that for gpt-oss tool parser, adding "no_stop_trim": True # to make sure the tool call token is not trimmed. sampling_params = { "max_new_tokens": 1024, "temperature": 0, "top_p": 0.95, "skip_special_tokens": False, } # 1) Offline generation result = llm.generate(input_ids=input_ids, sampling_params=sampling_params) generated_text = result["text"] # Assume there is only one prompt print_highlight("=== Offline Engine Output Text ===") print_highlight(generated_text) # 2) Parse using FunctionCallParser def convert_dict_to_tool(tool_dict: dict) -> Tool: function_dict = tool_dict.get("function", {}) return Tool( type=tool_dict.get("type", "function"), function=Function( name=function_dict.get("name"), description=function_dict.get("description"), parameters=function_dict.get("parameters"), ), ) tools = [convert_dict_to_tool(raw_tool) for raw_tool in tools] parser = FunctionCallParser(tools=tools, tool_call_parser="qwen25") normal_text, calls = parser.parse_non_stream(generated_text) print_highlight("=== Parsing Result ===") print("Normal text portion:", normal_text) print_highlight("Function call portion:") for call in calls: # call: ToolCallItem print_highlight(f" - tool name: {call.name}") print_highlight(f" parameters: {call.parameters}") # 3) If needed, perform additional logic on the parsed functions, such as automatically calling the corresponding function to obtain a return value, etc. ``` ```python Example theme={null} llm.shutdown() ``` ## Tool Choice Mode SGLang supports OpenAI's `tool_choice` parameter to control when and which tools the model should call. This feature is implemented using EBNF (Extended Backus-Naur Form) grammar to ensure reliable tool calling behavior. ### Supported Tool Choice Options * **`tool_choice="required"`**: Forces the model to call at least one tool * **`tool_choice={"type": "function", "function": {"name": "specific_function"}}`**: Forces the model to call a specific function ### Backend Compatibility Tool choice is fully supported with the **Xgrammar backend**, which is the default grammar backend (`--grammar-backend xgrammar`). However, it may not be fully supported with other backends such as `outlines`. ### Example: Required Tool Choice ```python Example theme={null} from openai import OpenAI from sglang.utils import wait_for_server, print_highlight, terminate_process from sglang.test.doc_patch import launch_server_cmd # Start a new server session for tool choice examples server_process_tool_choice, port_tool_choice = launch_server_cmd( "python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0 --log-level warning" ) wait_for_server(f"http://localhost:{port_tool_choice}") # Initialize client for tool choice examples client_tool_choice = OpenAI( api_key="None", base_url=f"http://0.0.0.0:{port_tool_choice}/v1" ) model_name_tool_choice = client_tool_choice.models.list().data[0].id # Example with tool_choice="required" - forces the model to call a tool messages_required = [ {"role": "user", "content": "Hello, what is the capital of France?"} ] # Define tools tools = [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": "The city to find the weather for, e.g. 'San Francisco'", }, "unit": { "type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"], }, }, "required": ["city", "unit"], }, }, } ] response_required = client_tool_choice.chat.completions.create( model=model_name_tool_choice, messages=messages_required, temperature=0, max_tokens=1024, tools=tools, tool_choice="required", # Force the model to call a tool ) print_highlight("Response with tool_choice='required':") print("Content:", response_required.choices[0].message.content) print("Tool calls:", response_required.choices[0].message.tool_calls) ``` ### Example: Specific Function Choice ```python Example theme={null} # Example with specific function choice - forces the model to call a specific function messages_specific = [ {"role": "user", "content": "What are the most attactive places in France?"} ] response_specific = client_tool_choice.chat.completions.create( model=model_name_tool_choice, messages=messages_specific, temperature=0, max_tokens=1024, tools=tools, tool_choice={ "type": "function", "function": {"name": "get_current_weather"}, }, # Force the model to call the specific get_current_weather function ) print_highlight("Response with specific function choice:") print("Content:", response_specific.choices[0].message.content) print("Tool calls:", response_specific.choices[0].message.tool_calls) if response_specific.choices[0].message.tool_calls: tool_call = response_specific.choices[0].message.tool_calls[0] print_highlight(f"Called function: {tool_call.function.name}") print_highlight(f"Arguments: {tool_call.function.arguments}") ``` ```python Example theme={null} terminate_process(server_process_tool_choice) ``` ## Pythonic Tool Call Format (Llama-3.2 / Llama-3.3 / Llama-4) Some Llama models (such as Llama-3.2-1B, Llama-3.2-3B, Llama-3.3-70B, and Llama-4) support a "pythonic" tool call format, where the model outputs function calls as Python code, e.g.: ```python Example theme={null} [get_current_weather(city="San Francisco", state="CA", unit="celsius")] ``` * The output is a Python list of function calls, with arguments as Python literals (not JSON). * Multiple tool calls can be returned in the same list: ```python Example theme={null} [get_current_weather(city="San Francisco", state="CA", unit="celsius"), get_current_weather(city="New York", state="NY", unit="fahrenheit")] ``` For more information, refer to Meta’s documentation on [Zero shot function calling](https://github.com/meta-llama/llama-models/blob/main/models/llama4/prompt_format.md#zero-shot-function-calling---system-message). Note that this feature is still under development on Blackwell. ### How to enable * Launch the server with `--tool-call-parser pythonic` * You may also specify --chat-template with the improved template for the model (e.g., `--chat-template=examples/chat_template/tool_chat_template_llama4_pythonic.jinja`). This is recommended because the model expects a special prompt format to reliably produce valid pythonic tool call outputs. The template ensures that the prompt structure (e.g., special tokens, message boundaries like `<|eom|>`, and function call delimiters) matches what the model was trained or fine-tuned on. If you do not use the correct chat template, tool calling may fail or produce inconsistent results. #### Forcing Pythonic Tool Call Output Without a Chat Template If you don't want to specify a chat template, you must give the model extremely explicit instructions in your messages to enforce pythonic output. For example, for `Llama-3.2-1B-Instruct`, you need: ```python Example theme={null} import openai server_process, port = launch_server_cmd( " python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-1B-Instruct --tool-call-parser pythonic --tp 1 --log-level warning" # llama-3.2-1b-instruct ) wait_for_server(f"http://localhost:{port}") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a given location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The name of the city or location.", } }, "required": ["location"], }, }, }, { "type": "function", "function": { "name": "get_tourist_attractions", "description": "Get a list of top tourist attractions for a given city.", "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": "The name of the city to find attractions for.", } }, "required": ["city"], }, }, }, ] def get_messages(): return [ { "role": "system", "content": ( "You are a travel assistant. " "When asked to call functions, ALWAYS respond ONLY with a python list of function calls, " "using this format: [func_name1(param1=value1, param2=value2), func_name2(param=value)]. " "Do NOT use JSON, do NOT use variables, do NOT use any other format. " "Here is an example:\n" '[get_weather(location="Paris"), get_tourist_attractions(city="Paris")]' ), }, { "role": "user", "content": ( "I'm planning a trip to Tokyo next week. What's the weather like and what are some top tourist attractions? " "Propose parallel tool calls at once, using the python list of function calls format as shown above." ), }, ] messages = get_messages() client = openai.Client(base_url=f"http://localhost:{port}/v1", api_key="xxxxxx") model_name = client.models.list().data[0].id response_non_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0, top_p=0.9, stream=False, # Non-streaming tools=tools, ) print_highlight("Non-stream response:") print_highlight(response_non_stream) response_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0, top_p=0.9, stream=True, tools=tools, ) texts = "" tool_calls = [] name = "" arguments = "" for chunk in response_stream: if chunk.choices[0].delta.content: texts += chunk.choices[0].delta.content if chunk.choices[0].delta.tool_calls: tool_calls.append(chunk.choices[0].delta.tool_calls[0]) print_highlight("Streaming Response:") print_highlight("==== Text ====") print_highlight(texts) print_highlight("==== Tool Call ====") for tool_call in tool_calls: print_highlight(tool_call) terminate_process(server_process) ``` > **Note:** > The model may still default to JSON if it was heavily finetuned on that format. Prompt engineering (including examples) is the only way to increase the chance of pythonic output if you are not using a chat template. ## How to support a new model? 1. Update the TOOLS\_TAG\_LIST in sglang/srt/function\_call\_parser.py with the model’s tool tags. Currently supported tags include: ```text Output theme={null} TOOLS_TAG_LIST = [ “<|plugin|>“, ““, “<|python_tag|>“, “[TOOL_CALLS]” ] ``` 2. Create a new detector class in sglang/srt/function\_call\_parser.py that inherits from BaseFormatDetector. The detector should handle the model’s specific function call format. For example: ```text Output theme={null} class NewModelDetector(BaseFormatDetector): ``` 3. Add the new detector to the MultiFormatParser class that manages all the format detectors. # Query VLM with Offline Engine Source: https://docs.sglang.io/docs/advanced_features/vlm_query This tutorial demonstrates how to use SGLang's **offline Engine API** to query VLMs. We will demonstrate usage with Qwen2.5-VL and Llama 4. This section demonstrates three different calling approaches: 1. **Basic Call**: Directly pass images and text. 2. **Processor Output**: Use HuggingFace processor for data preprocessing. 3. **Precomputed Embeddings**: Pre-calculate image features to improve inference efficiency. ## Understanding the Three Input Formats SGLang supports three ways to pass visual data, each optimized for different scenarios: ### 1. **Raw Images** - Simplest approach * Pass PIL Images, file paths, URLs, or base64 strings directly * SGLang handles all preprocessing automatically * Best for: Quick prototyping, simple applications ### 2. **Processor Output** - For custom preprocessing * Pre-process images with HuggingFace processor * Pass the complete processor output dict with `format: "processor_output"` * Best for: Custom image transformations, integration with existing pipelines * Requirement: Must use `input_ids` instead of text prompt ### 3. **Precomputed Embeddings** - For maximum performance * Pre-calculate visual embeddings using the vision encoder * Pass embeddings with `format: "precomputed_embedding"` * Best for: Repeated queries on same images, caching, high-throughput serving * Performance gain: Avoids redundant vision encoder computation (30-50% speedup) **Key Rule**: Within a single request, use only one format for all images. Don't mix formats. The examples below demonstrate all three approaches with both Qwen2.5-VL and Llama 4 models. ## Querying Qwen2.5-VL Model ```python Example theme={null} import nest_asyncio nest_asyncio.apply() import sglang.test.doc_patch # noqa: F401 model_path = "Qwen/Qwen2.5-VL-3B-Instruct" chat_template = "qwen2-vl" example_image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png" ``` ```python Example theme={null} from io import BytesIO import requests from PIL import Image from sglang.srt.parser.conversation import chat_templates image = Image.open(BytesIO(requests.get(example_image_url).content)) conv = chat_templates[chat_template].copy() conv.append_message(conv.roles[0], f"What's shown here: {conv.image_token}?") conv.append_message(conv.roles[1], "") conv.image_data = [image] print("Generated prompt text:") print(conv.get_prompt()) print(f"\nImage size: {image.size}") image ``` ### Basic Offline Engine API Call ```python Example theme={null} from sglang import Engine llm = Engine(model_path=model_path, chat_template=chat_template, log_level="warning") ``` ```python Example theme={null} out = llm.generate(prompt=conv.get_prompt(), image_data=[image]) print("Model response:") print(out["text"]) ``` ### Call with Processor Output Using a HuggingFace processor to preprocess text and images, and passing the `processor_output` directly into `Engine.generate`. ```python Example theme={null} from transformers import AutoProcessor processor = AutoProcessor.from_pretrained(model_path) processor_output = processor( images=[image], text=conv.get_prompt(), return_tensors="pt" ) out = llm.generate( input_ids=processor_output["input_ids"][0].detach().cpu().tolist(), image_data=[dict(processor_output, format="processor_output")], ) print("Response using processor output:") print(out["text"]) ``` ### Call with Precomputed Embeddings You can pre-calculate image features to avoid repeated visual encoding processes. ```python Example theme={null} from transformers import AutoProcessor from transformers import Qwen2_5_VLForConditionalGeneration processor = AutoProcessor.from_pretrained(model_path) model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_path).eval() vision = model.model.visual.cuda() ``` ```python Example theme={null} processor_output = processor( images=[image], text=conv.get_prompt(), return_tensors="pt" ) input_ids = processor_output["input_ids"][0].detach().cpu().tolist() precomputed_embeddings = vision( processor_output["pixel_values"].cuda(), processor_output["image_grid_thw"].cuda() ) precomputed_embeddings = precomputed_embeddings.pooler_output multi_modal_item = dict( processor_output, format="precomputed_embedding", feature=precomputed_embeddings, ) out = llm.generate(input_ids=input_ids, image_data=[multi_modal_item]) print("Response using precomputed embeddings:") print(out["text"]) llm.shutdown() ``` ## Querying Llama 4 Vision Model ```python Example theme={null} model_path = "meta-llama/Llama-4-Scout-17B-16E-Instruct" chat_template = "llama-4" from io import BytesIO import requests from PIL import Image from sglang.srt.parser.conversation import chat_templates # Download the same example image image = Image.open(BytesIO(requests.get(example_image_url).content)) conv = chat_templates[chat_template].copy() conv.append_message(conv.roles[0], f"What's shown here: {conv.image_token}?") conv.append_message(conv.roles[1], "") conv.image_data = [image] print("Llama 4 generated prompt text:") print(conv.get_prompt()) print(f"Image size: {image.size}") image ``` ### Llama 4 Basic Call Llama 4 requires more computational resources, so it's configured with multi-GPU parallelism (tp\_size=4) and larger context length. ```python Example theme={null} llm = Engine( model_path=model_path, enable_multimodal=True, attention_backend="fa3", tp_size=4, context_length=65536, ) out = llm.generate(prompt=conv.get_prompt(), image_data=[image]) print("Llama 4 response:") print(out["text"]) ``` ### Call with Processor Output Using HuggingFace processor to preprocess data can reduce computational overhead during inference. ```python Example theme={null} from transformers import AutoProcessor processor = AutoProcessor.from_pretrained(model_path) processor_output = processor( images=[image], text=conv.get_prompt(), return_tensors="pt" ) out = llm.generate( input_ids=processor_output["input_ids"][0].detach().cpu().tolist(), image_data=[dict(processor_output, format="processor_output")], ) print("Response using processor output:") print(out) ``` ### Call with Precomputed Embeddings ```python Example theme={null} from transformers import AutoProcessor from transformers import Llama4ForConditionalGeneration processor = AutoProcessor.from_pretrained(model_path) model = Llama4ForConditionalGeneration.from_pretrained( model_path, torch_dtype="auto" ).eval() vision = model.vision_model.cuda() multi_modal_projector = model.multi_modal_projector.cuda() print(f'Image pixel values shape: {processor_output["pixel_values"].shape}') input_ids = processor_output["input_ids"][0].detach().cpu().tolist() # Process image through vision encoder image_outputs = vision( processor_output["pixel_values"].to("cuda"), aspect_ratio_ids=processor_output["aspect_ratio_ids"].to("cuda"), aspect_ratio_mask=processor_output["aspect_ratio_mask"].to("cuda"), output_hidden_states=False ) image_features = image_outputs.last_hidden_state # Flatten image features and pass through multimodal projector vision_flat = image_features.view(-1, image_features.size(-1)) precomputed_embeddings = multi_modal_projector(vision_flat) # Build precomputed embedding data item mm_item = dict( processor_output, format="precomputed_embedding", feature=precomputed_embeddings ) # Use precomputed embeddings for efficient inference out = llm.generate(input_ids=input_ids, image_data=[mm_item]) print("Llama 4 precomputed embedding response:") print(out["text"]) ``` # Anthropic-Compatible API Source: https://docs.sglang.io/docs/basic_usage/anthropic_api Use the Anthropic Messages API (/v1/messages) with SGLang, including Claude Code integration and prefix-cache tuning. SGLang ships an Anthropic-compatible `/v1/messages` endpoint so any client built for the Anthropic Messages API — including the Anthropic SDKs and agentic CLIs such as Claude Code — can talk to a self-hosted SGLang server without changes. A complete reference for the API is available in the [Anthropic API Reference](https://docs.anthropic.com/en/api/messages). The endpoint is registered automatically on every SGLang server; no extra flag is required to enable it. It reuses the same model, chat template, and reasoning / tool-call parsers as the OpenAI-compatible endpoint, and supports both non-streaming and streaming responses, tool use, and a `count_tokens` route. This tutorial covers: * `POST /v1/messages` (non-streaming and streaming) * `POST /v1/messages/count_tokens` * Pointing **Claude Code** at the server, including the `CLAUDE_CODE_ATTRIBUTION_HEADER` setting that is required for good prefix-cache reuse. ## Launch A Server Launch the server in your terminal and wait for it to initialize. The Anthropic `/v1/messages` endpoint is registered automatically — no extra flag is required beyond the usual server launch. The example below is a single-node GLM-5.2-FP8 config; see the [GLM-5.2 cookbook](/cookbook/autoregressive/GLM/GLM-5.2) for verified commands across hardware and quantizations. ```bash Command theme={null} sglang serve \ --model-path zai-org/GLM-5.2-FP8 \ --tp 8 \ --speculative-algorithm EAGLE \ --speculative-num-steps 5 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 6 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --host 0.0.0.0 \ --port 30000 ``` * **The endpoint is model-agnostic.** The `/v1/messages` route is on by default for any model; GLM-5.2 is used here because its reasoning + tool-use output is where Claude Code integration shines, but any model works. * **Model name and `[1m]`.** SGLang does not validate the request `model` field, so Claude Code can send any name. The `[1m]` suffix is a **client-side hint**: Claude Code only enables its 1M-context beta when the model name ends in `[1m]` — without it, context is capped. Set the same `glm-5.2[1m]` in the `ANTHROPIC_DEFAULT_*_MODEL` env vars below. * **`--reasoning-parser` / `--tool-call-parser` are optional.** Add them when the model emits reasoning content (GLM-5.2, Qwen3, DeepSeek-R1, …) or when you want tool calls parsed into structured `tool_use` blocks. Without a tool-call parser, tool schemas are still accepted but the model's tool calls come back as raw text, and Claude Code cannot execute them. * **Context length** defaults to the model's own (1M for GLM-5.2); pass `--context-length` only to cap it. ## Send A Message ### Non-Streaming Use the Anthropic Python SDK pointed at the server. Unlike the OpenAI SDK, the Anthropic SDK appends `/v1/messages` itself, so `base_url` is the server root **without** a `/v1` suffix. ```python Example theme={null} from anthropic import Anthropic client = Anthropic( base_url="http://127.0.0.1:30000", api_key="EMPTY", # SGLang does not require a real key by default ) message = client.messages.create( model="zai-org/GLM-5.2-FP8", max_tokens=512, messages=[{"role": "user", "content": "List 3 countries and their capitals."}], ) # A reasoning model may emit a `thinking` block before the `text` block — # pick the text block rather than assuming content[0]. print(next(b.text for b in message.content if b.type == "text")) ``` **Example Output:** ```text Output theme={null} Here are 3 countries and their capitals: 1. **France** - Paris 2. **Japan** - Tokyo 3. **Brazil** - Brasília ``` ### Streaming Set `stream=True` to receive Server-Sent Events as they are produced. ```python Example theme={null} with client.messages.stream( model="zai-org/GLM-5.2-FP8", max_tokens=512, messages=[{"role": "user", "content": "Say this is a test"}], ) as stream: for text in stream.text_stream: print(text, end="", flush=True) ``` **Example Output:** ```text Output theme={null} This is a test. ``` ### System Prompt The top-level `system` field is accepted as a string or as a list of text blocks, matching the Anthropic API shape: ```python Example theme={null} message = client.messages.create( model="zai-org/GLM-5.2-FP8", max_tokens=512, system="You are a helpful assistant that answers concisely.", messages=[{"role": "user", "content": "What is the capital of France?"}], ) print(next(b.text for b in message.content if b.type == "text")) ``` **Example Output:** ```text Output theme={null} The capital of France is Paris. ``` ### Tool Use Tool definitions follow the Anthropic `tools` schema. When the server is launched with a `--tool-call-parser`, the model's tool calls are returned as `tool_use` content blocks: ```python Example theme={null} message = client.messages.create( model="zai-org/GLM-5.2-FP8", max_tokens=512, tools=[ { "name": "get_weather", "description": "Get the weather for a city", "input_schema": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, } ], messages=[{"role": "user", "content": "What is the weather in Paris?"}], ) print(message.stop_reason) print([b for b in message.content if b.type == "tool_use"]) ``` **Example Output:** ```text Output theme={null} tool_use [ToolUseBlock(type='tool_use', id='toolu_01XXXX', name='get_weather', input={'city': 'Paris'})] ``` ### Counting Tokens `POST /v1/messages/count_tokens` returns the tokenized length of a request without generating a response. It reuses the same request conversion as `/v1/messages`, so system prompts, tools, and multi-turn history are all accounted for. ```python Example theme={null} resp = client.messages.count_tokens( model="zai-org/GLM-5.2-FP8", messages=[{"role": "user", "content": "Hello, world"}], ) print(resp.input_tokens) ``` **Example Output:** ```text Output theme={null} 15 ``` ## Using Claude Code Claude Code can be pointed at an SGLang server by setting a few env vars in the shell that starts it. With the server already running on `:30000`, export the full set and launch `claude`: ```bash Command theme={null} export ANTHROPIC_BASE_URL="http://127.0.0.1:30000" export ANTHROPIC_AUTH_TOKEN="dummy" # required by Claude Code; any non-empty string works export API_TIMEOUT_MS="3000000" # long timeout — reasoning + 1M-context turns are slow export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000" # let auto-compact use the full 1M window export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 # drop autoupdater/telemetry/error-reporting noise export CLAUDE_CODE_ATTRIBUTION_HEADER=0 # required for prefix-cache reuse — see below export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]" # [1m] suffix enables Claude Code's 1M-context beta export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]" # [1m] suffix enables Claude Code's 1M-context beta export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]" # [1m] suffix enables Claude Code's 1M-context beta claude ``` Each var matters: * **`ANTHROPIC_BASE_URL`** — points Claude Code at your SGLang server instead of the Anthropic API. * **`ANTHROPIC_AUTH_TOKEN`** — Claude Code requires a non-empty auth token; SGLang accepts any value when launched without `--api-key`. * **`API_TIMEOUT_MS`** — raise it; reasoning models with long outputs and 1M-context turns routinely exceed the default timeout. * **`ANTHROPIC_DEFAULT_{HAIKU,SONNET,OPUS}_MODEL`** — the model name Claude Code sends for each tier. SGLang does not validate this field, so any name works. Use `glm-5.2[1m]`: the `[1m]` suffix is a client-side hint that enables Claude Code's 1M-context beta (without it, context is capped). * **`CLAUDE_CODE_AUTO_COMPACT_WINDOW`** — set to `1000000` so auto-compaction uses the full 1M window instead of the default, keeping long sessions alive. Instead of exporting these in every shell, persist them in `~/.claude/settings.json` under the `env` key — they apply to all Claude Code sessions: ```json theme={null} { "env": { "ANTHROPIC_BASE_URL": "http://127.0.0.1:30000", "ANTHROPIC_AUTH_TOKEN": "dummy", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1", "CLAUDE_CODE_ATTRIBUTION_HEADER": "0", "ANTHROPIC_DEFAULT_HAIKU_MODEL": "glm-5.2[1m]", "ANTHROPIC_DEFAULT_SONNET_MODEL": "glm-5.2[1m]", "ANTHROPIC_DEFAULT_OPUS_MODEL": "glm-5.2[1m]" } } ``` ### Required: `CLAUDE_CODE_ATTRIBUTION_HEADER=0` for prefix-cache reuse **Set this whenever Claude Code routes through SGLang (or any non-Anthropic gateway).** Without it, multi-turn conversations re-prefill the whole history every turn. Claude Code prepends a per-request attribution block to the start of the system prompt, of the form `x-anthropic-billing-header: cc_version=.; cc_entrypoint=...; cch=;`. The per-request hash is the **first token to differ between turns**, so the radix prefix cache can only reuse the short prefix before that hash and re-prefills the system prompt plus the entire conversation history on every turn. Setting `CLAUDE_CODE_ATTRIBUTION_HEADER=0` removes the whole attribution line from the system prompt. This is a documented Claude Code env var whose explicit purpose is to "improve prompt-cache hit rates when routing through an [LLM gateway](https://code.claude.com/docs/en/llm-gateway)" (see the [Claude Code env-vars reference](https://code.claude.com/docs/en/env-vars)). `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC` does **not** remove the attribution block — it only covers autoupdater/telemetry/error reporting. The attribution header is a separate code path; use `CLAUDE_CODE_ATTRIBUTION_HEADER=0` for it. ## Troubleshooting **Connection refused / `fetch failed`** — Ensure the server is up and the port in `ANTHROPIC_BASE_URL` matches `--port` (default 30000). If you set `ANTHROPIC_BASE_URL` to a remote host, confirm it's reachable and not behind a proxy that blocks the connection. **`Model not found` / 404 from the server** — SGLang does not validate the request `model` field and serves whatever model was loaded at startup, so a 404 usually means the request did not reach the `/v1/messages` route at all. Confirm `ANTHROPIC_BASE_URL` points at the server (not missing the port) and that the server finished loading. **Tool calls not working / returned as raw text** — Launch the server with the correct `--tool-call-parser` for your model (e.g. `glm47`, `qwen3`). Without it the `tools` field is still accepted but the model's tool calls come back as text instead of `tool_use` blocks, and Claude Code cannot execute them. **Slow / re-prefills the whole history every turn** — You are missing `CLAUDE_CODE_ATTRIBUTION_HEADER=0`. Claude Code's per-request attribution hash in the system prompt defeats radix prefix-cache reuse; see the section above. **Context capped below 1M** — The model name must end in `[1m]` for Claude Code to enable its 1M-context beta. Verify `ANTHROPIC_DEFAULT_*_MODEL` uses the `[1m]` suffix, and that the loaded model's native context is 1M (GLM-5.2 is 1048576; pass `--context-length` only to cap it, not to extend). ## Parameters The `/v1/messages` endpoint accepts the standard Anthropic Messages API parameters. Refer to the [Anthropic Messages API reference](https://docs.anthropic.com/en/api/messages) for the full list. Reasoning models are supported through the same `--reasoning-parser` mechanism as the OpenAI-compatible endpoint; pass the model's reasoning kwarg via the request (e.g. `thinking` for DeepSeek-V3-style models, `enable_thinking` for Qwen3-style models). See [OpenAI APIs - Completions](./openai_api_completions) for the reasoning-parser / chat-template mapping. # Amazon SageMaker AI Source: https://docs.sglang.io/docs/basic_usage/aws_sagemaker Deploy SGLang on Amazon SageMaker AI endpoints using the AWS Deep Learning Container. Deploy SGLang on [Amazon SageMaker AI](https://aws.amazon.com/sagemaker/) endpoints using the [AWS Deep Learning Container (DLC)](https://aws.github.io/deep-learning-containers/sglang/) for SGLang. The SageMaker image variant accepts model configuration via environment variables and serves on port 8080. This guide uses the pre-built DLC image. To build and deploy your own container instead, see [Method 7: Run on AWS SageMaker](/docs/get-started/install#aws-sagemaker) in the installation guide. ## Container image AWS publishes pre-built, security-patched SGLang DLCs. The SageMaker GPU image is available from the Amazon ECR registry (account `763104351884`) in each supported region. For example, in `us-west-2`: ```text theme={null} 763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0 ``` For the full list of image tags, see the [Available DLC Images](https://aws.github.io/deep-learning-containers/reference/available_images/) reference, and for region-specific account IDs and supported regions, see [Region Availability](https://aws.github.io/deep-learning-containers/reference/region_availability/). ## Specifying the model The SageMaker image resolves the model in this order: 1. **`SM_SGLANG_MODEL_PATH` environment variable** — explicit Hugging Face ID or path. 2. **`/opt/ml/model`** — when SageMaker mounts model artifacts via `ModelDataUrl` or `ModelDataSource`, the entrypoint uses this path by default. For gated models, also pass `HF_TOKEN`. Any `SM_SGLANG_*` environment variable is converted to a `--` SGLang server argument (for example, `SM_SGLANG_CONTEXT_LENGTH=4096` becomes `--context-length 4096`). ## Deploy with the SageMaker Python SDK ```python theme={null} from sagemaker.model import Model from sagemaker.predictor import Predictor from sagemaker.serializers import JSONSerializer model = Model( image_uri="763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0", role="arn:aws:iam:::role/", predictor_cls=Predictor, env={"SM_SGLANG_MODEL_PATH": "openai/gpt-oss-20b"}, ) predictor = model.deploy( instance_type="ml.g5.2xlarge", initial_instance_count=1, inference_ami_version="al2023-ami-sagemaker-inference-gpu-4-1", serializer=JSONSerializer(), ) response = predictor.predict({ "model": "openai/gpt-oss-20b", "messages": [{"role": "user", "content": "What is deep learning?"}], "max_tokens": 256, }) print(response) # Cleanup predictor.delete_model() predictor.delete_endpoint(delete_endpoint_config=True) ``` ## Deploy with Boto3 ```python theme={null} import json import boto3 sm = boto3.client("sagemaker") smrt = boto3.client("sagemaker-runtime") sm.create_model( ModelName="sglang-model", PrimaryContainer={ "Image": "763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0", "Environment": {"SM_SGLANG_MODEL_PATH": "openai/gpt-oss-20b"}, }, ExecutionRoleArn="arn:aws:iam:::role/", ) sm.create_endpoint_config( EndpointConfigName="sglang-config", ProductionVariants=[{ "VariantName": "default", "ModelName": "sglang-model", "InstanceType": "ml.g5.2xlarge", "InitialInstanceCount": 1, "InferenceAmiVersion": "al2023-ami-sagemaker-inference-gpu-4-1", }], ) sm.create_endpoint(EndpointName="sglang-endpoint", EndpointConfigName="sglang-config") sm.get_waiter("endpoint_in_service").wait(EndpointName="sglang-endpoint") resp = smrt.invoke_endpoint( EndpointName="sglang-endpoint", ContentType="application/json", Body=json.dumps({ "model": "openai/gpt-oss-20b", "messages": [{"role": "user", "content": "What is deep learning?"}], "max_tokens": 256, }), ) print(json.loads(resp["Body"].read())) # Cleanup sm.delete_endpoint(EndpointName="sglang-endpoint") sm.delete_endpoint_config(EndpointConfigName="sglang-config") sm.delete_model(ModelName="sglang-model") ``` ## Model artifacts When `ModelDataUrl` (or `ModelDataSource`) points to a tarball or S3 prefix, SageMaker mounts the contents at `/opt/ml/model`. The entrypoint defaults `--model-path` to that location, so `SM_SGLANG_MODEL_PATH` can be omitted: ```text theme={null} model.tar.gz ├── config.json # standard model files (Hugging Face layout) ├── tokenizer.json └── *.safetensors ``` ## Notes * GPU deployments require `inference_ami_version` — the default SageMaker host AMI has incompatible NVIDIA drivers for CUDA 13 images. See the [ProductionVariant API reference](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ProductionVariant.html) for valid values. * The endpoint exposes an OpenAI-compatible API, so the request body matches the SGLang server's `/v1/chat/completions` schema. # SGLang Native APIs Source: https://docs.sglang.io/docs/basic_usage/native_api Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce the following APIs: * `/generate` (text generation model) * `/get_model_info` * `/server_info` * `/health` * `/health_generate` * `/flush_cache` * `/update_weights` * `/encode`(embedding model) * `/v1/rerank`(cross encoder rerank model) * `/v1/score`(decoder-only scoring) * `/v1/decisions`(typed decisions on a chat model) * `/classify`(reward model) * `/start_expert_distribution_record` * `/stop_expert_distribution_record` * `/dump_expert_distribution_record` * `/tokenize` * `/detokenize` * A full list of these APIs can be found at [http\_server.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/http_server.py) We mainly use `requests` to test these APIs in the following examples. You can also use `curl`. ## Launch A Server ```python Example theme={null} from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process server_process, port = launch_server_cmd( "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning" ) wait_for_server(f"http://localhost:{port}", process=server_process) ``` ## Generate (text generation model) Generate completions. This is similar to the `/v1/completions` in OpenAI API. Detailed parameters can be found in the [sampling parameters](./sampling_params). ```python Example theme={null} import requests url = f"http://localhost:{port}/generate" data = {"text": "What is the capital of France?"} response = requests.post(url, json=data) print_highlight(response.json()) ``` ## Get Model Info Get the information of the model. * `model_path`: The path/name of the model. * `is_generation`: Whether the model is used as generation model or embedding model. * `tokenizer_path`: The path/name of the tokenizer. * `preferred_sampling_params`: The default sampling params specified via `--preferred-sampling-params`. `None` is returned in this example as we did not explicitly configure it in server args. * `weight_version`: This field contains the version of the model weights. This is often used to track changes or updates to the model’s trained parameters. * `has_image_understanding`: Whether the model has image-understanding capability. * `has_audio_understanding`: Whether the model has audio-understanding capability. * `model_type`: The model type from the HuggingFace config (e.g., "qwen2", "llama"). * `architectures`: The model architectures from the HuggingFace config (e.g., \["Qwen2ForCausalLM"]). * `embedding`: The resolved embedding-serving plan. It includes pooling, normalization, execution and attention style, Matryoshka dimensions, cache policy, and effective BCG prefill settings. This field is available when the model configuration exposes an embedding capability contract. ```python Example theme={null} url = f"http://localhost:{port}/get_model_info" response = requests.get(url) response_json = response.json() print_highlight(response_json) assert response_json["model_path"] == "qwen/qwen2.5-0.5b-instruct" assert response_json["is_generation"] is True assert response_json["tokenizer_path"] == "qwen/qwen2.5-0.5b-instruct" assert response_json["preferred_sampling_params"] is None assert response_json.keys() == { "model_path", "is_generation", "tokenizer_path", "preferred_sampling_params", "weight_version", "has_image_understanding", "has_audio_understanding", "model_type", "architectures", "embedding", } ``` ## Get Server Info Gets the server information including CLI arguments, token limits, and memory pool sizes. * Note: `get_server_info` merges the following deprecated endpoints: * `get_server_args` * `get_memory_pool_size` * `get_max_total_num_tokens` ```python Example theme={null} url = f"http://localhost:{port}/server_info" response = requests.get(url) print_highlight(response.text) ``` ## Health Check * `/health`: Check the health of the server. * `/health_generate`: Check the health of the server by generating one token. ```python Example theme={null} url = f"http://localhost:{port}/health_generate" response = requests.get(url) print_highlight(response.text) ``` ```python Example theme={null} url = f"http://localhost:{port}/health" response = requests.get(url) print_highlight(response.text) ``` ## Flush Cache Flush the radix cache. It will be automatically triggered when the model weights are updated by the `/update_weights` API. Parameters: * `timeout` (query, float, default `0`, unit: seconds): Wait time for idle state before flushing. `0` means fail fast if not idle. When HiCache async operations are in-flight, a non-zero timeout allows the server to wait until idle before flushing, avoiding unnecessary 400 errors. ```bash Command theme={null} # With timeout (wait up to 30s for idle state) curl -s -X POST "http://127.0.0.1:30000/flush_cache?timeout=30" ``` ```python Example theme={null} url = f"http://localhost:{port}/flush_cache" response = requests.post(url) print_highlight(response.text) ``` ## Update Weights From Disk Update model weights from disk without restarting the server. Only applicable for models with the same architecture and parameter size. SGLang support `update_weights_from_disk` API for continuous evaluation during training (save checkpoint to disk and update weights from disk). ```python Example theme={null} # successful update with same architecture and size url = f"http://localhost:{port}/update_weights_from_disk" data = {"model_path": "qwen/qwen2.5-0.5b-instruct"} response = requests.post(url, json=data) print_highlight(response.text) assert response.json()["success"] is True assert response.json()["message"] == "Succeeded to update model weights." ``` ```python Example theme={null} # failed update with different parameter size or wrong name url = f"http://localhost:{port}/update_weights_from_disk" data = {"model_path": "qwen/qwen2.5-0.5b-instruct-wrong"} response = requests.post(url, json=data) response_json = response.json() print_highlight(response_json) assert response_json["success"] is False assert response_json["message"] == ( "Failed to get weights iterator: " "qwen/qwen2.5-0.5b-instruct-wrong" " (repository not found)." ) ``` ```python Example theme={null} terminate_process(server_process) ``` ## Encode (embedding model) Encode text into embeddings. Note that this API is only available for [embedding models](./openai_api_embeddings) and will raise an error for generation models. Therefore, we launch a new server to server an embedding model. ```python Example theme={null} embedding_process, port = launch_server_cmd(""" python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \ --host 0.0.0.0 --is-embedding --log-level warning """) wait_for_server(f"http://localhost:{port}", process=embedding_process) ``` ```python Example theme={null} # successful encode for embedding model url = f"http://localhost:{port}/encode" data = {"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "text": "Once upon a time"} response = requests.post(url, json=data) response_json = response.json() print_highlight(f"Text embedding (first 10): {response_json['embedding'][:10]}") ``` ```python Example theme={null} terminate_process(embedding_process) ``` ## v1/rerank (cross encoder rerank model) Rerank a list of documents given a query using a cross-encoder model. Note that this API is only available for cross encoder model like [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) with `attention-backend` `triton` and `torch_native`. ```python Example theme={null} reranker_process, port = launch_server_cmd(""" python3 -m sglang.launch_server --model-path BAAI/bge-reranker-v2-m3 \ --host 0.0.0.0 --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton --is-embedding --log-level warning """) wait_for_server(f"http://localhost:{port}", process=reranker_process) ``` ```python Example theme={null} # compute rerank scores for query and documents url = f"http://localhost:{port}/v1/rerank" data = { "model": "BAAI/bge-reranker-v2-m3", "query": "what is panda?", "documents": [ "hi", "The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.", ], } response = requests.post(url, json=data) response_json = response.json() for item in response_json: print_highlight(f"Score: {item['score']:.2f} - Document: '{item['document']}'") ``` ```python Example theme={null} terminate_process(reranker_process) ``` ## v1/score (decoder-only scoring) Compute token probabilities for specified tokens given a query and items. This is useful for classification tasks, scoring responses, or computing log-probabilities. Parameters: * `query`: Query text * `items`: Item text(s) to score * `label_token_ids`: Token IDs to compute probabilities for * `apply_softmax`: Whether to apply softmax to get normalized probabilities (default: False) * `item_first`: Whether items come first in concatenation order (default: False) * `model`: Model name `query` and `items` also accept token ids, with `query` set to `[]` when each item is a complete prompt. `label_token_ids` can be one list per item. `temperature` divides the label logits before the softmax and needs `apply_softmax`, and `return_token_logprobs` adds `token_logprobs`, the full-vocabulary log-probabilities of the labels. The response contains `scores` - a list of probability lists, one per item, each in the order of `label_token_ids`. ```python Example theme={null} score_process, port = launch_server_cmd(""" python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \ --host 0.0.0.0 --log-level warning """) wait_for_server(f"http://localhost:{port}", process=score_process) ``` ```python Example theme={null} # Score the probability of different completions given a query query = "The capital of France is" items = ["Paris", "London", "Berlin"] url = f"http://localhost:{port}/v1/score" data = { "model": "qwen/qwen2.5-0.5b-instruct", "query": query, "items": items, "label_token_ids": [9454, 2753], # e.g. "Yes" and "No" token ids "apply_softmax": True, # Normalize probabilities to sum to 1 } response = requests.post(url, json=data) response_json = response.json() # Display scores for each item for item, scores in zip(items, response_json["scores"]): print_highlight(f"Item '{item}': probabilities = {[f'{s:.4f}' for s in scores]}") ``` ```python Example theme={null} terminate_process(score_process) ``` ## v1/decisions (typed decisions on a chat model) Answer typed questions about an input with per-option probabilities, without generating. The server renders each question with the model's chat template and thinking turned off, assigns one-token answer labels, and scores them through the same path as `v1/score`. Parameters: * `input`: Text, object, or array the questions are about * `questions`: List of questions, each with a unique `id` and a `type` of `choice` (with `options`), `score` (with `levels`), or `yes_no` * `temperature`: Divides the label logits before the softmax (default: 1) * `chat_template_kwargs`: Extra chat template arguments * `prompt_format_version`: Optional pin of the server-owned prompt wording * `return_prompt_token_ids`: Return the scored prompt and label token ids for replay through `v1/score` (default: False) The response contains `answers` keyed by question id, each with `type`, `probabilities`, and `label_mass`, plus `choice` for a choice question or `score` for a score question. A yes or no answer is read from `probabilities["yes"]`. See [Decision models](/docs/supported-models/decision_models) for examples, the full reference, and error cases. ## Classify (reward model) SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations. ```python Example theme={null} # Note that SGLang now treats embedding models and reward models as the same type of models. # This will be updated in the future. reward_process, port = launch_server_cmd(""" python3 -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --host 0.0.0.0 --is-embedding --log-level warning """) wait_for_server(f"http://localhost:{port}", process=reward_process) ``` ```python Example theme={null} from transformers import AutoTokenizer PROMPT = ( "What is the range of the numeric output of a sigmoid node in a neural network?" ) RESPONSE1 = "The output of a sigmoid node is bounded between -1 and 1." RESPONSE2 = "The output of a sigmoid node is bounded between 0 and 1." CONVS = [ [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE1}], [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE2}], ] tokenizer = AutoTokenizer.from_pretrained("Skywork/Skywork-Reward-Llama-3.1-8B-v0.2") prompts = tokenizer.apply_chat_template(CONVS, tokenize=False, return_dict=False) url = f"http://localhost:{port}/classify" data = {"model": "Skywork/Skywork-Reward-Llama-3.1-8B-v0.2", "text": prompts} responses = requests.post(url, json=data).json() for response in responses: print_highlight(f"reward: {response['embedding'][0]}") ``` ```python Example theme={null} terminate_process(reward_process) ``` ## Capture expert selection distribution in MoE models SGLang Runtime supports recording the number of times an expert is selected in a MoE model run for each expert in the model. This is useful when analyzing the throughput of the model and plan for optimization. *Note: We only print out the first 10 lines of the csv below for better readability. Please adjust accordingly if you want to analyze the results more deeply.* ```python Example theme={null} expert_record_server_process, port = launch_server_cmd( "python3 -m sglang.launch_server --model-path Qwen/Qwen1.5-MoE-A2.7B --host 0.0.0.0 --expert-distribution-recorder-mode stat --log-level warning" ) wait_for_server(f"http://localhost:{port}", process=expert_record_server_process) ``` ```python Example theme={null} response = requests.post(f"http://localhost:{port}/start_expert_distribution_record") print_highlight(response) url = f"http://localhost:{port}/generate" data = {"text": "What is the capital of France?"} response = requests.post(url, json=data) print_highlight(response.json()) response = requests.post(f"http://localhost:{port}/stop_expert_distribution_record") print_highlight(response) response = requests.post(f"http://localhost:{port}/dump_expert_distribution_record") print_highlight(response) ``` ```python Example theme={null} terminate_process(expert_record_server_process) ``` ## Tokenize/Detokenize Example (Round Trip) This example demonstrates how to use the /tokenize and /detokenize endpoints together. We first tokenize a string, then detokenize the resulting IDs to reconstruct the original text. This workflow is useful when you need to handle tokenization externally but still leverage the server for detokenization. ```python Example theme={null} tokenizer_free_server_process, port = launch_server_cmd(""" python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct """) wait_for_server(f"http://localhost:{port}", process=tokenizer_free_server_process) ``` ```python Example theme={null} import requests from sglang.utils import print_highlight base_url = f"http://localhost:{port}" tokenize_url = f"{base_url}/tokenize" detokenize_url = f"{base_url}/detokenize" model_name = "qwen/qwen2.5-0.5b-instruct" input_text = "SGLang provides efficient tokenization endpoints." print_highlight(f"Original Input Text:\n'{input_text}'") # --- tokenize the input text --- tokenize_payload = { "model": model_name, "prompt": input_text, "add_special_tokens": False, } try: tokenize_response = requests.post(tokenize_url, json=tokenize_payload) tokenize_response.raise_for_status() tokenization_result = tokenize_response.json() token_ids = tokenization_result.get("tokens") if not token_ids: raise ValueError("Tokenization returned empty tokens.") print_highlight(f"\nTokenized Output (IDs):\n{token_ids}") print_highlight(f"Token Count: {tokenization_result.get('count')}") print_highlight(f"Max Model Length: {tokenization_result.get('max_model_len')}") # --- detokenize the obtained token IDs --- detokenize_payload = { "model": model_name, "tokens": token_ids, "skip_special_tokens": True, } detokenize_response = requests.post(detokenize_url, json=detokenize_payload) detokenize_response.raise_for_status() detokenization_result = detokenize_response.json() reconstructed_text = detokenization_result.get("text") print_highlight(f"\nDetokenized Output (Text):\n'{reconstructed_text}'") if input_text == reconstructed_text: print_highlight( "\nRound Trip Successful: Original and reconstructed text match." ) else: print_highlight( "\nRound Trip Mismatch: Original and reconstructed text differ." ) except requests.exceptions.RequestException as e: print_highlight(f"\nHTTP Request Error: {e}") except Exception as e: print_highlight(f"\nAn error occurred: {e}") ``` ```python Example theme={null} terminate_process(tokenizer_free_server_process) ``` # Offline Engine API Source: https://docs.sglang.io/docs/basic_usage/offline_engine_api SGLang provides a direct inference engine without the need for an HTTP server, especially for use cases where additional HTTP server adds unnecessary complexity or overhead. Here are two general use cases: * Offline Batch Inference * Custom Server on Top of the Engine This document focuses on the offline batch inference, demonstrating four different inference modes: * Non-streaming synchronous generation * Streaming synchronous generation * Non-streaming asynchronous generation * Streaming asynchronous generation Additionally, you can easily build a custom server on top of the SGLang offline engine. A detailed example working in a python script can be found in [custom\_server](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/custom_server.py). ## Nest Asyncio Note that if you want to use **Offline Engine** in ipython or some other nested loop code, you need to add the following code: ```python Example theme={null} import nest_asyncio nest_asyncio.apply() ``` ## Advanced Usage The engine supports [vlm inference](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py) as well as [extracting hidden states](https://github.com/sgl-project/sglang/tree/main/examples/runtime/hidden_states). Please see [the examples](https://github.com/sgl-project/sglang/tree/main/examples/runtime/engine) for further use cases. ## Offline Batch Inference SGLang offline engine supports batch inference with efficient scheduling. ```python Example theme={null} # launch the offline engine import asyncio import sglang as sgl import sglang.test.doc_patch from sglang.utils import async_stream_and_merge, stream_and_merge llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct") ``` ### Non-streaming Synchronous Generation ```python Example theme={null} prompts = [ "Hello, my name is", "The president of the United States is", "The capital of France is", "The future of AI is", ] sampling_params = {"temperature": 0.8, "top_p": 0.95} outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print("===============================") print(f"Prompt: {prompt}\nGenerated text: {output['text']}") ``` ### Streaming Synchronous Generation ```python Example theme={null} prompts = [ "Write a short, neutral self-introduction for a fictional character. Hello, my name is", "Provide a concise factual statement about France’s capital city. The capital of France is", "Explain possible future trends in artificial intelligence. The future of AI is", ] sampling_params = { "temperature": 0.2, "top_p": 0.9, } print("\n=== Testing synchronous streaming generation with overlap removal ===\n") for prompt in prompts: print(f"Prompt: {prompt}") merged_output = stream_and_merge(llm, prompt, sampling_params) print("Generated text:", merged_output) print() ``` ### Non-streaming Asynchronous Generation ```python Example theme={null} prompts = [ "Write a short, neutral self-introduction for a fictional character. Hello, my name is", "Provide a concise factual statement about France’s capital city. The capital of France is", "Explain possible future trends in artificial intelligence. The future of AI is", ] sampling_params = {"temperature": 0.8, "top_p": 0.95} print("\n=== Testing asynchronous batch generation ===") async def main(): outputs = await llm.async_generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print(f"\nPrompt: {prompt}") print(f"Generated text: {output['text']}") asyncio.run(main()) ``` ### Streaming Asynchronous Generation ```python Example theme={null} prompts = [ "Write a short, neutral self-introduction for a fictional character. Hello, my name is", "Provide a concise factual statement about France’s capital city. The capital of France is", "Explain possible future trends in artificial intelligence. The future of AI is", ] sampling_params = {"temperature": 0.8, "top_p": 0.95} print("\n=== Testing asynchronous streaming generation (no repeats) ===") async def main(): for prompt in prompts: print(f"\nPrompt: {prompt}") print("Generated text: ", end="", flush=True) # Replace direct calls to async_generate with our custom overlap-aware version async for cleaned_chunk in async_stream_and_merge(llm, prompt, sampling_params): print(cleaned_chunk, end="", flush=True) print() # New line after each prompt asyncio.run(main()) ``` ```python Example theme={null} llm.shutdown() ``` # Ollama-Compatible API Source: https://docs.sglang.io/docs/basic_usage/ollama_api SGLang provides Ollama API compatibility, allowing you to use the Ollama CLI and Python library with SGLang as the inference backend. ## Prerequisites ```bash Command theme={null} # Install the Ollama Python library (for Python client usage) pip install ollama ``` You don't need the Ollama server installed - SGLang acts as the backend. You only need the `ollama` CLI or Python library as the client. ## Endpoints
Endpoint Method Description
`/` GET, HEAD Health check for Ollama CLI
`/api/tags` GET List available models
`/api/chat` POST Chat completions (streaming & non-streaming)
`/api/generate` POST Text generation (streaming & non-streaming)
`/api/show` POST Model information
## Quick Start ### 1. Launch SGLang Server ```bash Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen2.5-1.5B-Instruct \ --port 30001 \ --host 0.0.0.0 ``` The model name used with `ollama run` must match exactly what you passed to `--model`. ### 2. Use Ollama CLI ```bash Command theme={null} # List available models OLLAMA_HOST=http://localhost:30001 ollama list # Interactive chat OLLAMA_HOST=http://localhost:30001 ollama run "Qwen/Qwen2.5-1.5B-Instruct" ``` If connecting to a remote server behind a firewall: ```bash Command theme={null} # SSH tunnel ssh -L 30001:localhost:30001 user@gpu-server -N & # Then use Ollama CLI as above OLLAMA_HOST=http://localhost:30001 ollama list ``` ### 3. Use Ollama Python Library ```python Example theme={null} import ollama client = ollama.Client(host='http://localhost:30001') # Non-streaming response = client.chat( model='Qwen/Qwen2.5-1.5B-Instruct', messages=[{'role': 'user', 'content': 'Hello!'}] ) print(response['message']['content']) # Streaming stream = client.chat( model='Qwen/Qwen2.5-1.5B-Instruct', messages=[{'role': 'user', 'content': 'Tell me a story'}], stream=True ) for chunk in stream: print(chunk['message']['content'], end='', flush=True) ``` ## Smart Router For intelligent routing between local Ollama (fast) and remote SGLang (powerful) using an LLM judge, see the [Smart Router documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/ollama/README.md). ## Summary
Component Purpose
**Ollama API** Familiar CLI/API that developers already know
**SGLang Backend** High-performance inference engine
**Smart Router** Intelligent routing - fast local for simple tasks, powerful remote for complex tasks
# OpenAI-Compatible APIs Source: https://docs.sglang.io/docs/basic_usage/openai_api Documentation for OpenAI-Compatible APIs * [Openai Api Completions](./openai_api_completions) * [Openai Api Vision](./openai_api_vision) * [Openai Api Embeddings](./openai_api_embeddings) For the Anthropic-compatible `/v1/messages` endpoint (including Claude Code integration), see [Anthropic-Compatible API](./anthropic_api). # OpenAI APIs - Completions Source: https://docs.sglang.io/docs/basic_usage/openai_api_completions SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/api-reference). This tutorial covers the following popular APIs: * `chat/completions` * `completions` Check out other tutorials to learn about [vision APIs](./openai_api_vision) for vision-language models and [embedding APIs](./openai_api_embeddings) for embedding models. ## Launch A Server Launch the server in your terminal and wait for it to initialize. ```python Example theme={null} from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process server_process, port = launch_server_cmd( "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning" ) wait_for_server(f"http://localhost:{port}") print(f"Server started on http://localhost:{port}") ``` ## Chat Completions ### Usage The server fully implements the OpenAI API. It will automatically apply the chat template specified in the Hugging Face tokenizer, if one is available. You can also specify a custom chat template with `--chat-template` when launching the server. ```python Example theme={null} import openai client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print_highlight(f"Response: {response}") ``` ### Model Thinking/Reasoning Support Some models support internal reasoning or thinking processes that can be exposed in the API response. SGLang provides unified support for various reasoning models through the `chat_template_kwargs` parameter and compatible reasoning parsers. #### Supported Models and Configuration
Model Family Chat Template Parameter Reasoning Parser Notes
DeepSeek-R1 (R1, R1-0528, R1-Distill) `enable_thinking` `--reasoning-parser deepseek-r1` Standard reasoning models
DeepSeek-V3.1 `thinking` `--reasoning-parser deepseek-v3` Hybrid model (thinking/non-thinking modes)
Qwen3 (standard) `enable_thinking` `--reasoning-parser qwen3` Hybrid model (thinking/non-thinking modes)
Qwen3-Thinking N/A (always enabled) `--reasoning-parser qwen3-thinking` Always generates reasoning
Kimi N/A (always enabled) `--reasoning-parser kimi` Kimi thinking models
Gpt-Oss N/A (always enabled) `--reasoning-parser gpt-oss` Gpt-Oss thinking models
#### Basic Usage To use reasoning output: 1. Launch the server with the appropriate reasoning parser. 2. Use the model-specific parameter in `chat_template_kwargs` when you need to override the model's chat-template default. 3. Optionally use `separate_reasoning: False` to keep reasoning in the response content instead of returning it separately (defaults to `True`). **Note for Qwen3-Thinking models:** These models always generate thinking content and do not support the `enable_thinking` parameter. Use `--reasoning-parser qwen3-thinking` or `--reasoning-parser qwen3` to parse the thinking content. #### Example: Qwen3 Models ```python Example theme={null} # Launch server: # python3 -m sglang.launch_server --model Qwen/Qwen3-4B --reasoning-parser qwen3 from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url=f"http://127.0.0.1:30000/v1", ) model = "Qwen/Qwen3-4B" messages = [{"role": "user", "content": "How many r's are in 'strawberry'?"}] response = client.chat.completions.create( model=model, messages=messages, extra_body={ "chat_template_kwargs": {"enable_thinking": True}, "separate_reasoning": True } ) print("Reasoning:", response.choices[0].message.reasoning_content) print("-"*100) print("Answer:", response.choices[0].message.content) ``` **ExampleOutput:** ```text Output theme={null} Reasoning: Okay, so the user is asking how many 'r's are in the word 'strawberry'. Let me think. First, I need to make sure I have the word spelled correctly. Strawberry... S-T-R-A-W-B-E-R-R-Y. Wait, is that right? Let me break it down. Starting with 'strawberry', let's write out the letters one by one. S, T, R, A, W, B, E, R, R, Y. Hmm, wait, that's 10 letters. Let me check again. S (1), T (2), R (3), A (4), W (5), B (6), E (7), R (8), R (9), Y (10). So the letters are S-T-R-A-W-B-E-R-R-Y. ... Therefore, the answer should be three R's in 'strawberry'. But I need to make sure I'm not counting any other letters as R. Let me check again. S, T, R, A, W, B, E, R, R, Y. No other R's. So three in total. Yeah, that seems right. ---------------------------------------------------------------------------------------------------- Answer: The word "strawberry" contains **three** letters 'r'. Here's the breakdown: 1. **S-T-R-A-W-B-E-R-R-Y** - The **third letter** is 'R'. - The **eighth and ninth letters** are also 'R's. Thus, the total count is **3**. **Answer:** 3. ``` Setting `"enable_thinking": False` disables reasoning for Qwen3. Omitting the parameter follows the model's chat-template default; Qwen3 enables thinking by default. Qwen3-Thinking models always generate reasoning content and do not support the `enable_thinking` parameter. #### Set a Default for OpenAI Chat Completions If most OpenAI chat-completion requests should use the same chat-template mode, set a default when launching the server: ```bash Command theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3-4B \ --reasoning-parser qwen3 \ --default-chat-template-kwargs '{"enable_thinking": false}' ``` `--default-chat-template-kwargs` accepts a JSON object and merges it into each OpenAI chat-completion request's `chat_template_kwargs`. For ordinary template keys such as `enable_thinking`, the precedence is: 1. An explicit value in the request's `chat_template_kwargs` 2. The value in `--default-chat-template-kwargs` 3. The model's chat-template default `reasoning_effort` follows a separate normalization path, so the precedence list above does not apply to it. Configure `reasoning_effort` in either the server defaults or individual requests, not both. The accepted keys are model-specific. For example, Qwen3 uses `enable_thinking`, while other model families may use `thinking` or may not expose a reasoning toggle. The reasoning parser controls how reasoning tokens are separated from the response; it does not by itself enable reasoning in the chat template. #### Logit Bias Support SGLang supports the `logit_bias` parameter for both chat completions and completions APIs. This parameter allows you to modify the likelihood of specific tokens being generated by adding bias values to their logits. The bias values can range from -100 to 100, where: * **Positive values** (0 to 100) increase the likelihood of the token being selected * **Negative values** (-100 to 0) decrease the likelihood of the token being selected * **-100** effectively prevents the token from being generated The `logit_bias` parameter accepts a dictionary where keys are token IDs (as strings) and values are the bias amounts (as floats). #### Getting Token IDs To use `logit_bias` effectively, you need to know the token IDs for the words you want to bias. Here's how to get token IDs: ```python Example theme={null} # Get tokenizer to find token IDs import tiktoken # For OpenAI models, use the appropriate encoding tokenizer = tiktoken.encoding_for_model("gpt-3.5-turbo") # or your model # Get token IDs for specific words word = "sunny" token_ids = tokenizer.encode(word) print(f"Token IDs for '{word}': {token_ids}") # For SGLang models, you can access the tokenizer through the client # and get token IDs for bias ``` **Important:** The `logit_bias` parameter uses token IDs as string keys, not the actual words. #### Example: DeepSeek-V3 Models DeepSeek-V3 models support thinking mode through the `thinking` parameter: ```python Example theme={null} # Launch server: # python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.1 --tp 8 --reasoning-parser deepseek-v3 from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url=f"http://127.0.0.1:30000/v1", ) model = "deepseek-ai/DeepSeek-V3.1" messages = [{"role": "user", "content": "How many r's are in 'strawberry'?"}] response = client.chat.completions.create( model=model, messages=messages, extra_body={ "chat_template_kwargs": {"thinking": True}, "separate_reasoning": True } ) print("Reasoning:", response.choices[0].message.reasoning_content) print("-"*100) print("Answer:", response.choices[0].message.content) ``` **Example Output:** ```text Output theme={null} Reasoning: First, the question is: "How many r's are in 'strawberry'?" I need to count the number of times the letter 'r' appears in the word "strawberry". Let me write out the word: S-T-R-A-W-B-E-R-R-Y. Now, I'll go through each letter and count the 'r's. ... So, I have three 'r's in "strawberry". I should double-check. The word is spelled S-T-R-A-W-B-E-R-R-Y. The letters are at positions: 3, 8, and 9 are 'r's. Yes, that's correct. Therefore, the answer should be 3. ---------------------------------------------------------------------------------------------------- Answer: The word "strawberry" contains **3** instances of the letter "r". Here's a breakdown for clarity: - The word is spelled: S-T-R-A-W-B-E-R-R-Y - The "r" appears at the 3rd, 8th, and 9th positions. ``` DeepSeek-V3 models use the `thinking` parameter (not `enable_thinking`) to control reasoning output. ```python Example theme={null} # Example with logit_bias parameter # Note: You need to get the actual token IDs from your tokenizer # For demonstration, we'll use some example token IDs response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[ {"role": "user", "content": "Complete this sentence: The weather today is"} ], temperature=0.7, max_tokens=20, logit_bias={ "12345": 50, # Increase likelihood of token ID 12345 "67890": -50, # Decrease likelihood of token ID 67890 "11111": 25, # Slightly increase likelihood of token ID 11111 }, ) print_highlight(f"Response with logit bias: {response.choices[0].message.content}") ``` ### Parameters The chat completions API accepts OpenAI Chat Completions API's parameters. Refer to [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create) for more details. SGLang extends the standard API with the `extra_body` parameter, allowing for additional customization. One key option within `extra_body` is `chat_template_kwargs`, which can be used to pass arguments to the chat template processor. ```python Example theme={null} response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[ { "role": "system", "content": "You are a knowledgeable historian who provides concise responses.", }, {"role": "user", "content": "Tell me about ancient Rome"}, { "role": "assistant", "content": "Ancient Rome was a civilization centered in Italy.", }, {"role": "user", "content": "What were their major achievements?"}, ], temperature=0.3, # Lower temperature for more focused responses max_tokens=128, # Reasonable length for a concise response top_p=0.95, # Slightly higher for better fluency presence_penalty=0.2, # Mild penalty to avoid repetition frequency_penalty=0.2, # Mild penalty for more natural language n=1, # Single response is usually more stable seed=42, # Keep for reproducibility ) print_highlight(response.choices[0].message.content) ``` Streaming mode is also supported. #### Logit Bias Support The completions API also supports the `logit_bias` parameter with the same functionality as described in the chat completions section above. ```python Example theme={null} stream = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[{"role": "user", "content": "Say this is a test"}], stream=True, ) for chunk in stream: if chunk.choices[0].delta.content is not None: print(chunk.choices[0].delta.content, end="") ``` #### Returning Routed Experts (MoE Models) For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. By default, `routed_experts` is returned in the response-level `sglext` object as base64-encoded int32 expert IDs in a flattened array with logical shape `[num_tokens, num_layers, top_k]`. For non-streaming chat responses with `return_meta_info: true`, `routed_experts` is returned only in each choice's `meta_info` object and omitted from response-level `sglext` to avoid duplicating the routing data. Other requested extension fields, such as `cached_tokens_details` and `spec_tokens_details`, remain in response-level `sglext`. By default, routed experts cover `[0, seqlen - 1)`, the full available sequence, because RL workflows need them for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that causes bottlenecks. ```python Example theme={null} # Example with logit_bias parameter for completions API # Note: You need to get the actual token IDs from your tokenizer # For demonstration, we'll use some example token IDs response = client.completions.create( model="qwen/qwen2.5-0.5b-instruct", prompt="The best programming language for AI is", temperature=0.7, max_tokens=20, logit_bias={ "12345": 75, # Strongly favor token ID 12345 "67890": -100, # Completely avoid token ID 67890 "11111": -25, # Slightly discourage token ID 11111 }, ) print_highlight(f"Response with logit bias: {response.choices[0].text}") ``` ## Completions ### Usage Completions API is similar to Chat Completions API, but without the `messages` parameter or chat templates. ```python Example theme={null} response = client.completions.create( model="qwen/qwen2.5-0.5b-instruct", prompt="List 3 countries and their capitals.", temperature=0, max_tokens=64, n=1, stop=None, ) print_highlight(f"Response: {response}") ``` ### Parameters The completions API accepts OpenAI Completions API's parameters. Refer to [OpenAI Completions API](https://platform.openai.com/docs/api-reference/completions/create) for more details. Here is an example of a detailed completions request: ```python Example theme={null} response = client.completions.create( model="qwen/qwen2.5-0.5b-instruct", prompt="Write a short story about a space explorer.", temperature=0.7, # Moderate temperature for creative writing max_tokens=150, # Longer response for a story top_p=0.9, # Balanced diversity in word choice stop=["\n\n", "THE END"], # Multiple stop sequences presence_penalty=0.3, # Encourage novel elements frequency_penalty=0.3, # Reduce repetitive phrases n=1, # Generate one completion seed=123, # For reproducible results ) print_highlight(f"Response: {response}") ``` #### Returning Routed Experts (MoE Models) For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`. By default this returns `[0, seqlen - 1)`, the full available sequence, because RL workflows need routed experts for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks. ## Structured Outputs (JSON, Regex, EBNF) For OpenAI compatible structured outputs API, refer to [Structured Outputs](../advanced_features/structured_outputs) for more details. ## Using LoRA Adapters SGLang supports LoRA (Low-Rank Adaptation) adapters with OpenAI-compatible APIs. You can specify which adapter to use directly in the `model` parameter using the `base-model:adapter-name` syntax. **Server Setup:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path qwen/qwen2.5-0.5b-instruct \ --enable-lora \ --lora-paths adapter_a=/path/to/adapter_a adapter_b=/path/to/adapter_b ``` For more details on LoRA serving configuration, see the [LoRA documentation](../advanced_features/lora). **API Call:** (Recommended) Use the `model:adapter` syntax to specify which adapter to use: ```python Example theme={null} response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct:adapter_a", # ← base-model:adapter-name messages=[{"role": "user", "content": "Convert to SQL: show all users"}], max_tokens=50, ) ``` **Backward Compatible: Using `extra_body`** The old `extra_body` method is still supported for backward compatibility: ```python Example theme={null} # Backward compatible method response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[{"role": "user", "content": "Convert to SQL: show all users"}], extra_body={"lora_path": "adapter_a"}, # ← old method max_tokens=50, ) ``` **Note:** When both `model:adapter` and `extra_body["lora_path"]` are specified, the `model:adapter` syntax takes precedence. ```python Example theme={null} terminate_process(server_process) ``` # OpenAI APIs - Embedding Source: https://docs.sglang.io/docs/basic_usage/openai_api_embeddings SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/embeddings). This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the [corresponding overview page](../supported-models) ## Launch A Server Launch the server in your terminal and wait for it to initialize. Native encoder embedding architectures and `google/embeddinggemma-300m` are detected automatically. Decoder-style embedding models still require `--is-embedding`. ```python Example theme={null} from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process embedding_process, port = launch_server_cmd( """ sglang serve --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \ --is-embedding --log-level warning """ ) wait_for_server(f"http://localhost:{port}") ``` ## Using cURL ```python Example theme={null} import subprocess, json text = "Once upon a time" curl_text = f"""curl -s http://localhost:{port}/v1/embeddings \ -H "Content-Type: application/json" \ -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": "{text}"}}'""" result = subprocess.check_output(curl_text, shell=True) print(result) text_embedding = json.loads(result)["data"][0]["embedding"] print_highlight(f"Text embedding (first 10): {text_embedding[:10]}") ``` ## Using Python Requests ```python Example theme={null} import requests text = "Once upon a time" response = requests.post( f"http://localhost:{port}/v1/embeddings", json={"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": text}, ) text_embedding = response.json()["data"][0]["embedding"] print_highlight(f"Text embedding (first 10): {text_embedding[:10]}") ``` ## Using OpenAI Python Client ```python Example theme={null} import openai client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") # Text embedding example response = client.embeddings.create( model="Alibaba-NLP/gte-Qwen2-1.5B-instruct", input=text, ) embedding = response.data[0].embedding[:10] print_highlight(f"Text embedding (first 10): {embedding}") ``` ## Using Input IDs SGLang also supports `input_ids` as input to get the embedding. ```python Example theme={null} import json import os from transformers import AutoTokenizer os.environ["TOKENIZERS_PARALLELISM"] = "false" tokenizer = AutoTokenizer.from_pretrained("Alibaba-NLP/gte-Qwen2-1.5B-instruct") input_ids = tokenizer.encode(text) curl_ids = f"""curl -s http://localhost:{port}/v1/embeddings \ -H "Content-Type: application/json" \ -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": {json.dumps(input_ids)}}}'""" input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))["data"][ 0 ]["embedding"] print_highlight(f"Input IDs embedding (first 10): {input_ids_embedding[:10]}") ``` ## Compact Base64 Responses Set `encoding_format` to `base64` when JSON arrays would dominate response size. The encoded value contains little-endian FP32 values and can be decoded by OpenAI-compatible clients. ```python Example theme={null} response = requests.post( f"http://localhost:{port}/v1/embeddings", json={ "model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": text, "encoding_format": "base64", }, ) base64_embedding = response.json()["data"][0]["embedding"] print_highlight(f"Base64 embedding: {base64_embedding[:20]}...") ``` ```python Example theme={null} terminate_process(embedding_process) ``` ## Multi-Modal Embedding Model Please refer to [Multi-Modal Embedding Model](../supported-models) # OpenAI APIs - Vision Source: https://docs.sglang.io/docs/basic_usage/openai_api_vision SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/vision). This tutorial covers the vision APIs for vision language models. SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and [more](../supported-models/multimodal_language_models). As an alternative to the OpenAI API, you can also use the [SGLang offline engine](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py). ## Launch A Server Launch the server in your terminal and wait for it to initialize. ```python Example theme={null} from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process example_image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png" logo_image_url = ( "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png" ) vision_process, port = launch_server_cmd(""" python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --log-level warning """) wait_for_server(f"http://localhost:{port}", process=vision_process) ``` ## Using cURL Once the server is up, you can send test requests using curl or requests. ```python Example theme={null} import subprocess curl_command = f""" curl -s http://localhost:{port}/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{{ "model": "Qwen/Qwen2.5-VL-7B-Instruct", "messages": [ {{ "role": "user", "content": [ {{ "type": "text", "text": "What’s in this image?" }}, {{ "type": "image_url", "image_url": {{ "url": "{example_image_url}" }} }} ] }} ], "max_tokens": 300 }}' """ response = subprocess.check_output(curl_command, shell=True).decode() print_highlight(response) response = subprocess.check_output(curl_command, shell=True).decode() print_highlight(response) ``` ## Using Python Requests ```python Example theme={null} import requests url = f"http://localhost:{port}/v1/chat/completions" data = { "model": "Qwen/Qwen2.5-VL-7B-Instruct", "messages": [ { "role": "user", "content": [ {"type": "text", "text": "What’s in this image?"}, { "type": "image_url", "image_url": {"url": example_image_url}, }, ], } ], "max_tokens": 300, } response = requests.post(url, json=data) print_highlight(response.text) ``` ## Using OpenAI Python Client ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None") response = client.chat.completions.create( model="Qwen/Qwen2.5-VL-7B-Instruct", messages=[ { "role": "user", "content": [ { "type": "text", "text": "What is in this image?", }, { "type": "image_url", "image_url": {"url": example_image_url}, }, ], } ], max_tokens=300, ) print_highlight(response.choices[0].message.content) ``` ## Multiple-Image Inputs The server also supports multiple images and interleaved text and images if the model supports it. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None") response = client.chat.completions.create( model="Qwen/Qwen2.5-VL-7B-Instruct", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": example_image_url, }, }, { "type": "image_url", "image_url": { "url": logo_image_url, }, }, { "type": "text", "text": "I have two very different images. They are not related at all. " "Please describe the first image in one sentence, and then describe the second image in another sentence.", }, ], } ], temperature=0, ) print_highlight(response.choices[0].message.content) ``` ```python Example theme={null} terminate_process(vision_process) ``` # Basic Usage Source: https://docs.sglang.io/docs/basic_usage/overview Core APIs and common usage patterns for SGLang. * [OpenAI-Compatible APIs](./openai_api_completions) — Chat completions, vision, and embeddings * [Anthropic-Compatible API](./anthropic_api) — `/v1/messages`, including Claude Code integration * [Ollama API](./ollama_api) * [Offline Engine API](./offline_engine_api) * [Native API](./native_api) * [Sampling Parameters](./sampling_params) * [Popular Model Usage](/cookbook/autoregressive/intro) — DeepSeek, GLM, Qwen, Llama, and more # Sampling Parameters Source: https://docs.sglang.io/docs/basic_usage/sampling_params This doc describes the sampling parameters of the SGLang Runtime. It is the low-level endpoint of the runtime. If you want a high-level endpoint that can automatically handle chat templates, consider using the [OpenAI Compatible API](./openai_api_completions). ## `/generate` Endpoint The `/generate` endpoint accepts the following parameters in JSON format. For detailed usage, see the [native API doc](./native_api). The object is defined at `io_struct.py::GenerateReqInput`. You can also read the source code to find more arguments and docs.
Argument Type/Default Description
text `Optional[Union[List[str], str]] = None` The input prompt. Can be a single prompt or a batch of prompts.
input\_ids `Optional[Union[List[List[int]], List[int]]] = None` The token IDs for text; one can specify either text or input\_ids.
input\_embeds `Optional[Union[List[List[List[float]]], List[List[float]]]] = None` The embeddings for input\_ids; one can specify either text, input\_ids, or input\_embeds.
image\_data `Optional[Union[List[List[ImageDataItem]], List[ImageDataItem], ImageDataItem]] = None` The image input. Supports three formats: (1) **Raw images**: PIL Image, file path, URL, or base64 string; (2) **Processor output**: Dict with `format: "processor_output"` containing HuggingFace processor outputs; (3) **Precomputed embeddings**: Dict with `format: "precomputed_embedding"` and `feature` containing pre-calculated visual embeddings. Can be a single image, list of images, or list of lists of images. See [Multimodal Input Formats](#multimodal) for details.
audio\_data `Optional[Union[List[AudioDataItem], AudioDataItem]] = None` The audio input. Can be a file name, URL, or base64 encoded string.
sampling\_params `Optional[Union[List[Dict], Dict]] = None` The sampling parameters as described in the sections below.
rid `Optional[Union[List[str], str]] = None` The request ID.
return\_logprob `Optional[Union[List[bool], bool]] = None` Whether to return log probabilities for tokens.
logprob\_start\_len `Optional[Union[List[int], int]] = None` If return\_logprob, the start location in the prompt for returning logprobs. Default is "-1", which returns logprobs for output tokens only.
top\_logprobs\_num `Optional[Union[List[int], int]] = None` If return\_logprob, the number of top logprobs to return at each position.
token\_ids\_logprob `Optional[Union[List[List[int]], List[int]]] = None` If return\_logprob, the token IDs to return logprob for.
return\_text\_in\_logprobs `bool = False` Whether to detokenize tokens in text in the returned logprobs.
stream `bool = False` Whether to stream output.
lora\_path `Optional[Union[List[Optional[str]], Optional[str]]] = None` The path to the LoRA.
custom\_logit\_processor `Optional[Union[List[Optional[str]], str]] = None` Custom logit processor for advanced sampling control. Must be a serialized instance of `CustomLogitProcessor` using its `to_str()` method. For usage see below.
return\_hidden\_states `Union[List[bool], bool] = False` Whether to return hidden states.
return\_routed\_experts `bool = False` Whether to return routed experts for MoE models. Requires `--enable-return-routed-experts` server flag. With the default `routed_experts_start_len=0`, returns the full available sequence `[0, seqlen - 1)` because RL workflows need routed experts for the full sequence. The result is base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`.
routed\_experts\_start\_len `int = 0` If `return_routed_experts`, the absolute start position for returned routed-experts rows. `0` preserves the default full sequence; set it to an accumulated prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks. Must be in `[0, prompt_tokens]`.
## Sampling parameters The object is defined at `sampling_params.py::SamplingParams`. You can also read the source code to find more arguments and docs. ### Note on defaults By default, SGLang initializes several sampling parameters from the model's `generation_config.json` (when the server is launched with `--sampling-defaults model`, which is the default). To use SGLang/OpenAI constant defaults instead, start the server with `--sampling-defaults openai`. You can always override any parameter per request via `sampling_params`. ```bash Command theme={null} # Use model-provided defaults from generation_config.json (default behavior) python -m sglang.launch_server --model-path --sampling-defaults model # Use SGLang/OpenAI constant defaults instead python -m sglang.launch_server --model-path --sampling-defaults openai ``` ### Core parameters
Argument Type/Default Description
max\_new\_tokens `int = 128` The maximum output length measured in tokens.
stop `Optional[Union[str, List[str]]] = None` One or multiple [stop words](https://platform.openai.com/docs/api-reference/chat/create#chat-create-stop). Generation will stop if one of these words is sampled.
stop\_token\_ids `Optional[List[int]] = None` Provide stop words in the form of token IDs. Generation will stop if one of these token IDs is sampled.
stop\_regex `Optional[Union[str, List[str]]] = None` Stop when hitting any of the regex patterns in this list
temperature `float (model default; fallback 1.0)` [Temperature](https://platform.openai.com/docs/api-reference/chat/create#chat-create-temperature) when sampling the next token. `temperature = 0` corresponds to greedy sampling, a higher temperature leads to more diversity.
top\_p `float (model default; fallback 1.0)` [Top-p](https://platform.openai.com/docs/api-reference/chat/create#chat-create-top_p) selects tokens from the smallest sorted set whose cumulative probability exceeds `top_p`. When `top_p = 1`, this reduces to unrestricted sampling from all tokens.
top\_k `int (model default; fallback -1)` [Top-k](https://developer.nvidia.com/blog/how-to-get-better-outputs-from-your-large-language-model/#predictability_vs_creativity) randomly selects from the `k` highest-probability tokens.
min\_p `float (model default; fallback 0.0)` [Min-p](https://github.com/huggingface/transformers/issues/27670) samples from tokens with probability larger than `min_p * highest_token_probability`.
### Penalizers
Argument Type/Default Description
frequency\_penalty `float = 0.0` Penalizes tokens based on their frequency in generation so far. Must be between `-2` and `2` where negative numbers encourage repeatment of tokens and positive number encourages sampling of new tokens. The scaling of penalization grows linearly with each appearance of a token.
presence\_penalty `float = 0.0` Penalizes tokens if they appeared in the generation so far. Must be between `-2` and `2` where negative numbers encourage repeatment of tokens and positive number encourages sampling of new tokens. The scaling of the penalization is constant if a token occurred.
repetition\_penalty `float = 1.0` Scales the logits of previously generated tokens to discourage (values > 1) or encourage (values \< 1) repetition. Valid range is `(0, 2]`; `1.0` leaves probabilities unchanged.
min\_new\_tokens `int = 0` Forces the model to generate at least `min_new_tokens` until a stop word or EOS token is sampled. Note that this might lead to unintended behavior, for example, if the distribution is highly skewed towards these tokens.
### Constrained decoding Please refer to our dedicated guide on [constrained decoding](../advanced_features/structured_outputs) for the following parameters.
Argument Type/Default Description
json\_schema `Optional[str] = None` JSON schema for structured outputs.
regex `Optional[str] = None` Regex for structured outputs.
ebnf `Optional[str] = None` EBNF for structured outputs.
structural\_tag `Optional[str] = None` The structal tag for structured outputs.
### Other options
Argument Type/Default Description
n `int = 1` Specifies the number of output sequences to generate per request. (Generating multiple outputs in one request (n > 1) is discouraged; repeating the same prompts several times offers better control and efficiency.)
ignore\_eos `bool = False` Don't stop generation when EOS token is sampled.
skip\_special\_tokens `bool = True` Remove special tokens during decoding.
spaces\_between\_special\_tokens `bool = True` Whether or not to add spaces between special tokens during detokenization.
no\_stop\_trim `bool = False` Don't trim stop words or EOS token from the generated text.
custom\_params `Optional[List[Optional[Dict[str, Any]]]] = None` Used when employing `CustomLogitProcessor`. For usage, see below.
## Examples ### Normal Launch a server: ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000 ``` Send a request: ```python Example theme={null} import requests response = requests.post( "http://localhost:30000/generate", json={ "text": "The capital of France is", "sampling_params": { "temperature": 0, "max_new_tokens": 32, }, }, ) print(response.json()) ``` Detailed example in [send request](./send_request). ### Streaming Send a request and stream the output: ```python Example theme={null} import requests, json response = requests.post( "http://localhost:30000/generate", json={ "text": "The capital of France is", "sampling_params": { "temperature": 0, "max_new_tokens": 32, }, "stream": True, }, stream=True, ) prev = 0 for chunk in response.iter_lines(decode_unicode=False): chunk = chunk.decode("utf-8") if chunk and chunk.startswith("data:"): if chunk == "data: [DONE]": break data = json.loads(chunk[5:].strip("\n")) output = data["text"].strip() print(output[prev:], end="", flush=True) prev = len(output) print("") ``` Detailed example in [openai compatible api](./openai_api_completions). ### Multimodal Launch a server: ```bash Command theme={null} python3 -m sglang.launch_server --model-path lmms-lab/llava-onevision-qwen2-7b-ov ``` Download an image: ```bash Command theme={null} curl -o example_image.png -L https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true ``` Send a request: ```python Example theme={null} import requests response = requests.post( "http://localhost:30000/generate", json={ "text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n" "<|im_start|>user\n\nDescribe this image in a very short sentence.<|im_end|>\n" "<|im_start|>assistant\n", "image_data": "example_image.png", "sampling_params": { "temperature": 0, "max_new_tokens": 32, }, }, ) print(response.json()) ``` The `image_data` can be a file name, a URL, or a base64 encoded string. See also `python/sglang/srt/utils.py:load_image`. Streaming is supported in a similar manner as [above](#streaming). Detailed example in [OpenAI API Vision](./openai_api_vision). ### Structured Outputs (JSON, Regex, EBNF) You can specify a JSON schema, regular expression or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request. SGLang supports two grammar backends: * [XGrammar](https://github.com/mlc-ai/xgrammar) (default): Supports JSON schema, regular expression, and EBNF constraints. * XGrammar currently uses the [GGML BNF format](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md). * [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints. If instead you want to initialize the Outlines backend, you can use `--grammar-backend outlines` flag: ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --port 30000 --host 0.0.0.0 --grammar-backend [xgrammar|outlines] # xgrammar or outlines (default: xgrammar) ``` ```python Example theme={null} import json import requests json_schema = json.dumps({ "type": "object", "properties": { "name": {"type": "string", "pattern": "^[\\w]+$"}, "population": {"type": "integer"}, }, "required": ["name", "population"], }) # JSON (works with both Outlines and XGrammar) response = requests.post( "http://localhost:30000/generate", json={ "text": "Here is the information of the capital of France in the JSON format.\n", "sampling_params": { "temperature": 0, "max_new_tokens": 64, "json_schema": json_schema, }, }, ) print(response.json()) # Regular expression (Outlines backend only) response = requests.post( "http://localhost:30000/generate", json={ "text": "Paris is the capital of", "sampling_params": { "temperature": 0, "max_new_tokens": 64, "regex": "(France|England)", }, }, ) print(response.json()) # EBNF (XGrammar backend only) response = requests.post( "http://localhost:30000/generate", json={ "text": "Write a greeting.", "sampling_params": { "temperature": 0, "max_new_tokens": 64, "ebnf": 'root ::= "Hello" | "Hi" | "Hey"', }, }, ) print(response.json()) ``` Detailed example in [structured outputs](../advanced_features/structured_outputs). ### Custom logit processor Launch a server with `--enable-custom-logit-processor` flag on. ```bash Command theme={null} python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3-8B-Instruct \ --port 30000 \ --enable-custom-logit-processor ``` Define a custom logit processor that will always sample a specific token id. ```python Example theme={null} from sglang.srt.sampling.custom_logit_processor import CustomLogitProcessor class DeterministicLogitProcessor(CustomLogitProcessor): """A dummy logit processor that changes the logits to always sample the given token id. """ def __call__(self, logits, custom_param_list): # Check that the number of logits matches the number of custom parameters assert logits.shape[0] == len(custom_param_list) key = "token_id" for i, param_dict in enumerate(custom_param_list): # Mask all other tokens logits[i, :] = -float("inf") # Assign highest probability to the specified token logits[i, param_dict[key]] = 0.0 return logits ``` Send a request: ```python Example theme={null} import requests response = requests.post( "http://localhost:30000/generate", json={ "text": "The capital of France is", "custom_logit_processor": DeterministicLogitProcessor().to_str(), "sampling_params": { "temperature": 0.0, "max_new_tokens": 32, "custom_params": {"token_id": 5}, }, }, ) print(response.json()) ``` Send an OpenAI chat completion request: ```python Example theme={null} import openai from sglang.utils import print_highlight client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="meta-llama/Meta-Llama-3-8B-Instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0.0, max_tokens=32, extra_body={ "custom_logit_processor": DeterministicLogitProcessor().to_str(), "custom_params": {"token_id": 5}, }, ) print_highlight(f"Response: {response}") ``` # Tutorial: Sending a request Source: https://docs.sglang.io/docs/basic_usage/send_request This notebook provides a quick-start guide to use SGLang in chat completions after installation. Once your server is running, API documentation is available at `http://localhost:30000/docs` (Swagger UI), `http://localhost:30000/redoc` (ReDoc), or `http://localhost:30000/openapi.json` (OpenAPI spec, useful for AI agents). Replace `30000` with your port if using a different one. * For Vision Language Models, see [OpenAI APIs - Vision](./openai_api_vision). * For Embedding Models, see [OpenAI APIs - Embedding](./openai_api_embeddings) and [Encode (embedding model)](./native_api#encode-embedding-model). * For Reward Models, see [Classify (reward model)](./native_api#classify-reward-model). ## Launch A Server ```python Example theme={null} from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, terminate_process # This is equivalent to running the following command in your terminal # python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 server_process, port = launch_server_cmd( """ python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \ --host 0.0.0.0 --log-level warning """ ) wait_for_server(f"http://localhost:{port}") ``` ## Using cURL ```python Example theme={null} import subprocess, json curl_command = f""" curl -s http://localhost:{port}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{{"model": "qwen/qwen2.5-0.5b-instruct", "messages": [{{"role": "user", "content": "What is the capital of France?"}}]}}' """ response = json.loads(subprocess.check_output(curl_command, shell=True)) print(response) ``` ## Using Python Requests ```python Example theme={null} import requests url = f"http://localhost:{port}/v1/chat/completions" data = { "model": "qwen/qwen2.5-0.5b-instruct", "messages": [{"role": "user", "content": "What is the capital of France?"}], } response = requests.post(url, json=data) print(response.json()) ``` ## Using OpenAI Python Client ```python Example theme={null} import openai client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response) ``` ### Streaming ```python Example theme={null} import openai client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") # Use stream=True for streaming responses response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, stream=True, ) # Handle the streaming output for chunk in response: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) ``` ## Using Native Generation APIs You can also use the native `/generate` endpoint with requests, which provides more flexibility. An API reference is available at [Sampling Parameters](./sampling_params). ```python Example theme={null} import requests response = requests.post( f"http://localhost:{port}/generate", json={ "text": "The capital of France is", "sampling_params": { "temperature": 0, "max_new_tokens": 32, }, }, ) print(response.json()) ``` ### Streaming ```python Example theme={null} import requests, json response = requests.post( f"http://localhost:{port}/generate", json={ "text": "The capital of France is", "sampling_params": { "temperature": 0, "max_new_tokens": 32, }, "stream": True, }, stream=True, ) prev = 0 for chunk in response.iter_lines(decode_unicode=False): chunk = chunk.decode("utf-8") if chunk and chunk.startswith("data:"): if chunk == "data: [DONE]": break data = json.loads(chunk[5:].strip("\n")) output = data["text"] print(output[prev:], end="", flush=True) prev = len(output) ``` ```python Example theme={null} terminate_process(server_process) ``` # Bench Serving Guide Source: https://docs.sglang.io/docs/developer_guide/bench_serving This guide explains how to benchmark online serving throughput and latency using `python -m sglang.bench_serving`. It supports multiple inference backends via OpenAI-compatible and native endpoints, and produces both console metrics and optional JSONL outputs. ### What it does * Generates synthetic or dataset-driven prompts and submits them to a target serving endpoint * Measures throughput, time-to-first-token (TTFT), inter-token latency (ITL), per-request end-to-end latency, and more * Supports streaming or non-streaming modes, rate control, and concurrency limits ### Supported backends and endpoints * `sglang` / `sglang-native`: `POST /generate` * `sglang-oai`, `vllm`, `lmdeploy`: `POST /v1/completions` * `sglang-oai-chat`, `vllm-chat`, `lmdeploy-chat`: `POST /v1/chat/completions` * `sglang-embedding`, `vllm-embedding`: `POST /v1/embeddings` * `trt` (TensorRT-LLM): `POST /v2/models/ensemble/generate_stream` * `gserver`: Custom server (Not Implemented yet in this script) * `truss`: `POST /v1/models/model:predict` If `--base-url` is provided, requests are sent to it. Otherwise, `--host` and `--port` are used. When `--model` is not provided, the script will attempt to query `GET /v1/models` for an available model ID (OpenAI-compatible endpoints). ### Prerequisites * Python 3.10+ * Dependencies typically used by this script: `aiohttp`, `numpy`, `requests`, `tqdm`, `transformers`, and for some datasets `datasets`, `pillow`, `pybase64`. Install as needed. * An inference server running and reachable via the endpoints above * If your server requires authentication, set environment variable `OPENAI_API_KEY` (used as `Authorization: Bearer `) ### Quick start Run a basic benchmark against an sglang server exposing `/generate`: ```bash Command theme={null} python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct ``` ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --num-prompts 1000 \ --model meta-llama/Llama-3.1-8B-Instruct ``` Or, using an OpenAI-compatible endpoint (completions): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend vllm \ --base-url http://127.0.0.1:8000 \ --num-prompts 1000 \ --model meta-llama/Llama-3.1-8B-Instruct ``` ### Fair embedding comparison Use the two embedding backends with the same model, tokenizer, input length, prompt count, and concurrency. The benchmark reports input-token throughput and end-to-end latency; embeddings have no decode-side TTFT or TPOT. ```bash Command theme={null} # Start either server on the same hardware and precision, then run one at a time. python3 -m sglang.bench_serving \ --backend sglang-embedding \ --model google/embeddinggemma-300m \ --dataset-name random \ --random-input-len 2048 \ --num-prompts 300 \ --max-concurrency 64 \ --warmup-requests 3 \ --flush-cache ``` ```bash Command theme={null} # vLLM's cache reset endpoint requires VLLM_SERVER_DEV_MODE=1 at server startup. python3 -m sglang.bench_serving \ --backend vllm-embedding \ --model google/embeddinggemma-300m \ --dataset-name random \ --random-input-len 2048 \ --num-prompts 300 \ --max-concurrency 64 \ --warmup-requests 3 \ --flush-cache ``` `--flush-cache` calls `/flush_cache` for SGLang and `/reset_prefix_cache` for vLLM after warmup. For vLLM, start the server with `VLLM_SERVER_DEV_MODE=1`; without it the benchmark fails loudly rather than accidentally measuring warm-cache performance. ### Datasets Select with `--dataset-name`: * `sharegpt` (default): loads ShareGPT-style pairs; optionally restrict with `--sharegpt-context-len` and override outputs with `--sharegpt-output-len` * `random`: random text lengths; sampled from ShareGPT token space * `random-ids`: random token ids (can lead to gibberish) * `image`: generates images and wraps them in chat messages; supports custom resolutions, multiple formats, and different content types * `generated-shared-prefix`: synthetic dataset with shared long system prompts and short questions * `mmmu`: samples from MMMU (Math split) and includes images * `speed-bench`: [SPEED-Bench](https://huggingface.co/datasets/nvidia/SPEED-Bench) (**SPEculative Evaluation Dataset**) — a unified benchmark for evaluating [Speculative Decoding (SD)](https://arxiv.org/abs/2604.09557) algorithms. Uses the Throughput split, which provides fixed-length input sequences (1K–32K tokens) grouped into three output-entropy categories (`low_entropy`, `mixed`, `high_entropy`). Requires a pre-downloaded JSONL file passed via `--dataset-path`. * `agentic-trace`: replays pre-built multi-turn agentic traces (e.g. OpenHands / SWE-smith). Each conversation is replayed round by round, feeding the server's real assistant reply back into the next round's history. Requires a chat backend (`--backend sglang-oai-chat`) and a trace JSON passed via `--dataset-path`. Common dataset flags: * `--num-prompts N`: number of requests * `--random-input-len`, `--random-output-len`, `--random-range-ratio`: for random/random-ids/image * `--image-count`: Number of images per request (for `image` dataset). * `--apply-chat-template`: apply tokenizer chat template when constructing prompts * `--dataset-path PATH`: file path for ShareGPT json; if blank and missing, it will be downloaded and cached Generated Shared Prefix flags (for `generated-shared-prefix`): * `--gsp-num-groups` * `--gsp-prompts-per-group` * `--gsp-system-prompt-len` * `--gsp-question-len` * `--gsp-output-len` * `--gsp-group-distribution {uniform,zipf}`: per-request prefix-group sampling distribution (default: `uniform`). With `zipf`, each request's group is sampled by rank with `p(rank) = (1/rank**alpha) / sum_k(1/k**alpha)`; rank starts at 1 and group index 0 is the hottest. The on-disk dataset cache uses a distinct key per `(group_distribution, zipf_alpha)`, so uniform-mode caches are never mixed with zipf-mode caches. * `--gsp-zipf-alpha FLOAT`: Zipf exponent for `--gsp-group-distribution=zipf`. Must be a finite float strictly greater than 0; larger values concentrate requests on lower-ranked (hotter) groups. Required when the distribution is `zipf`; must be omitted otherwise. Image dataset flags (for `image`): * `--image-count`: Number of images per request * `--image-resolution`: Image resolution; supports presets (4k, 1080p, 720p, 360p) or custom 'heightxwidth' format (e.g., 1080x1920, 512x768) * `--image-format`: Image format (jpeg or png) * `--image-content`: Image content type (random or blank) Agentic trace flags (for `agentic-trace`): * `--dataset-path`: path to the pre-built trace JSON * `--sharegpt-output-len`: per-turn output length (default: 220) * `--dataset-offset`: rotate the conversation list by this many entries before sampling, so successive sweep steps start on fresh conversations * `--agentic-max-turns`: cap each conversation to at most this many turns (useful for small, fast profiling runs) SPEED-Bench flags (for `speed-bench`): * `--dataset-path PATH`: path to the pre-downloaded SPEED-Bench Throughput JSONL (e.g., `throughput_1k.jsonl`). Use the [SPEED-Bench measurement framework](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/specdec_bench) to generate it. * `--speed-bench-category`: filter to one entropy category: `low_entropy`, `mixed`, or `high_entropy` (default: all) * `--speed-bench-output-len`: fixed number of output tokens per request (default: 512) ### Examples 1. To benchmark image dataset with 3 images per request, 500 prompts, 512 input length, and 512 output length, you can run: ```bash Command theme={null} python -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-3B-Instruct --disable-radix-cache ``` ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang-oai-chat \ --dataset-name image \ --num-prompts 500 \ --image-count 3 \ --image-resolution 720p \ --random-input-len 512 \ --random-output-len 512 ``` 2. To benchmark random dataset with 3000 prompts, 1024 input length, and 1024 output length, you can run: ```bash Command theme={null} python -m sglang.launch_server --model-path Qwen/Qwen2.5-3B-Instruct ``` ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 3000 \ --random-input 1024 \ --random-output 1024 \ --random-range-ratio 0.5 ``` 3. To benchmark speculative decoding throughput using SPEED-Bench (mixed-entropy category, 1K ISL), you can run: ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \ --speculative-algorithm EAGLE --speculative-draft-model-path ``` ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name speed-bench \ --dataset-path /path/to/throughput_1k.jsonl \ --speed-bench-category mixed \ --speed-bench-output-len 512 \ --num-prompts 512 ``` ### Choosing model and tokenizer * `--model` is required unless the backend exposes `GET /v1/models`, in which case the first model ID is auto-selected. * `--tokenizer` defaults to `--model`. Both can be HF model IDs or local paths. * For ModelScope workflows, setting `SGLANG_USE_MODELSCOPE=true` enables fetching via ModelScope (weights are skipped for speed). * If your tokenizer lacks a chat template, the script warns because token counting can be less robust for gibberish outputs. ### Rate, concurrency, and streaming * `--request-rate`: requests per second. `inf` sends all immediately (burst). Non-infinite rate uses a Poisson process for arrival times. * `--max-concurrency`: caps concurrent in-flight requests regardless of arrival rate. * `--disable-stream`: switch to non-streaming mode when supported; TTFT then equals total latency for chat completions. ### Other key options * `--output-file FILE.jsonl`: append JSONL results to file; auto-named if unspecified * `--output-details`: include per-request arrays (generated texts, errors, ttfts, itls, input/output lens) * `--extra-request-body '{"top_p":0.9,"temperature":0.6}'`: merged into payload (sampling params, etc.) * `--disable-ignore-eos`: pass through EOS behavior (varies by backend) * `--warmup-requests N`: run warmup requests with short output first (default 1) * `--flush-cache`: call `/flush_cache` (sglang) before main run * `--profile`: call `/start_profile` and `/stop_profile` (requires server to enable profiling, e.g., `SGLANG_TORCH_PROFILER_DIR`) * `--lora-name name1 name2 ...`: randomly pick one per request and pass to backend (e.g., `lora_path` for sglang) * `--tokenize-prompt`: send integer IDs instead of text (currently supports `--backend sglang` only) ### Authentication If your target endpoint requires OpenAI-style auth, set: ```bash Command theme={null} export OPENAI_API_KEY=sk-...yourkey... ``` The script will add `Authorization: Bearer $OPENAI_API_KEY` automatically for OpenAI-compatible routes. ### Metrics explained Printed after each run: * Request throughput (req/s) * Input token throughput (tok/s) - includes both text and vision tokens * Output token throughput (tok/s) * Total token throughput (tok/s) - includes both text and vision tokens * Total input text tokens and Total input vision tokens - per-modality breakdown * Concurrency: aggregate time of all requests divided by wall time * End-to-End Latency (ms): mean/median/std/p99 per-request total latency * Time to First Token (TTFT, ms): mean/median/std/p99 for streaming mode * Inter-Token Latency (ITL, ms): mean/median/std/p95/p99/max between tokens * TPOT (ms): Token processing time after first token, i.e., `(latency - ttft)/(tokens-1)` * Accept length (sglang-only, if available): speculative decoding accept length The script also retokenizes generated text with the configured tokenizer and reports "retokenized" counts. ### JSONL output format When `--output-file` is set, one JSON object is appended per run. Base fields: * Arguments summary: backend, dataset, request\_rate, max\_concurrency, etc. * Duration and totals: completed, total\_input\_tokens, total\_output\_tokens, retokenized totals * Throughputs and latency statistics as printed in the console * `accept_length` when available (sglang) With `--output-details`, an extended object also includes arrays: * `input_lens`, `output_lens` * `ttfts`, `itls` (per request: ITL arrays) * `generated_texts`, `errors` ### End-to-end examples 1. sglang native `/generate` (streaming): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name random \ --random-input-len 1024 --random-output-len 1024 --random-range-ratio 0.5 \ --num-prompts 2000 \ --request-rate 100 \ --max-concurrency 512 \ --output-file sglang_random.jsonl --output-details ``` 2. OpenAI-compatible Completions (e.g., vLLM): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend vllm \ --base-url http://127.0.0.1:8000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name sharegpt \ --num-prompts 1000 \ --sharegpt-output-len 256 ``` 3. OpenAI-compatible Chat Completions (streaming): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend vllm-chat \ --base-url http://127.0.0.1:8000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name random \ --num-prompts 500 \ --apply-chat-template ``` 4. Images (VLM) with chat template: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model your-vlm-model \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 --random-output-len 256 \ --num-prompts 200 \ --apply-chat-template ``` 4a) Images with custom resolution: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model your-vlm-model \ --dataset-name image \ --image-count 1 \ --image-resolution 512x768 \ --random-input-len 64 --random-output-len 128 \ --num-prompts 100 \ --apply-chat-template ``` 4b) 1080p images with PNG format and blank content: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model your-vlm-model \ --dataset-name image \ --image-count 1 \ --image-resolution 1080p \ --image-format png \ --image-content blank \ --random-input-len 64 --random-output-len 128 \ --num-prompts 100 \ --apply-chat-template ``` 5. Generated shared prefix (long system prompts + short questions): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name generated-shared-prefix \ --gsp-num-groups 64 --gsp-prompts-per-group 16 \ --gsp-system-prompt-len 2048 --gsp-question-len 128 --gsp-output-len 256 \ --num-prompts 1024 ``` Zipfian / power-law prefix popularity (opt-in via `--gsp-group-distribution=zipf`): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name generated-shared-prefix \ --gsp-num-groups 64 --gsp-prompts-per-group 16 \ --gsp-system-prompt-len 2048 --gsp-question-len 128 --gsp-output-len 256 \ --gsp-group-distribution zipf --gsp-zipf-alpha 1.2 \ --seed 42 ``` `zipf` mode samples each request's prefix group from the rank-based distribution `p(rank) = (1/rank**alpha) / sum_k(1/k**alpha)` with rank starting at 1, so group index 0 is the hottest. The total request count stays `num_groups * prompts_per_group` — identical to `uniform` mode — and only the per-request group assignment changes. `alpha` must be a finite float strictly greater than 0; larger values concentrate requests on lower-ranked (hotter) groups. The on-disk dataset cache at `~/.cache/sglang/benchmark/gen_shared_prefix_*.pkl` includes `group_distribution` and `zipf_alpha` in its key, so uniform-mode and zipf-mode runs (or two zipf runs with different alpha) never share a cache file. Uniform-mode filenames are unchanged from the legacy format, so existing caches remain valid. This flag controls prefix-popularity shape only. It does not by itself reproduce any production trace or guarantee an observed cache-hit rate for a given engine. 6. Tokenized prompts (ids) for strict length control (sglang only): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name random \ --tokenize-prompt \ --random-input-len 2048 --random-output-len 256 --random-range-ratio 0.2 ``` 7. Profiling and cache flush (sglang): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --profile \ --flush-cache ``` 8. TensorRT-LLM streaming endpoint: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend trt \ --base-url http://127.0.0.1:8000 \ --model your-trt-llm-model \ --dataset-name random \ --num-prompts 100 \ --disable-ignore-eos ``` 9. Evaluating large-scale KVCache sharing with mooncake trace (sglang only): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model model-name \ --dataset-name mooncake \ --mooncake-slowdown-factor 1.0 \ --mooncake-num-rounds 1000 \ --mooncake-workload conversation|mooncake|agent|synthetic --use-trace-timestamps true \ --random-output-len 256 ``` 10. Fake decode stress testing (PD disaggregation, decode-only): When benchmarking pure decode performance in a PD disaggregation setup, you can bypass the prefill node entirely by using `--fake-prefill`. This requires the decode server to be started with `--disaggregation-transfer-backend fake`: ```bash Command theme={null} # Step 1: Start a decode-only server with fake transfer backend python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode decode \ --disaggregation-transfer-backend fake \ --port 30001 # Step 2: Run bench_serving with --fake-prefill python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30001 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name random \ --num-prompts 500 \ --random-input-len 1024 --random-output-len 256 \ --fake-prefill ``` Similarly, `bench_one_batch_server` also supports `--fake-prefill`: ```bash Command theme={null} python3 -m sglang.bench_one_batch_server \ --base-url http://127.0.0.1:30001 \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --batch-size 32 --input-len 1024 --output-len 256 \ --fake-prefill ``` The `--fake-prefill` flag automatically injects special sentinel values into each request, telling the decode server to skip real KV transfer and generate fake KV data locally. ### Troubleshooting * All requests failed: verify `--backend`, server URL/port, `--model`, and authentication. Check warmup errors printed by the script. * Throughput seems too low: adjust `--request-rate` and `--max-concurrency`; verify server batch size/scheduling; ensure streaming is enabled if appropriate. * Token counts look odd: prefer chat/instruct models with proper chat templates; otherwise tokenization of gibberish may be inconsistent. * Image/MMMU datasets: ensure you installed extra deps (`pillow`, `datasets`, `pybase64`). * Authentication errors (401/403): set `OPENAI_API_KEY` or disable auth on your server. ### Notes * The script raises the file descriptor soft limit (`RLIMIT_NOFILE`) to help with many concurrent connections. * For sglang, `/server_info` is queried post-run to report speculative decoding accept length when available. # Benchmark and Profiling Source: https://docs.sglang.io/docs/developer_guide/benchmark_and_profiling ## Benchmark SGLang provides four benchmark tools that operate at different levels of the stack. The table below summarizes their key differences:
Tool HTTP Server Scheduler Use Case
bench\_serving Yes (async HTTP client to a running server) Yes (indirectly, via server) Realistic online serving benchmarks with latency metrics (TTFT, TPOT, ITL)
bench\_one\_batch\_server Yes (sends HTTP requests to a running server) Yes (indirectly, via server) End-to-end single-batch latency including HTTP and scheduler overhead
bench\_offline\_throughput No Yes (directly uses Engine in-process) Maximum throughput measurement without HTTP overhead
bench\_one\_batch No No (directly calls ModelRunner) Kernel-level latency profiling of a single static batch
Use `bench_serving` by default unless there are specific needs. **`bench_serving`** is an async HTTP load-testing client that sends requests at controlled rates with configurable concurrency to a running server. It measures realistic online serving metrics including time-to-first-token (TTFT), time-per-output-token (TPOT), inter-token latency (ITL), and throughput. Use `num-prompts >= 5 * max-concurrency` to measure steady-state performance. Launch a server with `sglang.launch_server` first. ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang --max-concurrency 16 --num-prompts 80 --random-input-len 256 --random-output-len 32 --dataset-name random ``` **`bench_one_batch_server`** sends a single batch as one HTTP request to a running server. Due to only having a single batch, the server is never in a steady-state and metrics will be biased. Launch a server with `sglang.launch_server` first. ```bash Command theme={null} python3 -m sglang.bench_one_batch_server --base-url http://127.0.0.1:30000 --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 32 --input-len 256 --output-len 32 ``` * Pass `--enable-multi-batch` and set `--batch-size` to a multiple of the server's `--max-running-requests` to stabilize throughput measurements. Surplus requests are queued by the scheduler and promoted batch-by-batch, amortizing per-request prefill and first-step transients into steady-state decode. Under this flag, only `overall_throughput` is authoritative; `input_throughput`, `output_throughput`, `last_ttft`, and ITL include cross-batch queueing in their denominators and should be treated as informational. * Pass `--disable-ignore-eos` when benchmarking with real prompts, where forcing decode past EOS would shift the output distribution toward gibberish. Requests then early-stop, so the whole-run `output_throughput`/ITL include the decaying-batch tail; the report gains steady-state columns measured only over the window where every request is still decoding (from the last request's first token to the first request's finish). Replay recorded traffic with `--dataset-name sharegpt` or `--dataset-name custom --dataset-path ` (recorded completion lens are ignored; `--output-len` is the shared `max_new_tokens` cap). For OpenAI-format traces with per-request parameters, use `bench_serving`. * Pass `--lora-name ` to route every prompt through a pre-loaded LoRA adapter. Requires the server to be launched with `--enable-lora --lora-paths =`. **`bench_offline_throughput`** directly instantiates the `Engine` object in-process (no HTTP server) and submits all requests at once via `engine.generate()`. The engine's scheduler handles batching and execution. This measures maximum achievable throughput without any network overhead. ```bash Command theme={null} python3 -m sglang.benchmark.offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10 ``` **`bench_one_batch`** is the lowest-level tool. It directly instantiates a `ModelRunner` and calls `extend()` / `decode()` on a fixed static batch, bypassing the scheduler entirely. The prefill and decode phases are run separately, making profiling easier but rendering the metrics unrealistic. Because there is no dynamic batching, it may run out of memory for batch sizes that a real server can handle (a real server chunks prefill into smaller batches). This is best suited for profiling individual kernel performance. ```bash Command theme={null} python3 -m sglang.benchmark.one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 32 --input-len 256 --output-len 32 ``` ## Profile with PyTorch Profiler [Pytorch Profiler](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html) is a convenient basic tool to inspect kernel execution time, call stack, and kernel overlap and occupancy. ### Profile a server with `sglang.bench_serving` ```bash Command theme={null} # set trace path export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log # start server python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct # send profiling request from client python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile ``` For `bench_serving --profile`, the output directory is selected on the client side from `--profile-output-dir` or `SGLANG_TORCH_PROFILER_DIR` (fallback: `/tmp`), then sent in the `/start_profile` request. If you call `/start_profile` directly and do not provide `output_dir`, the server uses its own `SGLANG_TORCH_PROFILER_DIR` (fallback: `/tmp`). Setting `SGLANG_TORCH_PROFILER_DIR` on both server and client is still recommended to avoid confusion about where traces are written. For more details, please refer to [Bench Serving Guide](./bench_serving). ### Profile In PD Disaggregation Mode When profiling in PD disaggregation mode, prefill and decode workers **must be profiled separately** due to torch profiler limitations. The `bench_serving` command provides dedicated options for this: #### Profile Prefill Workers ```bash Command theme={null} # set trace path export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log # start prefill and decode servers (see PD disaggregation docs for setup) python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disaggregation-mode prefill python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disaggregation-mode decode --port 30001 --base-gpu-id 1 # start router python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000 # send profiling request targeting prefill workers python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile --pd-separated --profile-prefill-url http://127.0.0.1:30000 ``` #### Profile Decode Workers ```bash Command theme={null} # send profiling request targeting decode workers python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile --pd-separated --profile-decode-url http://127.0.0.1:30001 ``` #### Important Notes * `--profile-prefill-url` and `--profile-decode-url` are **mutually exclusive** - you cannot profile both at the same time * Both options support multiple worker URLs for multi-instance setups: ```bash Command theme={null} # Profile multiple prefill workers python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --profile --pd-separated --profile-prefill-url http://127.0.0.1:30000 http://127.0.0.1:30002 # Profile multiple decode workers python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --profile --pd-separated --profile-decode-url http://127.0.0.1:30001 http://127.0.0.1:30003 ``` * Make sure `SGLANG_TORCH_PROFILER_DIR` is set on all worker nodes before starting the servers * For more details on setting up PD disaggregation, see [PD Disaggregation Guide](../advanced_features/pd_disaggregation) ### Profile a server with `sglang.benchmark.offline_throughput` ```bash Command theme={null} export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log # profile one batch with bench_one_batch.py # batch size can be controlled with --batch argument python3 -m sglang.benchmark.one_batch --model-path meta-llama/Llama-3.1-8B-Instruct --batch 32 --input-len 1024 --output-len 10 --profile # profile multiple batches with bench_offline_throughput.py python -m sglang.benchmark.offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8 ``` ### Profile a server with `sglang.profiler` When the server is running (e.g., processing a decoding request), you can start live profiling immediately by sending a profile request to the server. You can do this by running `python3 -m sglang.profiler`. For example: ```text Output theme={null} # Terminal 1: Send a generation request python3 -m sglang.test.send_one # Terminal 2: Before the above request finishes, quickly launch the following command in a separate terminal. # It will generate a profile of the above request for several decoding batches. python3 -m sglang.profiler ``` You can also combine the above operations into a single command ```text Output theme={null} python3 -m sglang.test.send_one --profile ``` ### Profile a server with HTTP API endpoints SGLang provides HTTP API endpoints to control profiling on a running server. This allows you to start and stop profiling programmatically, which is useful for capturing specific workload patterns. #### Using `/start_profile` endpoint The `/start_profile` endpoint starts profiling on the server. You can control when profiling begins and how long it runs using the following parameters: **Basic usage:** ```bash Command theme={null} # Start profiling immediately for 10 steps curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "num_steps": 10 }' ``` **Parameters:** * `output_dir` (optional): Directory where profile traces will be saved. If not specified, uses `SGLANG_TORCH_PROFILER_DIR` environment variable, or `/tmp` as the default * `num_steps` (optional): Number of steps to profile. If not specified, profiling continues until manually stopped with `/stop_profile` * `start_step` (optional): Step number at which to start profiling (inclusive). Useful for skipping warmup iterations * `activities` (optional): List of activities to profile, e.g., `["CPU", "GPU"]`. Default is `["CPU", "GPU"]` * `merge_profiles` (optional): Whether to merge distributed traces. Default is `false` * `detailed_annotations` (optional): Whether to fold per-iteration request and KV-length aggregates into the trace's `step[...]` markers, for detailed analysis. Default is `false`. See [Detailed annotations](#detailed-annotations) below. **Note on step ranges:** Profiling starts at `start_step` (inclusive) and continues for `num_steps` iterations. For example, with `start_step=3` and `num_steps=10`, profiling captures steps 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12 (10 steps total, starting from step 3). **Advanced usage with `start_step`:** ```bash Command theme={null} # Wait 5 steps (warmup), then profile for 10 steps curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "output_dir": "/tmp/profiles", "start_step": 5, "num_steps": 10, "activities": ["CPU", "GPU"] }' ``` **Continuous profiling (manual stop):** ```bash Command theme={null} # Start profiling without num_steps - must manually stop with /stop_profile curl -X POST http://127.0.0.1:30000/start_profile ``` #### Detailed annotations Set `detailed_annotations` to `true` to fold per-iteration aggregates into SGLang's existing per-forward `step[...]` span. For every execution step that runs while profiling is active, SGLang augments that step's marker on the GPU stream with the request and KV-length distribution of the step, so you can reconstruct compute and memory bounds directly from the trace without per-request details. All four per-request aggregates are appended, prefixed by phase — `c_` for context (prefill) and `g_` for generation (decode). The per-phase `sq` is always emitted so each `step[...]` label is self-contained for roofline analysis, even where it duplicates the base label's `bs` (decode) or `toks` (prefill): * `sq`: total query tokens (`Σ N_Q`) * `sqsq`: sum of squared query tokens per request (`Σ N_Q²`) * `sqsk`: sum of query·KV tokens per request (`Σ N_Q·N_KV`) * `sk`: total KV tokens (`Σ N_KV`) A pure prefill (`EXTEND`) or decode (`DECODE`) forward emits a single group; a mixed forward emits both, with `c=`/`g=` request counts. Example labels: ```text theme={null} step[EXTEND bs=1 toks=1025 c_sq=1025 c_sqsq=1050625 c_sqsk=1050625 c_sk=1025] step[DECODE bs=64 g_sq=64 g_sqsq=64 g_sqsk=100032 g_sk=100032] step[MIXED bs=66 c=2 g=64 c_sq=2048 c_sk=2048 c_sqsq=2097152 c_sqsk=2097152 g_sq=64 g_sk=65600 g_sqsq=64 g_sqsk=65600] ``` With speculative decoding (EAGLE/MTP) each request contributes multiple query tokens per step, so `sq` no longer equals `bs`. Both draft-decode and target-verify (`TARGET_VERIFY`) steps are emitted in the **generation** group. For example, a verify step with 3 draft tokens across 2 requests (`seq_lens=[10, 20]`): ```text theme={null} step[TARGET_VERIFY bs=2 g_sq=6 g_sqsq=18 g_sqsk=90 g_sk=30] ``` ```bash Command theme={null} # Profile 10 steps with detailed annotations enabled curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "output_dir": "/tmp/profiles", "num_steps": 10, "activities": ["CPU", "GPU"], "detailed_annotations": true }' ``` The annotations only appear when profiling is active with `detailed_annotations` enabled, so they add no overhead on the normal serving path. The behavior is identical in eager and CUDA graph modes. When viewing the trace (see [View traces](#view-traces)), the augmented `step[...]` markers appear on the GPU stream alongside the kernels for each step. #### Using `/stop_profile` endpoint The `/stop_profile` endpoint stops an ongoing profiling session and saves the trace file. ```bash Command theme={null} # Stop profiling and save traces curl -X POST http://127.0.0.1:30000/stop_profile ``` This is only needed when you start profiling without specifying `num_steps`. If `num_steps` is specified, profiling will automatically stop after that many steps. #### Example workflow ```bash Command theme={null} # Terminal 1: Start the server export SGLANG_TORCH_PROFILER_DIR=/tmp/profiles python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct # Terminal 2: Start continuous profiling curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "start_step": 3 }' # Terminal 3: Send requests to generate load python -m sglang.bench_serving --backend sglang --num-prompts 100 # Terminal 2: Stop profiling when done curl -X POST http://127.0.0.1:30000/stop_profile ``` ### Profiler Trace Merger for Distributed Traces SGLang now supports automatic merging of profiling traces from distributed setups with multiple parallelism types (TP, DP, PP, EP). This feature is particularly useful for analyzing performance across distributed runs. #### Multi-Node Profiling and Shared Storage Considerations Single-node profiler output merging is completely supported. When profiling in distributed environments spanning multiple nodes, shared storage (e.g., NFS, Lustre) should be accessible by all nodes for the output directory to enable merging of trace files. If there is no shared storage accessible across nodes, automatic merging of trace files during profiling is not supported directly as of now. #### HTTP API Usage ```bash Command theme={null} # Start profiling with automatic trace merging enabled curl -X POST /start_profile \ -H "Content-Type: application/json" \ -d '{ "output_dir": "/tmp/profiles", # where to store profile traces "num_steps": 10, "activities": ["CPU", "GPU"], "merge_profiles": true # optional argument to merge profile traces (default=False) }' ``` #### Command Line Usage ```bash Command theme={null} # Start profiling with merge enabled python -m sglang.profiler \ --num-steps 10 \ --cpu \ --gpu \ --output-dir /tmp/profiles \ --merge-profiles # optional argument to merge profile traces (default=False) ``` #### Output Files The profile merger generates: * Individual rank trace files: `{profile_id}-TP-{tp}-DP-{dp}-PP-{pp}-EP-{ep}.trace.json.gz` * Merged trace file: `merged-{profile_id}.trace.json.gz` ### Profile the CUDA graph capture phase The tools above profile the steady-state runtime (prefill / decode). To instead profile the **CUDA graph capture phase** that runs once at server startup, launch the server with `--enable-profile-cuda-graph`. This runs a PyTorch Profiler pass over the decode CUDA-graph capture, which is useful for diagnosing slow or memory-heavy graph capture. `--enable-profile-cuda-graph` (server arg) builds the capture profiler and always emits the per-kernel CPU/CUDA time summary tables and a CUDA memory snapshot. Persisting Chrome traces to disk is opt-in via one of two env vars (both no-ops unless `--enable-profile-cuda-graph` is also set): * `SGLANG_ENABLE_CUDA_GRAPH_CAPTURE_TRACE=1` — writes **one combined trace per tensor-parallel rank** for the whole capture pass, named `cuda_graph_capture-{runner}-TP-{tp_rank}.json.gz`. * `SGLANG_GRAPH_BATCH_CAPTURE=1` — writes **one trace per captured batch size per rank**, named `{runner}_bs_{bs}_rank{tp_rank}.json.gz`. The profiler runs on a `wait=2, warmup=0, active=1` schedule (the two dummy runs before each capture are skipped) with `record_shapes`, `with_stack`, `with_flops`, and `profile_memory` enabled, giving per-shape kernel identities, input shapes, FLOPs, and memory. If both env vars are set, `SGLANG_ENABLE_CUDA_GRAPH_CAPTURE_TRACE` (the single combined trace) takes precedence. ```bash Command theme={null} # set trace path export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log # opt in to per-batch-size capture traces (or set # SGLANG_ENABLE_CUDA_GRAPH_CAPTURE_TRACE=1 for a single combined trace per rank) export SGLANG_GRAPH_BATCH_CAPTURE=1 # launch the server with CUDA graph capture profiling enabled python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --enable-profile-cuda-graph ``` Behavior and output: * All traces are written to `${SGLANG_TORCH_PROFILER_DIR}/graph_capture_profile/` (defaults to `/tmp/graph_capture_profile/` if the variable is unset). Files are namespaced by runner class and TP rank so concurrent capture passes (e.g. EAGLE target/draft/draft-extend) and ranks don't collide. * A CUDA memory snapshot (`cuda_graph_runner_memory_usage.pickle`) and per-kernel CPU/CUDA time summary tables are always emitted for the capture phase (independent of the env vars above). * Only the decode CUDA-graph runner is profiled. The capture traces are viewed the same way as other PyTorch Profiler traces (see [View traces](#view-traces)). ### Possible PyTorch bugs If in any cases you encounter the following error (for example, using qwen 2.5 VL): ```bash Command theme={null} RuntimeError: !stack.empty() INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/autograd/profiler_python.cpp":983, please report a bug to PyTorch. Python replay stack is empty. ``` This is likely a PyTorch Bug reported in [Bug: vLLM Profiler](https://github.com/vllm-project/vllm/issues/18240) and [Bug: torch.profiler.profile](https://github.com/pytorch/pytorch/issues/101632). As a workaround, you may disable `with_stack` with an environment variable such as follows: ```bash Command theme={null} export SGLANG_PROFILE_WITH_STACK=False python -m sglang.benchmark.offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8 ``` ### View traces Trace files can be loaded and visualized from: 1. [https://ui.perfetto.dev/](https://ui.perfetto.dev/) (any browser) 2. chrome://tracing (Chrome browser only) If browser cannot open trace file due to its large size, client can generate a small trace file (\<100MB) by controlling number of prompts and lengths of prompt outputs. For example, when profiling a server, ```bash Command theme={null} python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 2 --sharegpt-output-len 100 --profile ``` This command sets the number of prompts to 2 with `--num-prompts` argument and limits the length of output sequences to 100 with `--sharegpt-output-len` argument, which can generate a small trace file for browser to open smoothly. Additionally, if you want to locate the SGLang Python source code through the cuda kernel in Trace, you need to disable CUDA Graph when starting the service. This can be done by using the `--disable-cuda-graph` parameter in the command to start the service. ## Profile with Nsight [Nsight systems](https://docs.nvidia.com/nsight-systems/) is an advanced tool that exposes more profiling details, such as register and shared memory usage, annotated code regions and low-level CUDA APIs and events. 1. Prerequisite: Install using apt, or run inside a [NVIDIA Docker container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags) or [SGLang Docker container](https://github.com/sgl-project/sglang/tree/main/docker). ```bash Command theme={null} # install nsys # https://docs.nvidia.com/nsight-systems/InstallationGuide/index.html apt update apt install -y --no-install-recommends gnupg echo "deb http://developer.download.nvidia.com/devtools/repos/ubuntu$(source /etc/lsb-release; echo "$DISTRIB_RELEASE" | tr -d .)/$(dpkg --print-architecture) /" | tee /etc/apt/sources.list.d/nvidia-devtools.list apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub apt update apt install nsight-systems-cli ``` 2. To profile a single batch, use ```bash Command theme={null} nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node python3 -m sglang.benchmark.one_batch --model meta-llama/Meta-Llama-3-8B --batch-size 64 --input-len 512 ``` 3. To profile a server, e.g. ```bash Command theme={null} # launch the server, set the delay and duration times according to needs # after the duration time has been used up, server will be killed by nsys nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node -o sglang.out --delay 60 --duration 70 python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disable-radix-cache # client python3 -m sglang.bench_serving --backend sglang --num-prompts 1000 --dataset-name random --random-input 1024 --random-output 512 ``` In practice, we recommend users to set `--duration` argument to a large value. Whenever user wants the server to stop profiling. Firstly run: ```bash Command theme={null} nsys sessions list ``` to get the session id in the form of `profile-XXXXX`, then run: ```bash Command theme={null} nsys stop --session=profile-XXXXX ``` to manually kill the profiler and generate `nsys-rep` files instantly. 4. Use NVTX to annotate code regions, e.g. to see their execution time. ```bash Command theme={null} # install nvtx pip install nvtx ``` ```python Example theme={null} # code snippets import nvtx with nvtx.annotate("description", color="color"): # some critical code ``` ### Layer-wise NVTX Profiling with Nsight Systems SGLang provides built-in layerwise NVTX annotations that can be combined with the CUDA Profiler for detailed per-layer profiling in Nsight Systems. This is particularly useful for identifying performance bottlenecks at the layer level. #### Using `--enable-layerwise-nvtx-marker` with Nsight Systems and `/start_profile` The `--enable-layerwise-nvtx-marker` flag automatically adds NVTX markers to every layer in your model. This is particularly powerful when combined with Nsight Systems profiling to see detailed per-layer performance. **Method 1: Using `/start_profile` with CUDA\_PROFILER (for programmatic control)** This method allows you to control exactly when profiling starts/stops via HTTP API while Nsight Systems is running. 1. Launch the server with layerwise NVTX enabled under Nsight Systems: ```bash Command theme={null} # Terminal 1: Start server with nsys and capture-range option nsys profile --trace-fork-before-exec=true \ --cuda-graph-trace=node \ --capture-range=cudaProfilerApi \ --capture-range-end=stop \ -o layerwise_profile \ python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --enable-layerwise-nvtx-marker \ --disable-cuda-graph ``` Note: NVTX markers are not emitted for kernel launches captured by CUDA graphs. Use `--disable-cuda-graph` to ensure all layerwise NVTX markers are emitted in the trace. 2. In another terminal, control profiling via `/start_profile` with `CUDA_PROFILER` activity: ```bash Command theme={null} # Terminal 2: Wait for server to be ready, then start CUDA profiling # Wait 3 steps for warmup, then profile for 10 steps curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "start_step": 3, "num_steps": 10, "activities": ["CUDA_PROFILER"] }' ``` 3. Send requests to generate load: ```bash Command theme={null} # Terminal 3: Generate workload python -m sglang.bench_serving --backend sglang --num-prompts 100 ``` 4. Profiling will automatically stop after 10 steps (due to `num_steps: 10`). If you hadn't specified `num_steps`, you would need to manually stop it: ```bash Command theme={null} # Terminal 2: Only needed if num_steps was not specified curl -X POST http://127.0.0.1:30000/stop_profile ``` The `--capture-range=cudaProfilerApi` option tells Nsight Systems to only capture data between `cudaProfilerStart()` and `cudaProfilerStop()` calls (triggered by `/start_profile` and `/stop_profile`), reducing overhead and file size. The `start_step` parameter skips the first 3 steps to avoid capturing warmup overhead. **Method 2: Simpler approach without `/start_profile` API** For simpler use cases where you don't need fine-grained control over profiling start/stop, you can profile with Nsight Systems capturing the entire workload: ```bash Command theme={null} # Terminal 1: Start server with layerwise NVTX # Note: --disable-cuda-graph ensures all NVTX markers are emitted python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --enable-layerwise-nvtx-marker \ --disable-cuda-graph # Terminal 2: Profile the benchmarking client nsys profile --trace-fork-before-exec=true \ --cuda-graph-trace=node \ -o layerwise_profile \ python -m sglang.bench_serving --backend sglang --num-prompts 10 ``` This approach profiles the entire client execution, including all server interactions. The layerwise NVTX markers will be visible in the Nsight Systems timeline. **Viewing the profiling results:** Open the generated `.qdrep` file with Nsight Systems: ```bash Command theme={null} nsys-ui layerwise_profile.qdrep ``` In the Nsight Systems GUI, you'll see: * **NVTX ranges**: Each layer appears as a labeled range in the timeline with detailed information in the marker metadata * **CUDA kernels**: All GPU kernels are shown alongside the layer annotations * **Layer hierarchy**: The full module path (e.g., `meta-llama/Meta-Llama-3.1-8B-Instruct.model.layers.0.self_attn.qkv_proj`) helps identify specific layers. The prefix uses the full model path from `--model-path`. * **Tensor shapes**: Input/output dimensions and parameter shapes are included in the NVTX marker data **Benefits of layerwise NVTX profiling:** * **Granular visibility**: See exactly which layers are taking the most time * **Memory tracking**: Identify layers with large memory allocations * **Bottleneck identification**: Quickly locate inefficient operations * **Communication overhead**: In multi-GPU setups, see per-layer communication costs * **Development debugging**: Validate that model architecture changes have the expected performance impact ## Other tips 1. You can benchmark a model using dummy weights by only providing the config.json file. This allows for quick testing of model variants without training. To do so, add `--load-format dummy` to the above commands and then you only need a correct `config.json` under the checkpoint folder. 2. You can benchmark a model with modified configs (e.g., less layers) by using `--json-model-override-args`. For example, you can benchmark a model with only 2 layers and 2 kv heads using: ```bash Command theme={null} python -m sglang.benchmark.one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch 32 --input-len 256 --output-len 32 --load-format dummy --json-model-override-args '{"num_hidden_layers": 1, "num_key_value_heads": 1}' ``` 3. You can use `--python-backtrace=cuda` to see python call stack for all CUDA kernels, as in PyTorch Profiler. (Caveat: this can cause inaccurately long kernel runtimes for CUDA event based timing) 4. For more arguments see [Nsight Systems User Guide](https://docs.nvidia.com/nsight-systems/UserGuide/index.html). # Contribution Guide Source: https://docs.sglang.io/docs/developer_guide/contribution_guide Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you’re fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process. ## Install SGLang from Source ### Fork and clone the repository **Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally. ```bash theme={null} git clone https://github.com//sglang.git ``` ### Build from source Refer to [Install SGLang from Source](../get-started/install#method-2-from-source). ## Format code with pre-commit We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run: ```bash theme={null} pip3 install pre-commit pre-commit install pre-commit run --all-files ``` * **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request. * **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch. * Documentation links and anchors are checked with Mintlify in CI. To run the same version locally, install it with `npm install -g mint@4.2.559`, then run `cd docs && mint broken-links --check-anchors --check-redirects`. * The manual `lychee` pre-commit hook checks links in the repository-level `README.md`: `pre-commit run --hook-stage manual lychee --all-files`. ## Run and add unit tests If you add a feature or fix a bug, add focused regression coverage when the test protects a concrete behavior, invariant, or bookkeeping contract. ### Unit tests (no server required) Unit tests live under [`test/registered/unit/`](https://github.com/sgl-project/sglang/tree/main/test/registered/unit), organized to mirror the `python/sglang/srt/` source tree. These tests validate component logic **without** launching a server or loading real model weights. SGLang supports both Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework and [pytest](https://docs.pytest.org/). Registered CI tests must use `CustomTestCase` instead of raw `unittest.TestCase`, register themselves with `register_*_ci(...)`, and include a standard `__main__` entry point. CI discovers registrations with `test/run_suite.py` and executes each selected test file directly with fail-fast enabled. **When to add a unit test:** If you modify a file under `python/sglang/srt/`, check whether a corresponding test exists in `test/registered/unit/` and add coverage for your changes. For example: ```text theme={null} srt/mem_cache/radix_cache.py → unit/mem_cache/test_radix_cache_unit.py srt/sampling/sampling_params.py → unit/sampling/test_sampling_params.py ``` **Run tests locally:** ```bash theme={null} # Closest to how CI executes an individual registered test file python3 test/registered/unit/mem_cache/test_radix_cache_unit.py # Run a registered CI suite python3 test/run_suite.py --hw cpu --suite base-a-test-cpu # Pytest remains useful for local directory-level discovery pytest test/registered/unit/mem_cache/ -v ``` **Run with coverage:** ```bash Command theme={null} pytest test/registered/unit/ --cov --cov-config=.coveragerc -v ``` For conventions on CI registration, test structure, and examples, see [`test/registered/unit/README.md`](https://github.com/sgl-project/sglang/tree/main/test/registered/unit/README.md). ### E2E tests (server required) For tests that require launching a server, refer to [`test/registered/README.md`](https://github.com/sgl-project/sglang/tree/main/test/registered/README.md) for guidance on where to place your test. For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md). ## Write documentation Documentation is a good way for new contributors to learn the SGLang codebase. See [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md) for the current Mintlify setup and validation commands. ## Test the accuracy If your code changes model output, run an accuracy evaluation appropriate for the affected model and feature. For example, after launching a server, run the unified GSM8K evaluator: ```bash theme={null} # Launch a server sglang serve --model-path Qwen/Qwen3-8B # Evaluate sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test. This test can have significant variance (1%–5%) in accuracy due to batching and the non-deterministic nature of the inference engine. Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test. GSM8K is too easy for many state-of-the-art models, so use a more challenging or feature-specific evaluation when appropriate. See [Evaluating New Models with SGLang](/docs/developer_guide/evaluating_new_models) for current MMLU, GSM8K, HellaSwag, GPQA, HumanEval, and MMMU commands. Additional examples are available in [test/manual/eval](https://github.com/sgl-project/sglang/tree/main/test/manual/eval). ## Benchmark the speed Refer to [Benchmark and Profiling](./benchmark_and_profiling). ## Requesting a review for merge You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md). You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals. Then your PR can be merged. ## How to Trigger CI Tests We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests. Users with permission are listed in the [CI\_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) **PR authors** can use `/rerun-failed-ci` on their own PRs even if they are not listed in `CI_PERMISSIONS.json`. Selective reruns have additional rules because they execute PR code on self-hosted runners; see the permission table below. For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands: * `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`). * `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, skipped, cancelled, or timed out**. * `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`). * `/run-full-ci`: Baseline **and** extra CI. Short form of `/tag-and-rerun-ci extra` — applies both labels, then reruns everything at the current commit that didn't succeed. * `/run-extra-ci`: Extra CI **only**. Applies both labels, then reruns just `PR Test Extra`, leaving baseline CI runs alone. Use it when baseline is already green and you only need the extra tier. * `/rerun-test [ ...]`: Reruns one or more specific tests directly. A spec may select a file, class, or method using `::[.]`. Multiple specs and file globs are supported. `--changed` (short form `-c`) adds every test file the PR itself adds or modifies (under `test/registered/` or the multimodal test directory), so a PR that touches several tests can rerun them all without listing them. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py`, `/rerun-test test_*backend*.py`, `/rerun-test --changed`, or `/rerun-test -c`. * `/rerun-group [ ...]`: Expands one or more registered test groups (for example, `/rerun-group rust-server`) and dispatches their tests through the same selective-rerun workflow. Groups are defined in [`scripts/ci/rerun_test_groups.json`](https://github.com/sgl-project/sglang/blob/main/scripts/ci/rerun_test_groups.json). The rerun commands have the following permission rules: | Command | PR author | Other users | Additional rule for fork PRs | | - | - | - | - | | `/rerun-failed-ci` | Always allowed on their own PR | Must have `can_rerun_failed_ci` in `CI_PERMISSIONS.json` | None | | `/run-full-ci`, `/run-extra-ci` | Not auto-allowed -- these apply labels, which the PR-author allowance does not cover | Must have both `can_tag_run_ci_label` and `can_rerun_failed_ci` in `CI_PERMISSIONS.json` (every listed user has both) | None | | `/rerun-test`, `/rerun-group` | No special allowance; judged as any other commenter | Must have `cooldown_interval_minutes: 0` in `CI_PERMISSIONS.json`, or `write`/`admin` repository permission | None -- the same rule applies wherever the PR comes from | Selective reruns do not build or install a PR-local `sglang-kernel` wheel. If a PR changes `python/sglang/kernels/aot/` or code that depends on that AOT change, use the normal/full PR CI workflow. A `/rerun-test` or `/rerun-group` result may use the pinned released kernel and does not validate the combined change. If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302). If a command isn't recognized, the handler reacts 😕 instead, and replies with the closest matching command when there is one. A comment with no reaction at all means the handler never saw it — check the command is spelled as listed above. To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`). If you don’t have permission and you’re not the PR author, please ask maintainers to trigger CI for you. ### CI control labels Four labels change how a PR's CI runs. Label before you trigger: `max-concurrency` fixes the shard fan-out when the run is created. | Label | Effect | | - | - | | `bypass-fail-fast` | A job failing no longer aborts its siblings; the run continues instead of stopping at the first root-cause failure. A failing `lint` still stops everything. | | `parallel-stages` | Stages stop waiting on each other. `base-a`, `base-b` and `base-c` dispatch together, the way a scheduled run does. | | `max-concurrency` | A suite fans out to all of its shards at once rather than a third of them. | | `highest-priority` | All three of the above, and the PR is also skipped by the batch-cancel workflow, sorted first in the active-runs report, and never closed as stale. | Each spends extra runner capacity, so they unblock a specific PR rather than serve as a default. ### CI rate limits Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests. We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources. Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter: ```yaml Config theme={null} cool-down-minutes: description: "Cooldown period in minutes for low-permission users; 0 disables rate limiting" type: number default: 120 ``` Users listed in [CI\_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval. ## Code style guidance * Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function. * Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code. * Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code. * A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value in `__init__` whenever possible. * Make functions as pure as possible. Avoid in-place modification of arguments. * Prefer immutable data and compute configuration-derived values once during initialization when their inputs cannot change. * Keep functions under roughly 100 lines and make orchestration functions read like high-level pseudocode by extracting details into focused helpers. * Keep files concise. If a file exceeds 2,000 lines of code, split it into cohesive smaller modules. * Avoid adding mixins; prefer composition or plain functions. Use `msgspec.Struct` for new data containers instead of `dataclasses.dataclass` or `attrs`. * Prefer keyword arguments for calls with two or more arguments, and pass callees the specific values they need instead of a large state-holding object. * In a file, put core data structures at the top of the file. Put utility functions at the bottom of the file. * Keep tests run fast. * If a single test file runs longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`). * If a single job in a GitHub Actions workflow runs longer than 30 minutes, split it into smaller jobs or steps. * Reuse server launches across test methods in E2E test files. * Never use `pickle.loads()`, `pickle.load()`, or `recv_pyobj()` to deserialize untrusted or network-received data. Python's [pickle module is not secure](https://docs.python.org/3/library/pickle.html) — it can execute arbitrary code during deserialization. Use safe serialization formats such as [msgpack](https://github.com/jcrist/msgspec) or JSON instead. * When supporting new hardware or features, follow these guidelines: * Do not drastically change existing code. * Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_ascend.py`). * If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch. ## How to update kernels in SGLang Choose the implementation and release path based on the kernel's dependencies. For a lightweight kernel that does not depend on CUTLASS or another large C++ project, prefer the in-tree JIT kernel path. Use the AOT `sglang-kernel` path for heavyweight kernels, large C++ dependencies, or operations that need wheel packaging and Torch operator registration. FlashInfer-based kernels are an exception and may still use the JIT path. SGLang also consumes separately released custom builds of DeepGEMM and DeepEP; update those in their source repositories as described below. ### Update sglang-kernel The `sglang-kernel` distribution (formerly `sgl-kernel`) is a separate Python package, but its source now lives in this repository under `python/sglang/kernels/aot/`. Normal PR CI builds and installs a PR-local `sglang-kernel` wheel, so it can test an AOT kernel and its caller together. This does not establish compatibility after merge: installed SGLang and scheduled CI use the released version pinned in `python/pyproject.toml`. If a caller unconditionally requires a new operator or a changed kernel contract, first merge and release the kernel, update the pinned `sglang-kernel` version, and then land the caller. A combined PR is acceptable only when every supported path remains correct with the pinned wheel—for example, when the caller checks operator availability and retains a semantically equivalent fallback until the released wheel includes the change. Selective `/rerun-test` and `/rerun-group` workflows do not install PR-local wheels and cannot validate the combined AOT path. For an AOT kernel change: 1. Implement the kernel under `python/sglang/kernels/aot/csrc/` and update its declaration, Torch registration, and CMake source list. 2. Expose the Python API under `python/sglang/kernels/aot/python/sgl_kernel/`. 3. Add correctness tests under `python/sglang/kernels/aot/tests/` and a benchmark under `python/sglang/kernels/aot/benchmark/` when applicable. 4. Build and test from `python/sglang/kernels/aot/` following its [README](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/aot/README.md). Do not bump the pinned `sglang-kernel` version in `python/pyproject.toml` merely to test a PR-local kernel change. ### Update sgl-deep-gemm Develop SGLang's customized DeepGEMM package on the [`dev` branch of `sgl-project/DeepGEMM`](https://github.com/sgl-project/DeepGEMM/tree/dev). Rebase incoming changes onto `dev`, place new or modified package tests under `sgl_deep_gemm/tests/`, and follow the [`sgl-deep-gemm` README](https://github.com/sgl-project/DeepGEMM/blob/dev/sgl_deep_gemm/README.md) to build and install a local wheel. After the implementation is merged, ask the SGLang team to run the [sgl-deep-gemm release workflow](https://github.com/sgl-project/sglang/actions/workflows/release-whl-deepgemm.yml) with the new version, CUDA target, and DeepGEMM branch. Once all required wheels are published and verified, update the `sgl-deep-gemm` pin in `python/pyproject.toml` before landing SGLang code that requires the new or changed behavior. ### Update sgl-deep-ep Develop `sgl-deep-ep` in [`sgl-project/DeepEP`](https://github.com/sgl-project/DeepEP). Use `sgl-deepep` for both implementation and packaging changes for CUDA 13 on x86\_64 or aarch64. The [`sgl-deep-ep` README](https://github.com/sgl-project/DeepEP/blob/sgl-deepep/sgl_deep_ep/README.md) describes the platform prerequisites and release matrix. To validate locally, check out `sgl-deepep` (or your development branch containing both the implementation and `sgl_deep_ep`) as `DeepEP-source`. Install the required build dependencies first. The following example builds a wheel for the host architecture, installs that exact wheel, and verifies that its guarded package import succeeds: ```bash theme={null} DEEPEP_OUTPUT_DIR="$(mktemp -d)" bash DeepEP-source/sgl_deep_ep/build_sgl_deep_ep.sh \ "${DEEPEP_OUTPUT_DIR}" \ 13.0 \ "$(uname -m)" python3 -m pip install --force-reinstall --no-deps \ "${DEEPEP_OUTPUT_DIR}"/sgl_deep_ep-*.whl python3 -c "import deep_ep; print(deep_ep.__file__)" ``` The import check validates packaging and binary loading, but not communication correctness. On a configured multi-GPU host, also run: ```bash theme={null} python3 DeepEP-source/tests/elastic/test_ep.py --num-processes 8 ``` Adjust `--num-processes` to the available GPUs and run the internode or low-latency tests when those transports changed. After local validation, ask the SGLang team to run the [sgl-deep-ep release workflow](https://github.com/sgl-project/sglang/actions/workflows/release-whl-deepep.yml) with the new version and DeepEP ref. The workflow builds the implementation and packaging from that single ref (default: `sgl-deepep`). After verifying the published wheels for the supported Python versions and architectures, update the `sgl-deep-ep` pin in `python/pyproject.toml` before landing dependent SGLang changes. ## Tips for newcomers If you want to contribute but don’t have a specific idea in mind, pick issues labeled [“good first issue” or “help wanted”](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase. Also check out the following materials as startup guide: * [Mini-SGLang](https://github.com/sgl-project/mini-sglang) for a quick overview on the structure of sglang. * [Code Walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow. * [GTC-2026 Training Lab](https://drive.google.com/file/d/1mwOZEtipNLJzrflCTodj34KhuOZEoEw5/view?usp=drive_link) for hands-on practices of how to do optimization, benchmarking, or profiling on a launched SGLang instance. If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io). Thank you for your interest in SGLang. Happy coding! # Development Guide Using Docker Source: https://docs.sglang.io/docs/developer_guide/development_guide_using_docker The `lmsysorg/sglang:dev` image is a nightly build from `main` with additional development tools and dependencies for modifying and testing SGLang. For running models without a development environment, use the stable or nightly images described in the [Installation guide](/docs/get-started/install). ## Setup VSCode on a Remote Host (Optional - you can skip this step if you plan to run sglang dev container locally) 1. In the remote host, download `code` from [VSCode](https://code.visualstudio.com/download) and run `code tunnel` in a shell. Example ```bash Command theme={null} wget https://vscode.download.prss.microsoft.com/dbazure/download/stable/fabdb6a30b49f79a7aba0f2ad9df9b399473380f/vscode_cli_alpine_x64_cli.tar.gz tar xf vscode_cli_alpine_x64_cli.tar.gz # https://code.visualstudio.com/docs/remote/tunnels ./code tunnel ``` 2. In your local machine, press F1 in VSCode and choose "Remote Tunnels: Connect to Tunnel". ## Setup Docker Container ### Option 1. Use the default dev container automatically from VSCode There is a `.devcontainer` folder in the sglang repository root folder to allow VSCode to automatically start up within dev container. You can read more about this VSCode extension in VSCode official document [Developing inside a Container](https://code.visualstudio.com/docs/devcontainers/containers). VSCode Dev Container Architecture *Figure 1: Diagram from VSCode official documentation [Developing inside a Container](https://code.visualstudio.com/docs/devcontainers/containers).* To enable this, you only need to: 1. Start Visual Studio Code and install [VSCode dev container extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers). 2. Press F1, type and choose "Dev Container: Open Folder in Container. 3. Input the `sglang` local repo path in your machine and press enter. The first time you open it in dev container might take longer due to docker pull and build. Once it's successful, you should set on your status bar at the bottom left displaying that you are in a dev container: VSCode Dev Container Status Bar Now when you run `sglang.launch_server` in the VSCode terminal or start debugging using F5, sglang server will be started in the dev container with all your local changes applied automatically: SGLang Server Running in Dev Container ### Option 2. Start up containers manually (advanced) The following startup command is an example for internal development by the SGLang team. You can **modify or add directory mappings as needed**, especially for model weight downloads, to prevent repeated downloads by different Docker containers. ❗️ **Note on RDMA** 1. `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them but keeping them there does not harm. Thus, we enable these two flags by default in the commands below. 2. You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`. ```bash Command theme={null} # Change the name to yours docker run -itd --shm-size 32g --gpus all -v --ipc=host --network=host --privileged --name sglang_dev lmsysorg/sglang:dev /bin/zsh docker exec -it sglang_dev /bin/zsh ``` Some useful volumes to mount are: 1. **Huggingface model cache**: mounting model cache can avoid re-download every time docker restarts. Default location on Linux is `~/.cache/huggingface/`. 2. **SGLang repository**: code changes in the SGLang local repository will be automatically synced to the .devcontainer. Example 1: Monting local cache folder `/opt/dlami/nvme/.cache` but not the SGLang repo. Use this when you prefer to manually transfer local code changes to the devcontainer. ```bash Command theme={null} docker run -itd --shm-size 32g --gpus all -v /opt/dlami/nvme/.cache:/root/.cache --ipc=host --network=host --privileged --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh docker exec -it sglang_zhyncs /bin/zsh ``` Example 2: Mounting both HuggingFace cache and local SGLang repo. Local code changes are automatically synced to the devcontainer as the SGLang is installed in editable mode in the dev image. ```bash Command theme={null} docker run -itd --shm-size 32g --gpus all -v $HOME/.cache/huggingface/:/root/.cache/huggingface -v $HOME/src/sglang:/sgl-workspace/sglang --ipc=host --network=host --privileged --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh docker exec -it sglang_zhyncs /bin/zsh ``` ## Debug SGLang with VSCode Debugger 1. (Create if not exist) open `launch.json` in VSCode. 2. Add the following config and save. Please note that you can edit the script as needed to apply different parameters or debug a different program (e.g. benchmark script). ```JSON Config theme={null} { "version": "0.2.0", "configurations": [ { "name": "Python Debugger: launch_server", "type": "debugpy", "request": "launch", "module": "sglang.launch_server", "console": "integratedTerminal", "args": [ "--model-path", "meta-llama/Llama-3.2-1B", "--host", "0.0.0.0", "--port", "30000", "--trust-remote-code", ], "justMyCode": false } ] } ``` 3. Press "F5" to start. VSCode debugger will ensure that the program will pause at the breakpoints even if the program is running at remote SSH/Tunnel host + dev container. ## Profile ```bash Command theme={null} # Change batch size, input, output and add `disable-cuda-graph` (for easier analysis) # e.g. DeepSeek V3 nsys profile -o deepseek_v3 python3 -m sglang.benchmark.one_batch --batch-size 1 --input 128 --output 256 --model deepseek-ai/DeepSeek-V3 --trust-remote-code --tp 8 --disable-cuda-graph ``` ## Evaluation ```bash Command theme={null} # GSM8K via sgl-eval: all 1319 test questions sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 1319 \ --num-threads 2000 ``` # Development Guide for JIT Kernels Source: https://docs.sglang.io/docs/developer_guide/development_jit_kernel_guide ## Environment Setup We strongly recommend using `clangd` as the language server for JIT kernel development. For Ubuntu/Debian, you can download clangd from [apt.llvm.org](https://apt.llvm.org/). If you are using VS Code, we recommend installing the `clangd` extension for better IDE integration. All JIT-related files are located in `python/sglang/kernels/jit`. Unlike `sgl-kernel`, which compiles CUDA/C++ binaries ahead of time (AOT), just-in-time (JIT) kernels are compiled at runtime. Consequently, a static `compile_commands.json` cannot be generated. To enable code completion with `clangd`, run `python -m sglang.kernels.jit` to generate a `.clangd` configuration file in your current directory. After generating the file, restart the clangd language server. It should now recognize all JIT kernel files. ## Code Structure ### C++ Implementation C++ source code is located in `python/sglang/kernels/jit/csrc`. Reusable functions should be placed in `python/sglang/kernels/jit/include`. JIT C++ lives in `namespace sglang`: open it after the include block and close it at the end of the file, with the device kernels and the host wrapper both inside. The shared `host::` and `device::` helpers are nested in it as well, so they resolve unqualified and need no `sglang::` prefix. We use [tvm-ffi](https://github.com/apache/tvm-ffi) for efficient foreign language bindings. Refer to the [documentation](https://tvm.apache.org/ffi/) for advanced usage, such as exporting C++ objects. Typically, `tvm::ffi::TensorView` is sufficient for passing PyTorch Tensors from Python. ### Python Interface Python interfaces are defined in `python/sglang/kernels/jit`. The `load_jit` utility function in `python/sglang/kernels/jit/utils/compile.py` loads and returns the compiled module. To export a C++ function (e.g., `cpp_func`), pass `cuda_wrappers=[("func", "cpp_func")]` to `load_jit`. The function can then be called in Python as `module.func`. `load_jit` emits the export wrapper inside `namespace sglang`, so write `cpp_func` without a `sglang::` prefix. For caching compiled modules, prefer `sglang.kernels.jit.utils.cache_once` over `functools.lru_cache`. `functools.lru_cache` is not compatible with `torch.compile`. ### C++ Utilities The following C++ utilities are available: #### Integer Range Similar to PyTorch, we provide an `irange` function to represent an integer range. ```C++ Example theme={null} #include void test() { for (auto i : host::irange(100)) { // [0, 100) // do something } for (auto i : host::irange(0, 100)) { // [0, 100) // do something } } ``` #### Runtime Checking `CHECK_HOST` is the preferred runtime check: stream-style, and zero overhead when the check passes — the message expressions are only evaluated on failure. `RuntimeCheck` is the function-style alternative; note its message arguments are always evaluated, even when the check passes. `RuntimeDeviceCheck` verifies the status of the last kernel launch, and `CHECK_CUDA` is its stream-style equivalent for checking a `cudaError_t` with extra context. ```C++ Example theme={null} #include #include void test() { CHECK_HOST(1 + 1 == 2) << 1 + 1 << " != " << 2; // preferred host::RuntimeCheck(1 + 1 == 2, 1 + 1, " != ", 2); host::RuntimeDeviceCheck(); // check the provided `cudaError_t` host::RuntimeDeviceCheck(cudaGetLastError()); CHECK_CUDA(cudaGetLastError()) << "after my_kernel launch"; } ``` #### Tensor Checking `TensorMatcher` provides a readable way to validate and extract tensor shape information. ```cpp Example theme={null} #include void test(const tvm::ffi::TensorView k_cache, const tvm::ffi::TensorView v_cache) { using namespace host; auto D = SymbolicSize{"D"}; // cache dimension auto N = SymbolicSize{"N"}; // kvcache stride auto dtype = SymbolicDType{}; auto device = SymbolicDevice{}; TensorMatcher({-1, D}) // .with_strides({N, 1}) .with_dtype(dtype) .with_device(device) .verify(k_cache) .verify(v_cache); } ``` Configure the `TensorMatcher` with expected stride, dtype, and device properties before verification. * If `with_strides` is omitted, the tensor is expected to be contiguous. * Template arguments in `with_dtype` restrict the allowed data types. * Template arguments in `with_device` restrict the allowed devices. * Values passed to `with_xxx` methods enforce equality checks. * Passing `-1` for size or stride allows matching any value. A `Symbolic` variable must resolve to the same value across all verifications. Use `.unwrap()` to retrieve the matched value after verification. > Note: `TensorMatcher` is a temporary expression and should not be stored in a variable. > Tip: Add `//` at the end of the `TensorMatcher` chain to enforce proper indentation. #### Kernel Launching `LaunchKernel::resolve_device` retrieves the current `cudaStream` from PyTorch. Kernels can also be launched directly using `LaunchKernel`. ```cpp Example theme={null} #include #include __global__ void kernel() {} void test() { const auto num_blocks = 1; const auto num_threads = 32; const auto dynamic_smem = 0; DLDevice dev; // suppose this is initialized properly host::LaunchKernel(num_blocks, num_threads, dev)(kernel); cudaStream_t stream = host::LaunchKernel::resolve_device(dev); host::LaunchKernel(num_blocks, num_threads, stream, dynamic_smem)(kernel); } ``` ## Add new kernels This section walks through a complete, end-to-end example of adding a new JIT kernel to the system. We use a simple add\_constant kernel as a running example, which adds a constant integer value to every element of an input tensor. Conceptually, the Python interface looks like this: ```python Example theme={null} def add_constant(src: torch.Tensor, c: int): return src + c ``` ### STEP 1: Write the C++ kernel Write your CUDA kernel in [kernels/jit/csrc/elementwise/add\_constant.cuh](https://github.com/sgl-project/sglang/blob/main/python/sglang/kernels/jit/csrc/elementwise/add_constant.cuh). For demonstration purposes, we pass the constant value as a template parameter. ```cpp Example theme={null} #include // For TensorMatcher, SymbolicSize, SymbolicDevice #include // For LaunchKernel #include // For div_ceil, CHECK_HOST #include #include #include #include namespace sglang { template __global__ void add_constant_kernel(int32_t* dst, const int32_t* src, size_t length) { size_t idx = blockIdx.x * blockDim.x + threadIdx.x; if (idx < length) { dst[idx] = src[idx] + kConstant; } } constexpr size_t kBlockSize = 256; // You can also use struct with static method as an alternative template void add_constant(tvm::ffi::TensorView dst, tvm::ffi::TensorView src) { using namespace host; // 1. Validate input tensors SymbolicSize N = {"num_elements"}; SymbolicDevice device_; TensorMatcher({N}) // 1D tensor, must be contiguous .with_dtype() // must be int32 .with_device(device_) // must be on CUDA device .verify(dst) // check tensor dst .verify(src); // check tensor src // 2. Extract required parameters, prepare for kernel launch const size_t num_elements = N.unwrap(); const size_t grid_size = div_ceil(num_elements, kBlockSize); const DLDevice device = device_.unwrap(); // some extra runtime checks using CHECK_HOST CHECK_HOST(num_elements > 0) << "We only support non-empty tensors, got num_elements = " << num_elements; // 3. Launch the kernel. Error code will be automatically checked. LaunchKernel(grid_size, kBlockSize, device /*, dynamic_smem*/)( // kernel function add_constant_kernel, // kernel arguments static_cast(dst.data_ptr()), static_cast(src.data_ptr()), num_elements); } } // namespace sglang ``` ### STEP 2: Create Python Interfaces Next, expose the kernel through a Python wrapper. Create a new file at [kernels/ops/elementwise/add\_constant.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/kernels/ops/elementwise/add_constant.py) and expose the needed interfaces. ```python Example theme={null} from __future__ import annotations from typing import TYPE_CHECKING import torch from sglang.kernels.jit.utils import cache_once, load_jit, make_cpp_args if TYPE_CHECKING: from tvm_ffi.module import Module @cache_once def _jit_add_constant_module(constant: int) -> Module: args = make_cpp_args(constant) # pass all the template argument return load_jit( "add_constant", *args, cuda_files=["elementwise/add_constant.cuh"], cuda_wrappers=[("add_constant", f"add_constant<{args}>")], ) def add_constant(src: torch.Tensor, constant: int) -> torch.Tensor: if not src.is_cuda: raise RuntimeError("src must be a CUDA tensor") if src.dtype != torch.int32: raise RuntimeError(f"Unsupported dtype {src.dtype}. Supported: int32") dst = torch.empty_like(src) module = _jit_add_constant_module(constant) module.add_constant(dst, src) return dst ``` Keep the Python wrapper thin, but still validate the basic invariants such as device and dtype before dispatch. In the current JIT/FFI path, invalid tensors are not always rejected safely before launch. ### STEP 3: Use your kernel Finally, import and use the kernel like a regular Python function: ```python Example theme={null} from sglang.kernels.ops.elementwise.add_constant import add_constant ``` For a complete, runnable example, refer to [test\_add\_constant.py](https://github.com/sgl-project/sglang/blob/main/test/registered/kernels/ops/elementwise/test_add_constant.py). ## C++ Include Library Reference The JIT kernel framework provides a set of reusable C++ headers in `python/sglang/kernels/jit/include/sgl_kernel/`. Each header is designed to be lightweight and self-contained. Below is a summary of each header and its key APIs. ### Core Utilities
Header Namespace Purpose
utils.h host Host-side essentials: RuntimeCheck, CHECK\_HOST(cond) \<\< ..., Panic, div\_ceil, irange
utils.cuh device / host Type aliases (fp16\_t, bf16\_t, ...), SGL\_DEVICE macro, PDL helpers, LaunchKernel, RuntimeDeviceCheck, CHECK\_CUDA(expr) \<\< ...
source\_location.h (global) Portable std::source\_location wrapper for error reporting
runtime.cuh host::runtime CUDA runtime queries: get\_blocks\_per\_sm, get\_sm\_count, get\_cc\_major, get\_runtime\_version, get\_available\_dynamic\_smem\_per\_block
### Tensor Validation
Header Namespace Purpose
tensor.h host TensorMatcher, SymbolicSize, SymbolicDType, SymbolicDevice
### Math & Type System
Header Namespace Purpose
math.cuh device::math max, min, abs, sqrt, rsqrt, exp, sin, cos, constants
type.cuh (global) / device DTypeTrait\, packed\_t\, device::cast\(from)
### Memory Access
Header Namespace Purpose
vec.cuh device AlignedVector\ - vectorized load/store (up to 128-bit; 256-bit requires Blackwell GPUs)
tile.cuh device::tile Memory\ - cooperative tiled memory I/O (thread/warp/CTA)
### Parallel Primitives
Header Namespace Purpose
warp.cuh device::warp reduce\ (SUM/MAX/MIN, grouped or inter-group) and reduce\_sum / reduce\_max / reduce\_min wrappers via \_\_shfl\_xor\_sync
cta.cuh device::cta reduce\_max across warps via shared memory
atomic.cuh device::atomic max - atomic float max (CUDA + ROCm fallback)
### Reusable Kernel Templates
Header Namespace Purpose
impl/norm.cuh host::norm / device::norm RMSNorm building blocks (warp & CTA paths, StorageType)
# Evaluating New Models with SGLang Source: https://docs.sglang.io/docs/developer_guide/evaluating_new_models This document provides commands for evaluating models' accuracy and performance. Before open-sourcing new models, we strongly suggest running these commands to verify whether the score matches your internal benchmark results. **For cross verification, please submit commands for installation, server launching, and benchmark running with all the scores and hardware requirements when open-sourcing your models.** [Reference: MiniMax M2](https://github.com/sgl-project/sglang/pull/12129) ## Accuracy ### LLMs SGLang provides built-in scripts to evaluate common benchmarks. **MMLU** ```bash Command theme={null} sgl-eval run mmlu \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 1000 \ --max-tokens 8192 ``` **GSM8K** ```bash Command theme={null} sgl-eval run gsm8k \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 200 ``` **HellaSwag** ```bash Command theme={null} python benchmark/hellaswag/bench_sglang.py \ --host 127.0.0.1 \ --port 30000 \ --num-questions 200 \ --num-shots 20 ``` **GPQA** ```bash Command theme={null} sgl-eval run gpqa \ --base-url http://127.0.0.1:30000/v1 \ --num-examples 198 \ --max-tokens 120000 \ --n-repeats 8 ``` For reasoning models, add `--thinking` (DeepSeek-V3 / Kimi-K2 style `thinking` key) or `--chat-template-kwarg enable_thinking=true` (Qwen3 / GLM style). You may skip it if the model has forced thinking enabled. **HumanEval** ```bash Command theme={null} pip install human_eval python -m sglang.test.run_eval \ --eval-name humaneval \ --num-examples 10 \ --port 30000 ``` ### VLMs **MMMU** ```bash Command theme={null} python benchmark/mmmu/bench_sglang.py \ --port 30000 \ --concurrency 64 ``` You can set max tokens by passing `--extra-request-body '{"max_tokens": 4096}'`. For models capable of processing video, we recommend extending the evaluation to include `VideoMME`, `MVBench`, and other relevant benchmarks. ## Performance Performance benchmarks measure **Latency** (Time To First Token - TTFT) and **Throughput** (tokens/second). ### LLMs **Latency-Sensitive Benchmark** This simulates a scenario with low concurrency (e.g., single user) to measure latency. ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --host 0.0.0.0 \ --port 30000 \ --dataset-name random \ --num-prompts 10 \ --max-concurrency 1 ``` **Throughput-Sensitive Benchmark** This simulates a high-traffic scenario to measure maximum system throughput. ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --host 0.0.0.0 \ --port 30000 \ --dataset-name random \ --num-prompts 1000 \ --max-concurrency 100 ``` **Single Batch Performance** You can also benchmark the performance of processing a single batch offline. ```bash Command theme={null} python -m sglang.bench_one_batch_server \ --model \ --batch-size 8 \ --input-len 1024 \ --output-len 1024 ``` You can run more granular benchmarks: * **Low Concurrency**: `--num-prompts 10 --max-concurrency 1` * **Medium Concurrency**: `--num-prompts 80 --max-concurrency 16` * **High Concurrency**: `--num-prompts 500 --max-concurrency 100` ## Reporting Results For each evaluation, please report: 1. **Metric Score**: Accuracy % (LLMs and VLMs); Latency (ms) and Throughput (tok/s) (LLMs only). 2. **Environment settings**: GPU type/count, SGLang commit hash. 3. **Launch configuration**: Model path, TP size, and any special flags. 4. **Evaluation parameters**: Number of shots, examples, max tokens. # Layer boundaries Source: https://docs.sglang.io/docs/developer_guide/layer_boundary Declare stage inputs and outputs, bind communication and residual operations, and integrate model computation with layer boundaries. The `sglang.srt.layers.layer_boundary` package manages the transition between compute stages. It completes outstanding reductions, moves token rows, applies the producer's residual update, and prepares the consumer's normalized or quantized input. The model continues to run attention, dense FFNs, and MoE computation. Use this guide when integrating a model or changing a boundary implementation. The first part explains the concepts, the second shows how to integrate a model, and the last describes the internals you touch when you extend the package. It documents the current internal interfaces; they are not a versioned plugin API. For model development setup, see the [contribution guide](/docs/developer_guide/contribution_guide). ## Scope and ownership A boundary connects a **producer output** to a **consumer input**. An attention-to-FFN transition and an FFN-to-attention transition use the same contracts. A model can also have consecutive mixers, consecutive FFNs, or a branch containing three or more stages. Here a mixer is a token-mixing stage, such as attention or the Mamba block in Nemotron-H. | Responsibility | Owner | | - | - | | Attention, projections, activation functions, expert routing, EP dispatch/combine | Model compute modules | | How a producer contributes to the residual | Producer's `ResidualUpdate` | | How a consumer reads the residual | Consumer's `ResidualReadout` and norm | | Supported row transitions and reduction ordering | Bound boundary paths | | Actual outstanding work and residual tensors | Per-forward `ResidualStream` | | Hardware-specific fused completion | Fusion providers | | Retention of auxiliary tensors | Auxiliary-output collector | EP all-to-all is an operation inside MoE computation, not an input preparation step. The boundary does describe the MoE's input/output rows and any output reduction that compute delegates to it. ## Core concepts ### Token layouts and sums `Layout.sharded` describes which supported token axes partition a rank's rows: * `ATTN_DP`: different attention data-parallel token sets. * `ATTN_CP`: context-parallel (CP) token partitions for an active CP batch. * `ATTN_TP`: token slices distributed across attention-TP ranks. Size-one axes are omitted. Layout is not a `DTensor` placement or a general cross-mesh redistribution engine. It does not describe arbitrary hidden-dimension sharding. The actual row ordering, padding, and per-rank counts come from batch metadata and the paired movement operations. A sum is described separately by `SumGroup`: attention TP, full TP, or the MoE output reduction group. Separating rows from sums lets the boundary express both an unreduced contribution and the rows on which its completion must land. ### Producer contracts and output state A declaration states how its compute cooperates with the boundary's reduction decision: | Contract or state | Meaning | | - | - | | `ProducerReduction.ALWAYS_PARTIAL` | Attention only (the `declare_attn()` default): compute always returns a contribution requiring a sum and publishes it with `finish()`; the next stage's input completes the sum. Build its output projection so that it never reduces (for example `reduce_results=False`) | | `ProducerReduction.EXIT_SCOPED` | The exit completes the sum or carries it to the next stage's input: every FFN, and `declare_attn(reduction=ProducerReduction.EXIT_SCOPED)` for a mixer | | `OutputContract.may_*` | Construction-time permission to leave the sum to a particular later step | | `Contribution.owed` | Work this specific output still requires | | `OwedOutput` | Model-facing handle preventing ordinary tensor use of that unfinished output | `UnreducedOutput` is an internal adapter form. `ResidualStream.record()` converts it into an owned contribution and an opaque model handle. `DeferredFinalize` represents producer-specific work such as a deferred MoE finalize; its `complete()` supplies an unfused completion path. A handoff is consumed by the next layer's input or, at the end of the stack, by a final norm that accepts it. ### Residual stream `ForwardBatch.residual_stream` carries the residual and the producer's actual contribution from one boundary to the next. A stream has three states: | State | Stored data | Next preparation | | - | - | - | | Initial | Neither residual nor contribution | Initialize residual through the consumer's `init_residual` | | Written | Residual, no pending contribution | Read input without applying another update | | Pending | Contribution, optionally an existing residual | Complete owed work, apply the producer update, then read | The model keeps the tensor or handle that each boundary call returns, and passes exactly that value to the next call. Rebinding a layer's output to a different object, or passing another microbatch's output, raises an error instead of applying the wrong pending work. The check compares object identity, so an in-place change to a complete tensor is not detected; do not modify layer outputs in place, and add extra contributions through `add_to_output()`. Communication completion and residual update are distinct steps. The `residual.batch` helpers expose them: | Helper | Effect | | - | - | | `complete_output(hidden_states, forward_batch)` | Finishes outstanding communication/finalize work and leaves the update pending | | `fold(hidden_states, forward_batch)` | Completes a plain contribution and folds it into the residual, leaving a written stream | | `set_written(hidden_states, forward_batch)` | Replaces the stream with a written one whose residual is `hidden_states`. Use it after compute that produced the full residual itself. It discards any pending contribution, so call `fold()` first when one may be pending | | `take_output(hidden_states, forward_batch)` | Releases the stream at a terminal exit whose output is already written, such as Nemotron-H MTP after its internal fold and norm. It rejects a pending contribution | Always retain the returned tensor or handle after each of these calls. A stack-entry boundary accepts a written stream and skips its `init_residual` operation. ### Residual update and input read The producer owns `ResidualUpdate`; the consumer owns `ResidualReadout` and its norm. The actual update travels with the contribution. Bound paths use declared update capabilities to select valid ordering: * A plain add can participate in compatible fused add+norm kernels or add the residual on one rank before summation. * A nonlinear update cannot generally move before reduction. The post operation of manifold-constrained hyper-connections (MHC) is one such case. * `applied_at_exit` requests an update at the producer exit. A deferred update must guarantee that its parameters and state outlive that producer. * `ResidualReadout.reads_before_dp_gather` preserves reads that must execute on the source rows before a DP gather. * A read that needs every row of the attention's, such as one that adds a contribution computed for every row, declares `reads_after_attn_tp_gather`. A producer that would leave its output on this rank's attention-TP slice then returns it to the attention's rows; Qwen4-exp's per-layer embedding read and Kimi K3's attention-residual bank declare it. A read returns `(compute_input, residual)`. `update_and_read()` receives the producer's update together with the read, so an implementation can fuse them: the plain add+norm read does, and keeps its FP32 accumulation instead of normalizing a separately rounded snapshot. Fusion is not guaranteed. An implementation such as MHC may run the update and the read as separate steps, provided it preserves its existing rounding order. The MHC adapter implements the expanded residual and coefficient state used by its integrated models. Do not assume every MHC model uses it: DeepSeek V4 retains its model-specific MHC path. ## Integrate a model ### Declare a standard attention and FFN layer A `StageDeclaration` is the model-facing description. It contains read/update operations, dense versus sparse compute facts, reduction cooperation, and optional source declarations. It does not own a norm or an execution plan. The following constructor fragment assumes the model has already initialized its compute modules and norms, and the worker's runtime parallel configuration is available. It follows the dense Qwen3 integration: ```python theme={null} from sglang.srt.layers.layer_boundary import ( append_stages, declare_attn, declare_ffn, ) self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), (declare_ffn(), self.post_attention_layernorm), ) ``` `append_stages()` extends the open layer stack: the stack records the last stage appended, and the next append takes it as its producer, so a layer needs no description of its neighbours. `make_layers()` (and `make_pp_layers()`, which calls it) opens a stack around the layers it builds. A model that builds its layers another way, such as a hand-written `nn.ModuleList` or a NextN/multi-token-prediction (MTP) draft, opens the stack itself; appending outside an open stack raises: ```python theme={null} from sglang.srt.layers.layer_boundary import layer_stack with layer_stack(): self.layers = nn.ModuleList( DecoderLayer(config, layer_id) for layer_id in range(config.num_hidden_layers) ) ``` Every stage binds when the stack closes, once its producer and its consumer are known: the stage appended before it and the one appended after it. The last stage ends the model's layer stack and is marked terminal, unless a later layer declares a stage. A boundary's `declaration` is usable at once; when the stack closes, it is replaced by the declaration with its place in the stack (its `previous` and `terminal`), and the plan is bound. An error raised while binding carries a note naming the `append_stages()` call it came from. A terminal stage leaves no work for a following layer, except the final-norm finalize handoff described in [Enter and leave the layer stack](#enter-and-leave-the-layer-stack). A NextN/MTP draft module is its own stack: its layer enters the stack and ends it, even when it reuses the target's decoder class. Pipeline parallelism splits one layer stack across ranks, so the first and last local stages have neighbours that another rank builds. The model gives `make_layers()` (or `make_pp_layers()`) its shared declaration function, `stage_facts`: given a global layer index, it returns the stages that layer declares, in order (an empty tuple for a layer that declares none), from the config, the layer index, construction options known before the layer is built, and the fixed parallel topology, never from built modules or provider state. The layer declares its own stages with the same function, so both ranks derive a boundary from one source; under pipeline parallelism `make_layers()` also checks each layer it builds against the function. When the stack closes, it asks the function for the stages next to it, nearest layer first, passing over a layer that declares none, and keeps only what binding reads of them (`facts_of()`: the capabilities a read, update and output transform declare, not the objects). A model whose layers declare stages must give the function to run under pipeline parallelism; no rank builds a layer another rank holds. Each item may include a third element containing constructor options: an attention stage accepts `qkv_latent_func` (for example `{"qkv_latent_func": self.self_attn.prepare_qkv_latent}`) and `fusions`; an FFN stage accepts only `fusions` (see [Fusion providers](#fusion-providers-and-output-decisions)). An option a stage does not take is rejected when appended. Pass declarations fresh from `declare_attn()` / `declare_ffn()`: one taken from a bound boundary already has its place in a stack, and is rejected. ### Single-stage mixers and heterogeneous stacks For heterogeneous layer stacks, append the actual stages in order; the framework does not require an alternating attention/FFN pattern, and a layer may append a single stage. A single-stage `ProducerReduction.EXIT_SCOPED` mixer's consumer decides whether the mixer leaves its attention-TP sum to the next stage: always before an FFN, and per batch before another mixer. The stack binds the mixer only once its consumer is known, so the two stages always agree, across a pipeline boundary too: there the consumer is the next rank's first stage, which `make_layers()` reads from the neighbouring layer. At a pipeline boundary `to_pp()` completes the sum before the handoff, and the receiving FFN takes the completed path its entry binds beside the one for the declared sum. `models/nemotron_h_utils.py` (used by `models/nemotron_h.py`) builds each Nemotron-H stage as `append_stages((decl, norm))`, with mixers declared as `declare_attn(reduction=ProducerReduction.EXIT_SCOPED, gathers_attn_tp_input=False)`. ### More than two stages and branches There is no two-stage limit. For example, LongCat constructs a dense FFN, attention, and another dense FFN from an input already prepared for its MoE branch: ```python theme={null} first_ffn, second_attn, second_ffn = append_stages( (declare_ffn(), first_ffn_norm), (declare_attn(), second_attn_norm), (declare_ffn(), second_ffn_norm), prepared_from=moe_boundary.declaration, ) ``` This fragment assumes the source boundary and the three norms already exist. `prepared_from` means the source has already performed its read. A branch is a side path that the model merges back explicitly: it neither extends the stack nor waits for a consumer, so the next append still follows the source. Enter through `branch_input(source, hidden_states, forward_batch)`, not `prepare()`, to move that input and fork the residual stream without normalizing twice. The first stage's norm is therefore never applied; LongCat passes the MoE stage's norm there. The branch adapter also provides `branch_output()` to complete the sum a branch's contribution owes and place it on the branch's exit rows and `merge_branch()` to combine it with another branch's output. These methods encode row movement and residual ownership; the model still determines its compute schedule. Branch transport currently supports only ordinary token rows: construction rejects context-parallel, input-scattered, and sequence-parallel variants. See `python/sglang/srt/models/longcat_flash.py` for a complete integration. ### Execute a layer The corresponding forward fragment keeps computation in the model: ```python theme={null} hidden_states = self.attn_boundary.prepare(hidden_states, forward_batch) if hidden_states.shape[0] != 0: hidden_states = self.self_attn( positions=positions, hidden_states=hidden_states, forward_batch=forward_batch, ) hidden_states = self.attn_boundary.finish(hidden_states, forward_batch) hidden_states = self.ffn_boundary.prepare(hidden_states, forward_batch) hidden_states = self.mlp(hidden_states, forward_batch=forward_batch) hidden_states = self.ffn_boundary.finish(hidden_states, forward_batch) ``` This example uses Qwen3's compute member name `mlp`; boundary construction does not require renaming compute classes or checkpoint parameters. `prepare()` returns input in the consumer read's format, which can include quantized tensors. An attention with an `ALWAYS_PARTIAL` output contract uses `finish()` to register its contribution, and an FFN or single-stage mixer uses `finish()` to complete or carry on the sum its output owes. An FFN whose boundary has fusions, which may take a MoE finalize handoff, runs its compute under `exit()` instead and calls the returned object's `finish()` once (`finish()` on the boundary rejects it); the exit context restores the previous runtime flags even when compute raises, so do not call `finish()` on a failed computation. Compute never sums a stage's output: build every output projection, MLP and MoE block of a stage with `reduce_results=False`, and leave the sum the output owes to the boundary. The boundary completes it at the exit, or carries it to the next stage's input, where a fused all-reduce + add + norm may take it: * `FfnExit` decides before compute runs whether the exit completes the sum, a reduce-scatter on the way back completes it, or the next input does. It publishes only `defer_moe_finalize` on `get_forward()`: a MoE may then return a `DeferredFinalize`. * `MixerExit`: before an FFN, the mixer always leaves its attention-TP sum to the FFN's input, including under attention DP. Before another mixer the exit may carry the sum to that mixer's input, but never under attention DP; otherwise it completes the sum itself. One exception: when an FFN's output sum is fused with a reduction its computation needs anyway, so the two cannot be separated without an extra collective, the computation completes it and the stage declares `declare_ffn(output_complete=True)`; the exit then owes no sum. Kimi K3's latent MoE reduces its latent before the latent norm and completes the shared experts' sum beside it. A component every TP rank computes in full (a shared expert replicated with `tp_size=1`, LoRA-B on a partial sum) is added on TP rank 0 only, so that the one sum counts it once. Model loading fails if a module under a stage boundary is built with `reduce_results=True`. Returning a handoff while `defer_moe_finalize` is false fails in `finish()`. A complete tensor after `finish()` can still have a pending **residual update**. An incomplete reduction is exposed to the model as an opaque `OwedOutput`; pass it to the next boundary or a supported access method. Do not inspect it as a tensor, slice it, or unwrap its contribution in model code. ### Enter and leave the layer stack Initialize a fresh stream once on the embedding path, and leave the stack through the final norm: ```python theme={null} from sglang.srt.layers.layer_boundary.residual import batch as residual_batch residual_batch.start(forward_batch) hidden_states = self.embed_tokens(input_ids) for layer in self.layers: hidden_states = layer(positions, hidden_states, forward_batch) hidden_states = residual_batch.final_norm( hidden_states, forward_batch, self.norm, skip_empty=True ) ``` `skip_empty=True` still completes owed work on an empty batch, which may be a collective that other ranks join, and skips only the norm kernel. `final_norm()` and `to_pp()` release the batch's stream after they hand on its output. When the final read is more than a norm, pass it to `make_layers(final_read=...)` (or `layer_stack(final_read=...)`): it is then the last stage's consumer. Its `attn_tp_gather` gathers the rows the last FFN leaves on this rank's attention-TP slice, and with `reads_attn_tp_slices` it reads that slice itself and the last FFN leaves its output there. Kimi K3's `AttnBankOutputRead` is such a read. This is a non-pipeline stack fragment. Pipeline reception uses the first boundary's `from_pp(tensors, forward_batch)` to reconstruct a stream; the sender uses `residual_batch.to_pp()`. A stream the producer already wrote travels as `hidden_states` alone and arrives written: MHC's expanded streams, and the output of an FFN on its own rows (its attention-TP slice, or an unpadded batch's rows), which writes it at its exit before a pipeline handoff. The receiver knows this from the producer's declaration, not from which tensors arrive, since a captured CUDA graph's input buffers hold every key. `to_pp()` completes every sum the output owes before transport, a declared one too: the pipeline sends each tensor as one slice per attention-TP rank and gathers the slices back on the next rank, which reassembles only a tensor every attention-TP rank holds alike. The receiving stage's entry then takes the path for a sum already completed, which it binds beside the one for a sum its own producer leaves. A terminal FFN normally completes its output before the final norm. The exception is a MoE finalize handoff. When the stage's fusion provider can defer the finalize at the terminal layer (for CuteDSL, `install_cutedsl_fusion(..., terminal_finalize=True)`), the exit may leave a `DeferredFinalize` for the final norm. Pass the handle straight to `residual_batch.final_norm(hidden_states, forward_batch, self.norm, finalize_norm=service)`, where `service` is the object `install_cutedsl_fusion()` returned; its `finalize(handoff, residual, gamma)` fuses finalize, all-reduce, residual add, and norm. The fused path reads `layernorm.gemma_weight`, so it currently expects a `GemmaRMSNorm` final norm. Without `finalize_norm`, `final_norm()` first completes the handoff unfused, and `capture` is then allowed. With `finalize_norm`, `final_norm()` rejects `capture` when a handoff arrives. Calling `complete_output()` first runs the producer's unfused finalize, and `snapshot()` raises on a handoff. See `python/sglang/srt/models/qwen3_5.py`. `model_forward_stages()` in `python/sglang/srt/batch_overlap/two_batch_overlap.py` gives each TBO microbatch its own stream: it splits with `ResidualStream.from_handoff()` and merges the results afterwards. Do not share one mutable stream between interleaved microbatches or hold an `exit()` scope open across a TBO yield point. In the operation-scheduled TBO path, the FFN runs without an `exit()` scope and returns its partial output. `boundary.complete_now(hidden_states, forward_batch)` completes the sum it owes, applies any output transform, moves that output back to the layer's exit rows (a plain scatter under attention DP), and leaves nothing owed. ### Auxiliary capture and extra contributions Choose the access method according to what the consumer needs: | Operation | Effect on main output | | - | - | | `residual_batch.snapshot()` | Produces an owned plain residual snapshot; a sum can run on a copy without consuming the main contribution | | Attention `boundary.prepare(..., capture=collector.capture)` | Completes the main output's owed communication (unfused; the add+norm kernel still runs), then captures the updated residual this read produces, moved back to the producer's rows. At stack entry it captures the input before `init_residual`; with `post_residual_addition` it captures output plus residual before the addition. FFN `prepare()` does not accept it | | Attention `boundary.prepare(..., capture_gathered=collector)` | Captures the same updated residual on the attention's compute-input rows, after its input gather | | `boundary.capture_output()` | Preserves its explicit capture convention: completes dynamic work on the main contribution, but snapshots a statically declared sum on a copy; returns both updated handle and snapshot | | `residual_batch.add_to_output(..., extra)` | Completes the contribution before adding extra once, keeping the residual update pending | | `residual_batch.final_norm(..., capture=collector.capture)` | Final norm and capture of its updated residual | Snapshots currently require a plain residual update. Producer-specific finalize handoffs need an explicit main-output capture adapter. A separate BF16 snapshot followed by norm can change results relative to a fused add+norm kernel's FP32 accumulation. Capture from the existing update/read operation when that ordering matters. A capture callback has the signature `capture(value, *, owned=False)`, as `AuxHiddenStateList.capture` does. With `owned=False` the value may alias the mutable residual or a reusable communication buffer, and the collector must copy it before retaining it. The boundary passes `owned=True` only for storage nothing else will write: a capture move that gathers rows (`EntryPath.capture_move_allocates`), a residual sum computed only for the capture, or a residual that the following local stage's input path certifies it leaves untouched. `append_stages()` copies that certificate into the producer's entry at construction; no plan is consulted during forward. `residual_batch.final_norm(..., capture=...)` always passes a borrowed value. ### Choose a reduction policy The `--boundary-reduction` option controls optional FFN output transport choices for decoders built with these boundaries; other models ignore it: | Value | Permitted optional path | | - | - | | `ar` | All-reduce path | | `rs` | Fixed-size reduce-scatter where eligible | | `rsv` | Variable-size attention-DP reduce-scatter (RSv) where eligible | | `rs+rsv` | Both scatter choices where eligible, with RSv tried first | | `auto` | Default. Resolves a per-architecture policy during server-argument processing; a separate draft checkpoint resolves its own `speculative_boundary_reduction`. See [FFN boundary reduction](/docs/advanced_features/server_arguments#ffn-boundary-reduction) | Required attention and MoE collectives are unaffected. These are permissions, not commands to force unsupported collectives; an ineligible optional scatter falls back to the all-reduce path. Shape, topology, producer behavior, and required token movement still constrain the selected path. The resolved policy must be available before model construction. Model defaults are defined in `python/sglang/srt/arg_groups/boundary_reduction.py`. Reduction fusion has its own backend enablement and eligibility checks; selecting `ar` alone does not disable all-reduce fusion. ### Integration checklist and common errors Before you open a pull request for a new model, check each item: * Every decoder layer appends its stages inside a layer stack: `make_layers()`, or `layer_stack()` around any other construction, including a NextN/MTP draft's own layer. * The model gives `make_layers()` its shared declaration function (`stage_facts`), and each layer declares its stages with it: another pipeline rank binds the stages next to its own from that function alone. * The model forward calls `residual_batch.start()` before the first layer, and leaves the stack through `residual_batch.final_norm()`, `to_pp()`, or `take_output()`. * Pipeline reception calls `from_pp(tensors, forward_batch)` on the first local boundary. * Split prefill starts the stream only in the segment that runs the first layer and completes it only in the final segment. * Extra contributions such as deepstack embeddings go through `residual_batch.add_to_output()`, never an in-place add on the layer output. * Auxiliary capture uses `prepare(..., capture=...)` or `final_norm(..., capture=...)` with a callback that honours `owned`. Common errors and their usual cause: | Error | Usual cause | | - | - | | `start the layer stack before entering a stage` | The forward path, a split-prefill segment, or an MTP head skipped `residual_batch.start()` | | `output does not belong to this residual stream; change layer outputs through boundary accessors` | The model replaced a layer's output object (for example rebinding it to a new tensor) or passed another microbatch's output. This check compares object identity, so it does not catch in-place changes; the interface still forbids modifying layer outputs in place | | `write the residual update before taking the final output` | `take_output()` ran while a contribution was still pending; fold or norm it first | | `no stage boundary path for the active batch` | The active batch variant (CP, input-scattered, sequence parallel, unpadded) is not supported by this stage; add a server-argument check that rejects the configuration | | `append_stages needs an open layer stack` | The layers are built outside `make_layers()`; wrap their construction in `layer_stack()` | | `a prepared branch must enter through branch_input, not prepare` | A `prepared_from` stage was entered with `prepare()` | | `snapshot requires a plain residual update` | A snapshot was taken on a nonlinear (for example MHC) update | ## Boundary internals ### Architecture ```mermaid theme={null} flowchart TD D["Stage
declarations"] --> C["Connections
per batch
variant"] C --> P["StagePlan
entry and
exit paths"] P --> B["StageBoundary"] F["ForwardBatch
runtime facts"] --> B S["ResidualStream
residual and
pending work"] <--> B B --> I["prepare()
complete,
update, read,
move rows"] I -->|"FFN or mixer"| X["exit()
choose
completion"] X --> M["Model
computation"] I -->|"ALWAYS_PARTIAL"| M M --> O["finish()
record the
contribution"] O --> S ``` Construction and execution have different lifetimes: 1. **At model initialization**, `declare_attn()` and `declare_ffn()` describe compute behavior. `append_stages()` connects those declarations in stack order and binds each stage's own norm and hooks. 2. **During construction**, factories resolve token rows and reduction contracts for supported batch variants. `bind_entry()` binds consumer work; `bind_exit()` binds output transport. `StagePlan` retains those paths. 3. **At each forward**, batch facts select an existing path. An exit can make a batch-dependent completion decision, but it does not rebuild the declarations. The decision supplies the matching completion action; the only flag it publishes to compute is `defer_moe_finalize`, for a MoE that may hand off its finalize. 4. **Across stages**, `ForwardBatch.residual_stream` carries residual state and the producer's actual contribution. Stages do not store forward tensors on their reusable plans. The resolved `StageContract` is a lower-level type than `StageDeclaration`: one `InputContract` and one `OutputContract` after topology and batch variant have been applied. An `EdgeContract` combines a producer output, consumer requirement, and the residual's source/destination layouts. ### Module map Paths in this table are relative to `python/sglang/srt/layers/layer_boundary/`. | Module | Role | | - | - | | `factories.py` | Model declarations, local sequence assembly, and stage factories | | `contracts.py` | Resolved input/output contracts, edge declarations, batch variants, and bound step records | | `layout.py` | Token axes, layouts, named sum groups, and topology/batch facts | | `construction.py` | Build a `StagePlan` and select its batch variant | | `boundary.py` | Select and bind producer/consumer boundary operations from contracts | | `prepare.py` | Execute completion, update, read, and input movement in the selected order | | `ops.py` | Collective and token-row movement primitives | | `stage.py` | Model-facing `StageBoundary` methods | | `exit.py` | Select output completion and publish scoped compute flags | | `output.py` | Unreduced outputs, deferred finalizes and output transforms | | `residual/__init__.py` | `ResidualUpdate`/`ResidualReadout` protocols and `LayerResidualOps` | | `residual/stream.py` | Per-forward state and ownership of outstanding work | | `residual/batch.py` | Model-stack entry, final norm, pipeline-parallel (PP) transport, snapshots, and extra contributions | | `residual/access.py` | Tensor access and materialization helpers | | `residual/add_norm.py`, `residual/mhc.py` | Plain add/norm and MHC read/update implementations | | `adapters/` | Attention input, CP, TBO, branches, and LoRA integration | | `fusions/` | Consumer all-reduce fusion and CuteDSL integration | Models normally use factories, `StageBoundary`, and `residual.batch`. Keep row arithmetic, process-group selection, and calls such as `token_axis_sizes()` inside the boundary implementation. ### Batch variants `StagePlan` preconstructs ordinary, CP, input-scattered, sequence-parallel, and unpadded variants when supported by the configured topology. The unpadded variant serves batches whose rows do not divide over attention TP, which `--disable-attn-tp-gather` lets through without attention DP: an FFN that would run on this rank's attention-TP slice stays on the attention's rows instead. A forward selects a path from active runtime facts. An absent active variant raises an error rather than silently using ordinary token rows. Configuration validation rejects unsupported model/parallel combinations. Boundary construction also rejects transitions without a correct implementation. Adding an enum value or a layout does not by itself provide a new collective path. ### Fusion providers and output decisions Each exit has its own policy, and what it decides from fixed facts (the parallel configuration, the declarations and the bound path) is decided when the plan is built and recorded on the path as `ExitFacts`; only what depends on the batch is decided for it: | Exit | Entry point | May decide | | - | - | - | | FFN | `exit()` or `finish()` | Complete the sum now, possibly in an attention-DP or CP / input-scattered reduce-scatter on the way back; leave it to the next input (an all-reduce, or under attention DP the scatter back to this rank's tokens); hand a MoE finalize to the next input | | Operation-scheduled (TBO) | `complete_now()` | Only complete it now: under attention DP the move back is a plain scatter; never defers or hands off | | Mixer | `finish()` | Leave its declared sum to the next stage's input (before an FFN); carry it to the next mixer's input (without attention DP); or complete it | | Stack end | `exit()` or `finish()` | Complete it now; a MoE finalize handoff may reach a final norm that accepts it | | Pipeline handoff | `exit()` or `finish()`, then `to_pp()` | Complete it now; an FFN on its own rows also writes it into the residual | Fixed per path: whether an FFN may leave its sum to the next input at all, whether a reduce-scatter completes it without an attention-DP one, which sum an input-scattered batch's next input completes, whether the attention-DP return is the reduce-scatterv, whether the post-expert sum is one all-reduce, and whether a mixer carries its sum. Decided per batch: its padding, whether the attention-DP reduce-scatter is usable (an elastic EP scale-up changes the DP group), whether it has tokens, and whether a fused consumer takes the sum. An output decision is made once. Its flags control whether compute performs reduction/finalize work; the matching completion action either executes that work or records exactly what the next boundary owes. Do not set a skip flag independently of a completion action. A backend fusion provider is passed per stage as the `fusions` option, `(declaration, norm, {"fusions": provider})`; give a layer's attention and FFN stages the same provider. The boundary calls it as follows: * `attn_input_fusions(plan)` and `ffn_input_fusions(plan)` return ordered consumer candidates, tried before the built-in all-reduce fusion. * `can_defer_finalize(plan, forward_batch)` decides whether an FFN may return a `DeferredFinalize`. It is called on every FFN exit that has a provider. * `can_defer_all_reduce(plan, forward_batch)` lets a LoRA or `SGLANG_SHARED_EXPERT_TP1` FFN leave its all-reduce to the next input. Such an FFN defers only without an expert all-to-all backend or quantized communication, when its sum is over the TP group, and when the provider, FlashInfer fusion, or AITER fusion accepts the batch. Other FFNs defer when the post-expert sum is a single all-reduce. `install_cutedsl_fusion()` and `prepare_cutedsl_fusion()` find providers through the decoder layer's `attn_boundary` and `ffn_boundary` attributes. Consumer fusion candidates are ordered. A candidate returning `None` must do so before modifying inputs or starting a collective, so fallback remains valid. `FfnInputFusion` declares the sum group it completes with residual add and norm. A residual read may supply its own candidates as `completing_fusions`: each `ReadoutFusion` completes a sum with the residual add, ahead of the read, and is tried before the built-in collective; one with `reads=True` also does the read. Where a stage reads its input on this rank's attention-TP slice and then gathers it, a read's `gathering_reads` may do both in one kernel. A stage may declare its own implementation of the gather over attention TP that brings rows into it (`declare_attn(attn_tp_gather=...)`), tried on every batch before the built-in gather: at its own entry, or at its producer's exit, since an exit runs its consumer's gather. Across a pipeline boundary the producer's rank takes that gather from the neighbouring stage's declaration, so the gather may use only the communication state of the rank it runs on, not the declaring layer's weights or buffers. The consumer only tries compatible candidates for the declared update/read capabilities. An `OutputTransform` declares contribution processing before residual update. Its explicit `before_reduce_scatter` setting preserves a supported implementation's order; it is not permission to commute arbitrary operations across a reduction. Producer GEMM plus reduce-scatter fusion is not a generic implemented API here. When adding such a backend, its output contract and actual completion must describe both the finished sum and returned token rows; do not leave the next boundary believing it still owes either operation. ### Special adapters * **Attention/DeepSeek Sparse Attention (DSA) adapters:** retain `AttnTpContext` and its input-scattered/latent-input hooks. They are not a general asynchronous dependency scheduler. * **CP adapters:** pair gather, return-to-local-rows, and eligible reduce-scatter operations using the same token mapping. * **LoRA adapter:** publishes the selected compute input's rows for LoRA execution. * **TBO and branch adapters:** manage layer exits and entries that cannot be represented as a normal sequential prepare call. ### Extend and validate the module Place changes according to their responsibility: 1. Add model compute facts to a declaration only when they describe compute behavior. Put user policy and unsupported configuration checks in server-argument handling. 2. Add new update/read semantics in `residual/`; declare only capabilities the implementation really supports. 3. Add a token transition in boundary selection and movement primitives, with its padding and row-order contract. Do not duplicate that logic in each model. 4. Put backend completion in `fusions/` and specialized exit and entry behavior in `adapters/`. 5. Keep models on stage factories and boundary methods. Avoid reading a neighbour's execution plan or reconstructing a collective decision during forward. Focused tests live in `test/registered/unit/layer_boundary/`. Cover numerical output and residual state, collective group/row correctness, and supported fallback behavior for the changed path. Relevant examples include `test_ffn_exit.py`, `test_reduction_fusion.py`, and `test_append_stages.py`. A declaration-only assertion cannot establish that a new parallel path moves and reduces the correct values. # MSProbe Debugging Guide Source: https://docs.sglang.io/docs/developer_guide/msprobe_debugging_guide MSProbe is a debugging tool for AI models that diagnoses accuracy anomalies and numerical errors during model training and inference. It captures and monitors intermediate data (feature maps, weights, activations, layer outputs) and contextual metadata (prompts, tensor dtypes, hardware configuration), and supports visual analysis to systematically trace the root cause of accuracy degradation or numerical errors (e.g., NaN/Inf, output drift, mismatched predictions). ## Basic Details ### Background Concepts: MSProbe Dumping Levels MSProbe supports three accuracy levels for data dumping, each for different debugging needs: * **L0**: Dumps tensors/statistics at the **module level** and generates `construct.json` (for network structure reconstruction in visualization). Requires passing a model/submodule handle. * **L1**: Dumps tensors/statistics at the **torch API level**, suitable for fine-grained API-level numerical checking. * **mix**: Combines L0 + L1, ideal for scenarios that require both **graph reconstruction** and **numerical comparison**. ### Prerequisites: Install MSProbe Install MSProbe with pip: ```shell theme={null} pip install mindstudio-probe --pre ``` ### Key Configuration Parameters MSProbe uses a JSON configuration file for customized data dumping. All core parameters are listed in the table below, with the default JSON configuration provided for reference. #### Configuration Parameter Table | Field | Description | Required | | :-: | :- | :-: | | `task` | Type of dump task. Common PyTorch values include `"statistics"` and `"tensor"`. A statistics task collects tensor statistics (mean, variance, max, min, etc.) while a tensor task captures arbitrary tensors. | Yes | | `dump_path` | Directory where dump results are stored. When omitted, `MSProbe` uses its default path. | No | | `rank` | Ranks to sample. An empty list collects every rank. For single-card tasks you must set this field to `[]`. | No | | `step` | Token iteration(s) to sample. An empty list means every iteration. | No | | `level` | Dump level string (`"L0"`, `"L1"`, or `"mix"`). `L0` targets `nn.Module`, `L1` targets `torch.api`, and `mix` collects both. | Yes | | `async_dump` | Whether to enable asynchronous dump (supported for PyTorch `statistics`/`tensor` tasks). Defaults to `false`. | No | | `scope` | Customize the scope of dump. Provide two module or API names that follow the tool's naming convention to lock a range, only data between the two names will be dumped. An empty list dumps every module or torch API.

Examples:
`"scope": ["Module.conv1.Conv2d.forward.0", "Module.fc2.Linear.forward.0"]`
`"scope": ["Tensor.add.0.forward", "Functional.square.2.forward"]`

The `level` setting determines what can be provided—modules when `level=L0`, APIs when `level=L1`, and either modules or APIs when `level=mix`. | No | | `list` | Customize dump list, only dumps elements from the list. An empty list dumps every module or torch API. Options include:

•Supply the full names of specific APIs in PyTorch eager mode to only dump those APIs. Example: `"list": ["Tensor.permute.1.forward", "Tensor.transpose.2.forward", "Torch.relu.3.backward"]`.
•When `level=mix`, you can provide module names so that the dump expands to everything produced while the module is running. Example: `"list": ["Module.module.language_model.encoder.layers.0.mlp.ParallelMlp.forward.0"]`.
•Provide a substring such as `"list": ["relu"]` to dump every API whose name contains the substring. When `level=mix`, modules whose names contain the substring are also expanded. | No | #### Default configuration ```json theme={null} { "task": "statistics", "dump_path": "./dump_path", "rank": [], "step": [], "level": "L1", "async_dump": false, "statistics": { "scope": [], "list": [], "data_mode": [ "all" ], "summary_mode": "statistics" }, "tensor": { "scope": [], "list": [], "data_mode": [ "all" ], "file_format": "npy" }, "acc_check": { "white_list": [], "black_list": [], "error_data_path": "./" } } ``` #### Outputs Dump files are written into `dump_path` you defined. They usually contain: * `dump.json`, which records metadata such as dtype, shape, min, max, mean, L2 norm, and `requires_grad`. * `construct.json`, hierarchical structure description, when `level` is `L0` or `mix` (required for visualization), its content is not empty. * `stack.json`, record the call stack information of API/Module. * `dump_tensor_data`, generated when `task` is `tensor` and save the collected tensor data. See [dump directory description](#dump-directory-description) for details. > **Note**: When MSProbe is enabled, cuda graph is disabled (disable\_cuda\_graph=True) because MSProbe only supports dump > in eager mode, warmup is disabled (skip\_server\_warmup=True) because there is no need to dump data for this stage. ## End-to-End Examples MSProbe’s full debugging workflow follows **Enable → Collect Data → Visualize → Analyze Root Cause**. Below is a common E2E example for SGLang-based model inference debugging. ### Example : Advanced Debugging with Custom Configuration Suitable for targeted debugging (e.g., only collect statistics data for specific ranks/steps, enable mix level for graph reconstruction + numerical comparison) and root cause analysis via **problem vs. benchmark comparison**. #### Step 1: Enable ##### Prepare Custom Configuration JSON Create `msprobe-config.json` (dump statistics data for rank0/1, step0/1, mix level): ```json theme={null} { "task": "statistics", "dump_path": "./problem_dump", "rank": [ 0, 1 ], "step": [ 0, 1 ], "level": "mix", "async_dump": false, "statistics": { "scope": [], "list": [], "data_mode": [ "all" ], "summary_mode": "statistics" } } ``` ##### Enable MSProbe with Custom Configuration in SGLang Launch the SGLang server and specify the configuration file path with `--msprobe-dump-config`: ```bash theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen2.5-0.5B-Instruct \ --host 127.0.0.1 \ --port 1027 \ --msprobe-dump-config /home/msprobe-config.json ``` #### Step 2: Collect Data ##### Collect Dump Data for Problem & Benchmark Sides Send normal inference requests to trigger model running (MSProbe automatically collects data during request processing): ```bash theme={null} curl -H "Content-type: application/json" \ -X POST \ -d '{ "model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [ { "role": "user", "content": "Hello, my name is" } ], "max_tokens": 10 }' \ http://127.0.0.1:1027/v1/chat/completions ``` * **Problem side**: Run the above SGLang server (with the accuracy/numerical issue) and send inference request; dump data is saved to `./problem_dump`. * **Benchmark side**: Launch a normal SGLang server (without the issue, e.g., stable framework version/operator) with the **same custom configuration** and send the **same inference request**; rename the dump directory to `./bench_dump`. > **Key Requirement**: Problem and benchmark dumps must use the same inputs and sampling points (rank/step) > for valid comparison. ##### Check Generated Dump Files Dump files are saved to `./problem_dump` and `./bench_dump` you defined and include core files for subsequent analysis: * `dump.json`: Records tensor metadata of APIs and modules (dtype, shape, min/max/mean, L2 norm, `requires_grad`, etc.). * `stack.json`: Logs call stack information of APIs and modules. * `construct.json`: hierarchical structure description, required for visualization, its content is not empty. #### Step 3: Visualize ##### Visualize Problem vs. Benchmark Comparison (Multi-Rank) Generate a multi-rank comparison visualization file (mix level generates `construct.json` for graph reconstruction): ```shell theme={null} msprobe graph_visualize -tp ./problem_dump/step0 -gp ./bench_dump/step0 -o ./graph_output ``` * `-tp`: Path to problem-side dump data * `-gp`: Path to benchmark-side dump data * `-o`: Output directory for visualization files If you want overflow check (for NaN/Inf detection), please specify the parameter `-oc` ```shell theme={null} msprobe graph_visualize -tp ./problem_dump/step0 -gp ./bench_dump/step0 -o ./graph_output -oc ``` After the comparison or build task finishes, a `compare_{timestamp}.vis.db` file is created under `graph_output`. The dumped data can be used to visualize and analyze differences using tables or charts generated with visualization tools such as Matplotlib and Excel. ##### Launch TensorBoard Start TensorBoard: ```bash theme={null} tensorboard --logdir ./graph_output --bind_all --port 6006 ``` #### Step 4: Analyze Root Cause ##### Locate Root Cause Root Cause Analysis in TensorBoard: * Divergent nodes (with accuracy/numerical differences) are highlighted in **red** (darker red = larger difference). * Click on divergent nodes to view detailed tensor data (inputs/outputs, parameters) and API/module call stacks. * Use the **search/filter** function to quickly locate key layers/APIs (e.g., "relu", "conv"). * Switch between ranks/steps via the UI to check cross-rank/cross-step divergence. * Check the **overflow check** tab for NaN/Inf values in specific nodes (the direct cause of numerical instability). ##### Verify the Root Cause After locating the divergent node (e.g., a specific Conv layer or torch API with abnormal tensor values), verify by: * Narrowing the dump scope to this node (via `scope`/`list` in the configuration file) for fine-grained data collection. * Modifying the problematic layer/API (e.g., replacing the operator, adjusting the dtype) and re-running the debugging workflow to confirm the issue is resolved. ## Troubleshooting ### No Dump Files Generated 1. To confirm if MSProbe is installed, use `pip show mindstudio-probe` to troubleshoot. If it is installed, the MSProbe version information will be printed. If it is confirmed that it has not been installed, please use `pip install mindstudio-probe --pre` for installation; 2. Confirm the `--msprobe-dump-config` parameter points to the **correct JSON file path**. ### Dump Files Are Too Large (Excessive Data) 1. Start with `task: "statistics"` instead of `"tensor"` to collect only tensor statistics (avoids raw tensor dump); 2. Narrow the dump range with the `scope` field (specify start/end module/API); 3. Filter dump targets with the `list` field (only dump specific modules/APIs or substrings); 4. Sample specific `rank` and `step` (avoid dumping all ranks/iterations). ### TensorBoard Visualization Fails 1. Confirm `construct.json` is not empty (requires `level: L0` or `mix` – L1 does not generate graph files); 2. Check that the `-tp` (problem dump) and `-gp` (benchmark dump) paths point to **valid rank/step subdirectories** ( e.g., `step0/rank0`); 3. Ensure the MSProbe version is up-to-date (reinstall with `pip install mindstudio-probe --pre --upgrade`); 4. Verify TensorBoard is installed and the `--logdir` parameter points to the directory containing `.vis.db` files (not the file itself). ### Numerical Comparison Shows No Divergence But Model Accuracy Is Low 1. Expand the dump `step` range (check more token iterations for late-stage divergence); 2. Switch to `task: "tensor"` (statistics may mask subtle numerical differences in raw tensor data); 3. Ensure the problem and benchmark dumps use **the same input data/hardware configuration** (different inputs lead to invalid comparisons); 4. Use the `manual mapping` feature in TensorBoard (automatic mapping may miss some nodes for custom models). *** ## Appendix ### Dump directory description ```text theme={null} ├── problem_dump or bench_dump │ ├── step0 │ │ ├── rank0 │ │ │ ├── dump_tensor_data │ │ │ │ ├── Tensor.permute.1.forward.pt │ │ │ │ ├── Functional.linear.5.backward.output.pt # Format: {api_type}.{api_name}.{call_count}.{forward/backward}.{input/output}.{arg_index}. │ │ │ │ │ # arg_index is the nth input or output of the API. If an input is a list, keep numbering with decimals (e.g., 1.1 is the first element of the first argument). │ │ │ │ ├── Module.conv1.Conv2d.forward.0.input.0.pt # Format: {Module}.{module_name}.{class_name}.{forward/backward}.{call_count}.{input/output}.{arg_index}. │ │ │ │ ├── Module.conv1.Conv2d.forward.0.parameters.bias.pt # Module parameter data: {Module}.{module_name}.{class_name}.forward.{call_count}.parameters.{parameter_name}. │ │ │ │ └── Module.conv1.Conv2d.parameters_grad.weight.pt # Module parameter gradients: {Module}.{module_name}.{class_name}.parameters_grad.{parameter_name}. Gradients do not include call_count because the same gradient updates all invocations. │ │ │ │ # When the `model` argument passed to dump is a List[torch.nn.Module] or Tuple[torch.nn.Module], module-level data names also include the index inside the list ({Module}.{index}.*), e.g., Module.0.conv1.Conv2d.forward.0.input.0.pt. │ │ │ ├── dump.json │ │ │ ├── stack.json │ │ │ ├── dump_error_info.log │ │ │ └── construct.json │ │ ├── rank1 │ │ │ ├── dump_tensor_data │ │ │ │ └── ... │ │ │ ├── dump.json │ │ │ ├── stack.json │ │ │ ├── dump_error_info.log │ │ │ └── construct.json │ │ ├── ... │ │ │ │ │ └── rank7 │ ├── step1 │ │ ├── ... │ ├── step2 ``` * `rank`: Device ID. Each card writes its data to the corresponding `rank{ID}` directory. In non-distributed scenarios the directory is simply named `rank`. * `dump_tensor_data`: Save the collected tensor data. * `dump.json`: Statistics for the forward data of each API or module, including names, dtype, shape, max, min, mean, L2 norm (square root of the L2 variance), and CRC-32 when `summary_mode="md5"`. See [dump.json file description](#dump-json-file-description) for details. * `dump_error_info.log`: Present only when the dump tool encountered an error and records the failure log. * `stack.json`: Call stacks for APIs/modules. * `construct.json`: Hierarchical structure description. Empty when `level=L1`. ### dump.json file description #### L0 level An L0 `dump.json` contains forward/backward I/O for modules together with parameters and parameter gradients. Using PyTorch's `Conv2d` as an example, the network code looks like: `output = self.conv2(input) # self.conv2 = torch.nn.Conv2d(64, 128, 5, padding=2, bias=True)` `dump.json` contains the following entries: * `Module.conv2.Conv2d.forward.0`: Forward data of the module. `input_args` represents positional inputs, `input_kwargs` represents keyword inputs, `output` stores forward outputs, and `parameters` stores weights/biases. * `Module.conv2.Conv2d.parameters_grad`: Parameter gradients (weight and bias). * `Module.conv2.Conv2d.backward.0`: Backward data of the module. `input` represents gradients that flow into the module (gradients of the forward outputs) and `output` represents gradients that flow out (gradients of the module inputs). **Note**: When the `model` parameter passed to the dump API is `List[torch.nn.Module]` or `Tuple[torch.nn.Module]`, module-level names include the index inside the list (`{Module}.{index}.*`). Example: `Module.0.conv1.Conv2d.forward.0`.
L0 dump.json ```json theme={null} { "task": "tensor", "level": "L0", "framework": "pytorch", "dump_data_dir": "/dump/path", "data": { "Module.conv2.Conv2d.forward.0": { "input_args": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 8, 16, 14, 14 ], "Max": 1.638758659362793, "Min": 0.0, "Mean": 0.2544615864753723, "Norm": 70.50277709960938, "requires_grad": true, "data_name": "Module.conv2.Conv2d.forward.0.input.0.pt" } ], "input_kwargs": {}, "output": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 8, 32, 10, 10 ], "Max": 1.6815717220306396, "Min": -1.5120246410369873, "Mean": -0.025344856083393097, "Norm": 149.65576171875, "requires_grad": true, "data_name": "Module.conv2.Conv2d.forward.0.output.0.pt" } ], "parameters": { "weight": { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32, 16, 5, 5 ], "Max": 0.05992485210299492, "Min": -0.05999220535159111, "Mean": -0.0006165213999338448, "Norm": 3.421217441558838, "requires_grad": true, "data_name": "Module.conv2.Conv2d.forward.0.parameters.weight.pt" }, "bias": { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32 ], "Max": 0.05744686722755432, "Min": -0.04894155263900757, "Mean": 0.006410328671336174, "Norm": 0.17263513803482056, "requires_grad": true, "data_name": "Module.conv2.Conv2d.forward.0.parameters.bias.pt" } } }, "Module.conv2.Conv2d.parameters_grad": { "weight": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32, 16, 5, 5 ], "Max": 0.018550323322415352, "Min": -0.008627401664853096, "Mean": 0.0006675920449197292, "Norm": 0.26084786653518677, "requires_grad": false, "data_name": "Module.conv2.Conv2d.parameters_grad.weight.pt" } ], "bias": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32 ], "Max": 0.014914230443537235, "Min": -0.006656786892563105, "Mean": 0.002657240955159068, "Norm": 0.029451673850417137, "requires_grad": false, "data_name": "Module.conv2.Conv2d.parameters_grad.bias.pt" } ] }, "Module.conv2.Conv2d.backward.0": { "input": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 8, 32, 10, 10 ], "Max": 0.0015069986693561077, "Min": -0.001139344065450132, "Mean": 3.3215508210560074e-06, "Norm": 0.020567523315548897, "requires_grad": false, "data_name": "Module.conv2.Conv2d.backward.0.input.0.pt" } ], "output": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 8, 16, 14, 14 ], "Max": 0.0007466732058674097, "Min": -0.00044813455315306783, "Mean": 6.814070275140693e-06, "Norm": 0.01474067009985447, "requires_grad": false, "data_name": "Module.conv2.Conv2d.backward.0.output.0.pt" } ] } } } ```
#### L1 level An L1 `dump.json` records forward/backward I/O for APIs. Using PyTorch's `relu` function as an example (`output = torch.nn.functional.relu(input)`), the file contains: * `Functional.relu.0.forward`: Forward data of the API. `input_args` are positional inputs, `input_kwargs` are keyword inputs, and `output` stores the forward outputs. * `Functional.relu.0.backward`: Backward data of the API. `input` represents the gradients of the forward outputs, and `output` represents the gradients that flow back to the forward inputs.
L1 dump.json ```json theme={null} { "task": "tensor", "level": "L1", "framework": "pytorch", "dump_data_dir": "/dump/path", "data": { "Functional.relu.0.forward": { "input_args": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32, 16, 28, 28 ], "Max": 1.3864083290100098, "Min": -1.3364859819412231, "Mean": 0.03711778670549393, "Norm": 236.20692443847656, "requires_grad": true, "data_name": "Functional.relu.0.forward.input.0.pt" } ], "input_kwargs": {}, "output": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32, 16, 28, 28 ], "Max": 1.3864083290100098, "Min": 0.0, "Mean": 0.16849493980407715, "Norm": 175.23345947265625, "requires_grad": true, "data_name": "Functional.relu.0.forward.output.0.pt" } ] }, "Functional.relu.0.backward": { "input": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32, 16, 28, 28 ], "Max": 0.0001815402356442064, "Min": -0.00013352684618439525, "Mean": 0.00011915402356442064, "Norm": 0.007598237134516239, "requires_grad": false, "data_name": "Functional.relu.0.backward.input.0.pt" } ], "output": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32, 16, 28, 28 ], "Max": 0.0001815402356442064, "Min": -0.00012117840378778055, "Mean": 2.0098118724831693e-08, "Norm": 0.006532244384288788, "requires_grad": false, "data_name": "Functional.relu.0.backward.output.0.pt" } ] } } } ```
#### mix level A `mix` dump.json contains both L0 and L1 level data; the file format is the same as the examples above. # Developer Guide Source: https://docs.sglang.io/docs/developer_guide/overview Contributing to SGLang — development setup, benchmarking, and evaluation. * [Contribution Guide](./contribution_guide) * [Add an out-of-tree serve backend](/docs/developer_guide/serve_backend_plugins) * [Development Guide (Docker)](./development_guide_using_docker) * [JIT Kernels](./development_jit_kernel_guide) * [Quantization Contribution Guide](./quantization_contribution_guide) * [Layer boundaries](/docs/developer_guide/layer_boundary) * [Benchmark and Profiling](./benchmark_and_profiling) * [Bench Serving](./bench_serving) * [Evaluating New Models](./evaluating_new_models) * [MSProbe Debugging Guide](./msprobe_debugging_guide) # Quantization Contribution Guide Source: https://docs.sglang.io/docs/developer_guide/quantization_contribution_guide This guide describes how to add or refactor quantization support in SGLang. It focuses on the common structure used by weight-only and weight-activation quantization methods such as AWQ, GPTQ, compressed-tensors, ModelSlim, Quark, and related backend kernels. ## Design Goals Quantization code should keep the quantization format semantics separate from hardware-specific execution. This makes it easier to add new formats, reuse kernels across formats, and review platform-specific changes independently. Follow the architecture proposed in [Quantization Modifications](https://github.com/sgl-project/sglang/issues/15194): * **Config**: parses model and runtime quantization parameters, validates supported options, and selects the proper scheme. * **Scheme**: owns quantized weight creation, weight loading, post-processing, and quantized layer wiring for Linear, MoE, embedding, or other module types. * **Backend kernel**: wraps hardware-specific execution, layout conversion, kernel selection, and kernel calls for GPU (CUDA/HIP/XPU), NPU, or other backends. Avoid putting config parsing, weight loading, and backend kernel calls in a single monolithic file. If a method needs multiple formats or backends, add a package under `python/sglang/srt/layers/quantization//` and split schemes into `schemes/`. ## Recommended File Layout Use this layout for a quantization method that has multiple schemes or backend-specific execution paths: ```text theme={null} python/sglang/srt/layers/quantization// __init__.py .py schemes/ __init__.py _scheme.py _linear.py _moe.py _.py ``` Backend kernels should live under the hardware backend they target: ```text theme={null} python/sglang/srt/hardware_backend/gpu/quantization/_kernels.py python/sglang/srt/hardware_backend/npu/quantization/_kernels.py ``` Keep shared method selection in the quantization package and keep backend imports narrow. This prevents circular imports and keeps non-target platforms from importing unavailable kernel dependencies. ## Adding or Refactoring a Quantization Method 1. Define the config entry point and register it through `python/sglang/srt/layers/quantization/__init__.py` when needed. 2. Add explicit scheme selection helpers such as `get_linear_scheme` and `get_moe_scheme`. 3. Move layer-specific weight creation and weight loading into scheme classes. 4. Move GPU (CUDA/HIP/XPU), NPU, or other hardware kernel calls into backend kernel modules. 5. Keep Linear, MoE, embedding, and non-linear module handling explicit. Do not assign a Linear quantization method to a module type that needs different semantics. 6. Preserve compatibility for existing quantized checkpoints and runtime flags. 7. Add tests that cover both config parsing and execution paths touched by the change. For examples, see the AWQ and GPTQ refactors: * [PR #21126](https://github.com/sgl-project/sglang/pull/21126): splits AWQ schemes, weight initialization, and backend kernel calls. * [PR #26402](https://github.com/sgl-project/sglang/pull/26402): applies the same scheme/kernel split to GPTQ. ## Tests and Validation Quantization changes can affect both accuracy and performance. Include validation that matches the blast radius of the change. For Python-only structure changes: ```bash theme={null} ruff check git diff --check ``` For quantized model behavior: * Launch at least one representative model for each touched quantization method. * Send a `/generate` request and confirm the output path succeeds. * Run an accuracy sanity test if the change can affect numerics. * Include warmup-aware benchmark results when the change affects kernel calls, layout conversion, or dispatch. For backend-specific changes: * Validate GPU changes on a supported GPU environment (NVIDIA, AMD, or Intel). * Validate NPU changes on a supported Ascend environment. * Include the exact model, quantization flag, backend flag, hardware, and command used in the PR description. ## PR Checklist Before requesting review, make sure the PR description includes: * The quantization method and backend paths changed. * The issue, design proposal, or roadmap item the PR follows. * Any compatibility notes for existing checkpoints or flags. * Accuracy results when model outputs can change. * Benchmark or profiling results when runtime performance can change. * The exact local checks and model launch tests that were run. Use the general [Contribution Guide](./contribution_guide) for source setup, formatting, unit tests, CI triggering, and review process details. # Add an out-of-tree serve backend Source: https://docs.sglang.io/docs/developer_guide/serve_backend_plugins Connect an ecosystem runtime to sglang serve through the versioned serve backend plugin API. A serve backend plugin connects an out-of-tree runtime to the SGLang-owned CLI: ```bash theme={null} sglang serve MODEL_PATH --model-type BACKEND_NAME [BACKEND_OPTIONS] ``` The extension retains its own argument parser, process topology, API endpoints, hardware requirements, and release cycle. The core CLI owns backend selection and common child-process cleanup. ## Prerequisites * Install your extension and a compatible SGLang version in the same Python 3.10+ environment. * Declare the SGLang version range tested by your extension in its package dependencies. * Keep the backend factory and detector independent of GPU initialization and model loading. The plugin API itself is platform-independent. Your backend defines its supported operating systems, accelerators, parallelism options, and authentication requirements. ## Keep one owner for the executable Only the `sglang` distribution should publish a console script named `sglang`. Your extension registers package metadata under `sglang.serve_backends`; it must not publish another `sglang` script. This prevents installation order from replacing the command and ensures uninstalling an extension does not remove the core executable. You can retain a project-specific executable as a compatibility alias: ```bash theme={null} my-runtime serve MODEL_PATH sglang serve MODEL_PATH --model-type my_runtime ``` Both commands should call the same backend implementation. ## Register a backend factory Add a zero-argument factory to your extension's `pyproject.toml`: ```toml theme={null} [project] name = "my-sglang-runtime" version = "0.1.0" dependencies = ["sglang"] [project.entry-points."sglang.serve_backends"] my_runtime = "my_sglang_runtime.sglang_backend:create_backend" ``` The entry point name, `my_runtime`, becomes an accepted `--model-type` value. Choose a distinctive name. `auto` and SGLang's in-tree backend names are reserved. ## Implement the backend Create `my_sglang_runtime/sglang_backend.py`: ```python theme={null} import argparse from sglang.cli.serve_backends import ( ServeBackend, ServeBackendDetection, ServeRequest, ) def detect(request: ServeRequest) -> ServeBackendDetection: if request.model_path is None: return ServeBackendDetection.UNKNOWN if supports_model(request.model_path): return ServeBackendDetection.MATCH return ServeBackendDetection.NO_MATCH def run(request: ServeRequest) -> None: parser = argparse.ArgumentParser(prog="sglang serve") parser.add_argument("--model-path", required=True) parser.add_argument("--pipeline-parallel", type=int, default=1) args, remaining = parser.parse_known_args(request.argv) launch_runtime(args, remaining) def create_backend() -> ServeBackend: return ServeBackend(api_version=1, run=run, detect=detect) ``` Replace `supports_model()` and `launch_runtime()` with your extension's lightweight metadata check and blocking server launcher. A real `run()` call should block for the server lifetime. It must also honor `-h` and `--help` without launching a server; `argparse` does this automatically. Among serve backend entry points, explicit selection imports only the selected provider. Automatic selection loads installed backend factories and invokes their detectors, so importing this module and calling `detect()` must not initialize accelerators, import model weights, or start workers. ## Handle forwarded arguments SGLang removes `--model-type` and normalizes a positional Hugging Face model ID or local model directory before dispatch. For example: ```bash theme={null} sglang serve org/model --model-type my_runtime --pipeline-parallel 2 ``` Your backend receives: ```python theme={null} ("--model-path", "org/model", "--pipeline-parallel", "2") ``` The selected backend owns all remaining argument parsing and validation. Target backend-specific help with: ```bash theme={null} sglang serve --model-type my_runtime --help ``` ## Support config-only runtimes SGLang requires a model path by default. If your runtime resolves its model and parallelism settings from a configuration file, disable that validation: ```python theme={null} def create_backend() -> ServeBackend: return ServeBackend( api_version=1, run=run, detect=detect, requires_model_path=False, ) ``` You can then accept commands such as: ```bash theme={null} sglang serve --model-type my_runtime --config pipeline.yaml ``` Config-only requests generally require explicit `--model-type` unless your detector can identify the backend from the remaining arguments. ## Understand automatic routing The default `--model-type auto` follows these rules: 1. Backends without a detector remain explicit-only. 2. One `MATCH` selects that backend. 3. Multiple matches fail and require an explicit `--model-type`. 4. `UNKNOWN` and detector failures do not claim the request. 5. No matches preserve the existing LLM fallback. The registry does not resolve overlap by package installation order or a hidden priority. A backend can opt out of automatic routing by omitting its detector: ```python theme={null} ServeBackend(api_version=1, run=run, requires_model_path=False) ``` ## Maintain compatibility Declare the API version implemented by your extension as a literal. Do not copy SGLang's current version constant at runtime; a fixed value lets a future SGLang release detect an older plugin contract. SGLang rejects incompatible, duplicate, and reserved backend registrations with an actionable error. The public extension contract consists of: * `ServeRequest`: normalized backend arguments and the optional model path * `ServeBackend`: the runner, optional detector, and model-path requirement * `ServeBackendDetection`: `MATCH`, `NO_MATCH`, or `UNKNOWN` Test explicit selection, backend-specific help, automatic detection, ambiguous models, and installation or removal alongside the core `sglang` package. # Examples Source: https://docs.sglang.io/docs/get-started/examples Run a model locally, add chat to an application, process a batch of texts, or build a tool-using assistant. Choose an example for what you want to do: | I want to… | Example | | - | - | | Run a model on my own computer | [Local inference](#run-a-model-locally) on an NVIDIA GPU or Apple Silicon Mac. | | Serve a larger model on a GPU server | [Deploy Qwen3.8-27B](#serve-qwen3-8-27b) on an NVIDIA H200. | | Add chat to an application | [Stream a reply](#stream-a-reply) through the OpenAI-compatible API. | | Process a collection of documents | [Summarize multiple texts](#process-multiple-prompts) concurrently. | | Build an agent that uses tools | [Look up an order](#build-a-customer-support-agent) and answer a customer’s question. | ## Run a model locally Run `Qwen/Qwen3-0.6B` on your own computer and ask it a question. This small model is useful for trying the workflow; choose a larger model for more demanding tasks. This example uses one CUDA 13-compatible NVIDIA GPU, Linux, Docker, and NVIDIA Container Toolkit. For setup, see [Installation](/docs/get-started/install). Run this command on your computer. Docker pulls the image if needed, and SGLang downloads the model on first use: ```bash theme={null} docker run --rm --gpus all --ipc=host \ -p 127.0.0.1:30000:30000 \ lmsysorg/sglang:latest \ sglang serve Qwen/Qwen3-0.6B \ --host 0.0.0.0 --port 30000 ``` If you already installed SGLang with uv, run this in that Python environment instead: ```bash theme={null} sglang serve Qwen/Qwen3-0.6B --host 127.0.0.1 --port 30000 ``` On an Apple Silicon Mac with macOS 14 or newer, first follow the [Apple Silicon installation instructions](/docs/hardware-platforms/apple_metal#install-sglang) to install SGLang with its MLX dependencies. Use that Python environment to launch the model with 4-bit quantization: ```bash theme={null} SGLANG_USE_MLX=1 python -m sglang.launch_server \ --model-path Qwen/Qwen3-0.6B \ --quantization mlx_q4 \ --disable-cuda-graph \ --host 127.0.0.1 --port 30000 ``` The model runs on your Mac through MLX. It downloads the weights on first use and quantizes them in memory when loading. Keep the server running. Once you see `The server is fired up and ready to roll!`, open another terminal on the same computer and send a request: ```bash theme={null} curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "What is the capital of France? /no_think"}], "max_tokens": 128 }' ``` Look for the answer in `choices[0].message.content`. Both the model and the API server are running locally. Press `Ctrl+C` in the server terminal when you are done, or leave it running for the chat and batch examples below. ## Serve Qwen3.8-27B This configuration runs `Qwen/Qwen3.8-27B-FP8` on one NVIDIA H200. It requires Linux, a CUDA 13-compatible driver, Docker, and NVIDIA Container Toolkit configured for GPU access. For environment setup, see [Installation](/docs/get-started/install). For other hardware configurations, see the [Qwen3.8-27B Cookbook](/cookbook/autoregressive/Qwen/Qwen3.8-27B). Set `HF_CACHE_DIR` to an absolute directory on the host for downloaded model weights, replacing the placeholder below. Run this command on the GPU host: ```bash theme={null} HF_CACHE_DIR="/path/to/huggingface-cache" docker run --rm --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v "$HF_CACHE_DIR:/root/.cache/huggingface" \ --ipc=host \ lmsysorg/sglang:latest \ sglang serve \ --trust-remote-code \ --model-path Qwen/Qwen3.8-27B-FP8 \ --kv-cache-dtype fp8_e4m3 \ --mem-fraction-static 0.85 \ --attention-backend flashinfer \ --chunked-prefill-size 32768 \ --max-prefill-tokens 32768 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --mamba-full-memory-ratio 4.59 \ --host 0.0.0.0 \ --port 30000 \ --mamba-radix-cache-strategy extra_buffer \ --mamba-ssm-dtype float32 ``` Docker downloads the image if needed. SGLang downloads the model on first use and starts the server. Keep this terminal open and wait for `The server is fired up and ready to roll!` before sending a request. Run this command in the activated Python environment where you installed SGLang. It starts the same server without Docker: ```bash theme={null} sglang serve \ --trust-remote-code \ --model-path Qwen/Qwen3.8-27B-FP8 \ --kv-cache-dtype fp8_e4m3 \ --mem-fraction-static 0.85 \ --attention-backend flashinfer \ --chunked-prefill-size 32768 \ --max-prefill-tokens 32768 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --mamba-full-memory-ratio 4.59 \ --host 0.0.0.0 \ --port 30000 \ --mamba-radix-cache-strategy extra_buffer \ --mamba-ssm-dtype float32 ``` For Python environment setup, see [Install with uv](/docs/get-started/install#method-1-with-pip-or-uv). Verify the running server with the [Quickstart request](/docs/get-started/quickstart#send-a-request), using `Qwen/Qwen3.8-27B-FP8` as the model name. ## Connect to a model The following Python examples connect to the local server above. You can also use a remote SGLang server; only the server needs hardware that can run the model. Install the Python client in your activated environment: ```bash theme={null} uv pip install openai ``` Run this setup before each example in the same Python session: ```python theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) model = "Qwen/Qwen3-0.6B" ``` These settings match the local Qwen3-0.6B server above. For a remote server, replace `localhost` with its reachable address, use its model name, and supply an API key if authentication is enabled. ## Stream a reply Ask a question and print the answer as it arrives: ```python theme={null} stream = client.chat.completions.create( model=model, messages=[{"role": "user", "content": "Explain prefix caching in two sentences."}], stream=True, ) for chunk in stream: if chunk.choices: print(chunk.choices[0].delta.content or "", end="", flush=True) print() ``` The terminal displays the model's explanation incrementally. You can use the same stream to update a chat interface. See the [API guide](/docs/basic_usage/openai_api_completions) for more request options. ## Process multiple prompts Send several independent requests concurrently, for example to summarize customer feedback: ```python theme={null} from concurrent.futures import ThreadPoolExecutor feedback = [ "Setup was quick, but I could not find the export button.", "The new search is faster and finds the documents I need.", "Please add a way to share saved reports with my team.", ] def summarize(text): response = client.chat.completions.create( model=model, messages=[{"role": "user", "content": f"Summarize in one sentence: {text}"}], ) return response.choices[0].message.content with ThreadPoolExecutor(max_workers=3) as pool: for summary in pool.map(summarize, feedback): print(summary) ``` You get one summary per input, in the original order. For batch processing inside a Python process without an HTTP server, see the [offline engine API](/docs/basic_usage/offline_engine_api). ## Build a customer support agent Let an agent answer “Where is my order?” by looking up order data. The model chooses a tool, your application executes it, and the model uses the returned data to answer the customer. Use the [Qwen3.8-27B server above](#serve-qwen3-8-27b), which enables a tool parser, or configure another model with the [tool-calling guide](/docs/advanced_features/tool_parser). The small local example does not configure tool calling. This script uses the OpenAI client installed above; update the endpoint and model ID to match your server. Save this as `order_agent.py` and run `python order_agent.py`. The order data is a local fixture, so no external service or API key is needed for the lookup. ```python theme={null} import json from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") model = "Qwen/Qwen3.8-27B-FP8" orders = { "SGL-1042": {"status": "shipped", "estimated_delivery": "Friday"}, } def lookup_order(order_id): return orders.get(order_id, {"error": "Order not found"}) tools = [{ "type": "function", "function": { "name": "lookup_order", "description": "Look up an order's shipping status and estimated delivery.", "parameters": { "type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"], "additionalProperties": False, }, }, }] messages = [ {"role": "system", "content": "Use lookup_order for order status. Do not invent order details."}, {"role": "user", "content": "Where is order SGL-1042, and when should it arrive?"}, ] for _ in range(3): response = client.chat.completions.create( model=model, messages=messages, tools=tools, tool_choice="auto", ) message = response.choices[0].message messages.append(message.model_dump(exclude_none=True)) if not message.tool_calls: print(message.content) break for call in message.tool_calls: try: if call.function.name != "lookup_order": raise ValueError("Unknown tool") arguments = json.loads(call.function.arguments) result = lookup_order(**arguments) except (ValueError, TypeError) as error: result = {"error": str(error)} print(f"{call.function.name}: {result}") messages.append({ "role": "tool", "tool_call_id": call.id, "content": json.dumps(result), }) else: print("Stopped after three model turns without a final answer.") ``` You should see the lookup result followed by an answer that the order has shipped and is expected on Friday. The wording can vary. SGLang serves the model; the Python application runs the tools and manages the conversation. Replace `lookup_order` with an authorized order-system query to use real data. # Installation Source: https://docs.sglang.io/docs/get-started/install Install SGLang with pip/uv, source, Docker, Kubernetes, and cloud deployment options. Choose one installation method below. These instructions target Linux with NVIDIA GPUs; for other devices, see [Hardware platforms](/docs/hardware-platforms/overview). ## Install SGLang Docker includes Python, SGLang, and its dependencies; no separate host Python installation is required. The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker). Use Linux with a supported NVIDIA GPU, a CUDA 13-compatible driver, [Docker Engine](https://docs.docker.com/engine/install/), and [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). Complete the Toolkit guide's Docker runtime configuration before starting a GPU container. SGLang images ship a CUDA 13 environment. CUDA 12 images (`-cu12` / `-cu129`) are no longer published; `lmsysorg/sglang:v0.5.19-cu129` is the last CUDA 12 tag. **Pull the image** `latest` contains the newest stable release: ```bash theme={null} docker pull lmsysorg/sglang:latest ``` **Image options** The examples use `latest`. To use another image, replace the tag in the pull and run commands: * Nightly: Built daily from the latest `main`, including changes not yet in a stable release. Choose a tag from the [nightly image list](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly). * Runtime: A smaller image of the same stable release, with inference dependencies and fewer development utilities. Use the `latest-runtime` tag. For reproducible deployments, use a specific release or dated nightly tag from [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang/tags). **Start a container** Set `HF_CACHE_DIR` to an absolute directory path on the host for downloaded model weights, replacing the placeholder below. For gated or private models, export your [Hugging Face token](https://huggingface.co/docs/hub/en/security-tokens) as `HF_TOKEN` on the host and add `--env HF_TOKEN` before the image name in this command. ```bash theme={null} HF_CACHE_DIR="/path/to/huggingface-cache" docker run -it --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v "$HF_CACHE_DIR:/root/.cache/huggingface" \ --ipc=host \ lmsysorg/sglang:latest bash ``` The container shell is ready for `sglang serve`. Use Python 3.10 or higher and a compatible CUDA 13 environment. SGLang requires CUDA 13. CUDA 12 (`cu129`) wheels and images were retired with the upgrade to PyTorch 2.14, which publishes no CUDA 12.9 builds. SGLang 0.5.19 is the last release with a CUDA 12 lane (PyTorch 2.13). **Prepare the environment** Use uv for faster installation. Install it with pip, or follow the [uv installation instructions](https://docs.astral.sh/uv/getting-started/installation/): ```bash theme={null} pip install --upgrade pip pip install uv ``` Create and activate a virtual environment: ```bash Command theme={null} uv venv --python 3.12 source .venv/bin/activate ``` **Install SGLang** ```bash theme={null} uv pip install --prerelease=allow sglang ``` Some of SGLang's dependencies only publish pre-releases on PyPI, so without `--prerelease=allow` uv older than 0.12.0 silently installs SGLang 0.5.9. On [uv 0.12.0](https://github.com/astral-sh/uv/releases/tag/0.12.0) and newer the flag is a harmless no-op. To pick up the latest features and fixes before the next stable release, install a nightly build. Nightly wheels are built from the latest `main` and published to the SGLang wheel index. Add that index with `--extra-index-url`, and combine `--prerelease=allow` with `--index-strategy unsafe-best-match` so uv considers the nightly (pre-release) version alongside PyPI: ```bash Command theme={null} uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang ``` For `CUDA_HOME` errors, see [Troubleshooting](#cuda_home). Use Python 3.10 or higher in an activated Python environment, with CUDA dependencies appropriate for the version you build. **Get the source** Clone a release tag (the example below pins one); choose another from [GitHub releases](https://github.com/sgl-project/sglang/releases) as needed: ```bash theme={null} git clone -b v0.5.21 https://github.com/sgl-project/sglang.git cd sglang ``` **Install SGLang** Install the Python package in editable mode so local source changes are reflected in your environment: ```bash theme={null} pip install --upgrade pip pip install -e "python" ``` For contributing to SGLang, follow the [development environment guide](/docs/developer_guide/development_guide_using_docker#setup-docker-container), which covers a container with development tools and dependencies installed. For `CUDA_HOME` errors, see [Troubleshooting](#cuda_home). ## Start a model server Start a model server inside your Docker container or activated Python environment. Replace `MODEL_PATH` with a Hugging Face model ID or a local model directory: ```bash theme={null} sglang serve MODEL_PATH --host 0.0.0.0 --port 30000 ``` Additional launch arguments depend on the model and your hardware. * [Quickstart](/docs/get-started/quickstart): Launch a model and send a request to verify the server. * [Cookbook](/cookbook/intro): Find model-specific launch arguments for your hardware. ## Deployment options Choose how to run SGLang as a service. Expand an option for setup instructions. Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operator for enterprise-grade management and serving of large language models (LLMs). 1. Option 1: For single node serving (typically when the model size fits into GPUs on one node) Execute command `kubectl apply -f docker/k8s-sglang-service.yaml`, to create k8s deployment and service, with llama-31-8b as example. 2. Option 2: For multi-node serving (usually when a large model requires more than one GPU node, such as `DeepSeek-R1`) Modify the LLM model path and arguments as necessary, then execute command `kubectl apply -f docker/k8s-sglang-distributed-sts.yaml`, to create two nodes k8s statefulset and serving service. > This method is recommended if you plan to serve it as a service. > A better approach is to use the [k8s-sglang-service.yaml](https://github.com/sgl-project/sglang/blob/main/docker/k8s-sglang-service.yaml). 1. Copy the [compose.yml](https://github.com/sgl-project/sglang/blob/main/docker/compose.yaml) to your local machine 2. Execute the command `docker compose up -d` in your terminal. To deploy on Kubernetes or 12+ clouds, you can use [SkyPilot](https://github.com/skypilot-org/skypilot). 1. Install SkyPilot and set up Kubernetes cluster or cloud access: see [SkyPilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html). 2. Deploy on your own infra with a single command and get the HTTP API endpoint: ```yaml Config theme={null} # sglang.yaml envs: HF_TOKEN: null resources: image_id: docker:lmsysorg/sglang:latest accelerators: A100 ports: 30000 run: | conda deactivate python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --host 0.0.0.0 \ --port 30000 ``` ```bash Command theme={null} # Deploy on any cloud or Kubernetes cluster. Use --cloud to select a specific cloud provider. HF_TOKEN= sky launch -c sglang --env HF_TOKEN sglang.yaml # Get the HTTP API endpoint sky status --endpoint 30000 sglang ``` 3. To further scale up your deployment with autoscaling and failure recovery, check out the [SkyServe + SGLang guide](https://github.com/skypilot-org/skypilot/tree/master/llm/sglang#serving-llama-2-with-sglang-for-more-traffic-using-skyserve). To deploy on SGLang on AWS SageMaker, check out [AWS SageMaker Inference](https://aws.amazon.com/sagemaker/ai/deploy) Amazon Web Services provide supports for SGLang containers along with routine security patching. For available SGLang containers, check out [AWS SGLang DLCs](https://aws.github.io/deep-learning-containers/reference/available_images/#sglang). To deploy a pre-built SGLang Deep Learning Container without building your own image, see [Amazon SageMaker AI](/docs/basic_usage/aws_sagemaker). To host a model with your own container, follow the following steps: 1. Build a docker container with [sagemaker.Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/sagemaker.Dockerfile) alongside the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script. 2. Push your container onto AWS ECR. ```bash Command theme={null} #!/bin/bash AWS_ACCOUNT="" AWS_REGION="" REPOSITORY_NAME="" IMAGE_TAG="" ECR_REGISTRY="${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com" IMAGE_URI="${ECR_REGISTRY}/${REPOSITORY_NAME}:${IMAGE_TAG}" echo "Starting build and push process..." # Login to ECR echo "Logging into ECR..." aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_REGISTRY} # Build the image echo "Building Docker image..." docker build -t ${IMAGE_URI} -f sagemaker.Dockerfile . echo "Pushing ${IMAGE_URI}" docker push ${IMAGE_URI} echo "Build and push completed successfully!" ``` 3. Deploy a model for serving on AWS Sagemaker, refer to [deploy\_and\_serve\_endpoint.py](https://github.com/sgl-project/sglang/blob/main/examples/sagemaker/deploy_and_serve_endpoint.py). For more information, check out [sagemaker-python-sdk](https://github.com/aws/sagemaker-python-sdk). 1. By default, the model server on SageMaker will run with the following command: `python3 -m sglang.launch_server --model-path opt/ml/model --host 0.0.0.0 --port 8080`. This is optimal for hosting your own model with SageMaker. 2. To modify your model serving parameters, the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script allows for all available options within `python3 -m sglang.launch_server --help` cli by specifying environment variables with prefix `SM_SGLANG_`. 3. The serve script will automatically convert all environment variables with prefix `SM_SGLANG_` from `SM_SGLANG_INPUT_ARGUMENT` into `--input-argument` to be parsed into `python3 -m sglang.launch_server` cli. 4. For example, to run [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) with reasoning parser, simply add additional environment variables `SM_SGLANG_MODEL_PATH=Qwen/Qwen3-0.6B` and `SM_SGLANG_REASONING_PARSER=qwen3`. ## Troubleshooting ### CUDA\_HOME If installation fails with `OSError: CUDA_HOME environment variable is not set`, set `CUDA_HOME` to your CUDA Toolkit installation directory. Replace the path below with the actual path on your machine: ```bash theme={null} export CUDA_HOME="/path/to/cuda" ``` For a versioned installation, this is typically `/usr/local/cuda-VERSION`. Alternatively, install FlashInfer first using the [FlashInfer installation instructions](https://docs.flashinfer.ai/installation.html), then retry the SGLang installation. ### FlashInfer * [FlashInfer](https://github.com/flashinfer-ai/flashinfer) is the default attention kernel backend. It only supports sm75 and above. If you encounter any FlashInfer-related issues on sm75+ devices (e.g., T4, A10, A100, L4, L40S, H100), please switch to other kernels by adding `--attention-backend triton --sampling-backend pytorch` and open an issue on GitHub. * To reinstall flashinfer locally, use the following command: `pip3 install --upgrade flashinfer-python --force-reinstall --no-deps` and then delete the cache with `rm -rf ~/.cache/flashinfer`. # Quickstart Source: https://docs.sglang.io/docs/get-started/quickstart Start a model server and verify it with one request. ## Install and start These commands target Linux with an NVIDIA GPU and a CUDA 13-compatible driver. See [Installation](/docs/get-started/install) for environment requirements and other setup options. With Docker and NVIDIA Container Toolkit configured, pull the image. Skip this command if you have already pulled it: ```bash theme={null} docker pull lmsysorg/sglang:latest ``` Start the server on the GPU host. Replace `MODEL_PATH` with your Hugging Face model ID (for example, `Qwen/Qwen3-0.6B`): ```bash theme={null} docker run --rm --gpus all --ipc=host \ -p 30000:30000 \ lmsysorg/sglang:latest \ sglang serve MODEL_PATH --host 0.0.0.0 --port 30000 ``` With uv installed, create a Python environment and install SGLang. Skip this block if you already have an environment with SGLang installed: ```bash theme={null} uv venv --python 3.12 source .venv/bin/activate uv pip install --prerelease=allow sglang ``` Start the server in that environment. Replace `MODEL_PATH` with your Hugging Face model ID (for example, `Qwen/Qwen3-0.6B`): ```bash theme={null} sglang serve MODEL_PATH --host 0.0.0.0 --port 30000 ``` For additional model-specific launch arguments, see the [Cookbook](/cookbook/intro). Wait for `The server is fired up and ready to roll!`. ## Send a request In another terminal on the same host, send a request to verify the server. Use the same model ID for `MODEL_PATH`: ```bash theme={null} curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "MODEL_PATH", "messages": [ {"role": "user", "content": "What is the capital of France?"} ] }' ``` Check `choices[0].message.content` for the answer. Press `Ctrl+C` in the server terminal to stop. See [Examples](/docs/get-started/examples) for a complete 27B deployment, streaming, and tool calls. # AMD GPUs Source: https://docs.sglang.io/docs/hardware-platforms/amd_gpu This document describes how to run SGLang on AMD GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues). ## System Configuration When using AMD GPUs (such as MI300X), certain system-level optimizations help ensure stable performance. Here we take MI300X as an example. AMD provides official documentation for MI300X optimization and system tuning: * [AMD MI300X Tuning Guides](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html) * [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/vllm-benchmark.html) * [AMD Instinct MI300X System Optimization](https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html) * [AMD Instinct MI300X Workload Optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html) * [Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html) **NOTE:** We strongly recommend reading these docs and guides entirely to fully utilize your system. Below are a few key settings to confirm or enable for SGLang: ### Update GRUB Settings In `/etc/default/grub`, append the following to `GRUB_CMDLINE_LINUX`: ```text GRUB Configuration theme={null} pci=realloc=off iommu=pt ``` Afterward, run `sudo update-grub` (or your distro’s equivalent) and reboot. ### Disable NUMA Auto-Balancing ```bash Disable NUMA theme={null} sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing' ``` You can automate or verify this change using [this helpful script](https://github.com/ROCm/triton/blob/rocm_env/scripts/amd/env_check.sh). Again, please go through the entire documentation to confirm your system is using the recommended configuration. ## Install SGLang You can install SGLang using one of the methods below. ### Install from Source ```bash Command theme={null} # Use the last release branch git clone -b v0.5.21 https://github.com/sgl-project/sglang.git cd sglang # Compile sgl-kernel pip install --upgrade pip cd python/sglang/kernels/aot python setup_rocm.py install # Install sglang python package along with diffusion support cd ../../../.. rm -rf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml pip install -e "python[all_hip]" ``` ### Install Using Docker (Recommended) The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [rocm.Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker). The steps below show how to build and use an image. 1. Build the docker image. If you use pre-built images, you can skip this step and replace `sglang_image` with the pre-built image names in the steps below. ```bash Command theme={null} docker build -t sglang_image -f rocm.Dockerfile . ``` 2. Create a convenient alias. ```bash Command theme={null} alias drun='docker run -it --rm --network=host --privileged --device=/dev/kfd --device=/dev/dri \ --ipc=host --shm-size 16G --group-add video --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v $HOME/dockerx:/dockerx \ -v /data:/data' ``` If you are using RDMA, please note that: * `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them. * You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`. 3. Launch the server. **NOTE:** Replace `` below with your [huggingface hub token](https://huggingface.co/docs/hub/en/security-tokens). ```bash Command theme={null} drun -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ sglang_image \ python3 -m sglang.launch_server \ --model-path NousResearch/Meta-Llama-3.1-8B \ --host 0.0.0.0 \ --port 30000 ``` 4. To verify the utility, you can run a benchmark in another terminal or refer to [other docs](../basic_usage/openai_api_completions) to send requests to the engine. ```bash Command theme={null} drun sglang_image \ python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 4000 \ --random-input 128 \ --random-output 128 ``` With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang’s machine learning capabilities. ## Quantization on AMD GPUs The [Quantization documentation](../advanced_features/quantization#platform-compatibility) has a full compatibility matrix. The short version: FP8, AWQ, MXFP4, W8A8, GPTQ, compressed-tensors, Quark, and **petit\_nvfp4** (NVFP4 on ROCm via [Petit](https://github.com/causalflow-ai/petit-kernel)) all work on AMD. Methods that depend on Marlin or NVIDIA-specific kernels (`awq_marlin`, `gptq_marlin`, `gguf`, `modelopt_fp8`, `modelopt_fp4`) do not. A few things to keep in mind: * FP8 works via Aiter or Triton. Pre-quantized FP8 models like DeepSeek-V3/R1 work out of the box. * AWQ uses Triton dequantization kernels on AMD. The faster Marlin path is not available. * MXFP4 requires CDNA3/CDNA4 and `SGLANG_USE_AITER=1`. * `petit_nvfp4` enables NVFP4 models (e.g., [Llama 3.3 70B FP4](https://huggingface.co/nvidia/Llama-3.3-70B-Instruct-FP4)) on MI250/MI300X via [Petit](https://github.com/causalflow-ai/petit-kernel). Install with `pip install petit-kernel`; no `--quantization` flag needed when loading pre-quantized NVFP4 models. * `quark_int4fp8_moe` is an AMD-only online quantization method for MoE models on CDNA3/CDNA4. Several of these backends are accelerated by [Aiter](https://github.com/ROCm/aiter). Enable it with: ```bash Command theme={null} export SGLANG_USE_AITER=1 ``` Example -- serving an AWQ model: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path hugging-quants/Mixtral-8x7B-Instruct-v0.1-AWQ-INT4 \ --trust-remote-code \ --port 30000 --host 0.0.0.0 ``` Example -- FP8 online quantization: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --quantization fp8 \ --port 30000 --host 0.0.0.0 ``` ## Examples ### Running DeepSeek-V3 The only difference when running DeepSeek-V3 is in how you start the server. Here's an example command: ```bash Command theme={null} drun -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --ipc=host \ --env "HF_TOKEN=" \ sglang_image \ python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3 \ # <- here --tp 8 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` [Running DeepSeek-R1 on a single NDv5 MI300X VM](https://techcommunity.microsoft.com/blog/azurehighperformancecomputingblog/running-deepseek-r1-on-a-single-ndv5-mi300x-vm/4372726) could also be a good reference. ### Running Llama3.1 Running Llama3.1 is nearly identical to running DeepSeek-V3. The only difference is in the model specified when starting the server, shown by the following example command: ```bash Command theme={null} drun -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --ipc=host \ --env "HF_TOKEN=" \ sglang_image \ python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ # <- here --tp 8 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` ### Warmup Step When the server displays `The server is fired up and ready to roll!`, it means the startup is successful. ## Strix Halo / gfx1151 (RDNA3.5 iGPU) AMD Ryzen AI MAX+ / Strix Halo (`gfx1151`) and Strix Point (`gfx1150`) are wave32 RDNA3.5 iGPUs. They are **not** covered by the Instinct/CDNA `rocm.Dockerfile` image. Use the dedicated gfx1151 image instead. **Official image tag pattern** (nightly on Docker Hub [`rocm/sgl-dev`](https://hub.docker.com/r/rocm/sgl-dev/tags)): ```bash theme={null} rocm/sgl-dev:*-gfx1151* # example: rocm/sgl-dev:v0.5.20-rocm724-gfx1151-20260924 ``` The image sets `SGLANG_USE_AITER=0` (see `docker/rocm-gfx1151.Dockerfile`). AITER kernels are CDNA/wave64-oriented and are unsafe as defaults on gfx115x. **Host requirements**: expose `/dev/kfd` and `/dev/dri`, and add the container to the `video` and `render` groups. Strix Halo has no discrete VRAM — the GPU carves memory from system RAM (configure GTT / BIOS aperture as needed on the host). ```bash theme={null} docker run -it --rm \ --device=/dev/kfd --device=/dev/dri \ --group-add video --group-add render \ --security-opt seccomp=unconfined \ --ipc=host --shm-size 16g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ rocm/sgl-dev:v0.5.20-rocm724-gfx1151-20260924 \ bash ``` **Serving defaults on gfx1151**: * LLM / SRT: `--attention-backend triton` (or rely on the gfx115 auto-default), `--num-gpus 1` * Diffusion / DiT (e.g. Wan2.2): `--attention-backend torch_sdpa`, `--num-gpus 1` * Keep `SGLANG_USE_AITER=0` (already set in the official image) # Apple Silicon with Metal Source: https://docs.sglang.io/docs/hardware-platforms/apple_metal This document describes how to run the SGLang serving runtime on Apple Silicon using [MLX](https://opensource.apple.com/projects/mlx/). SGLang Diffusion uses PyTorch MPS instead; see its [installation guide](/docs/sglang-diffusion/installation#platform-specific-apple-mps). If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues). ## Prerequisites The MLX runtime requires Apple Silicon with macOS 14 or newer, stable PyTorch 2.13.x, and stable MLX 0.32.0 or newer. The `srt_mps` extra installs PyTorch 2.13.0 and MLX 0.32.0 or newer; startup accepts stable PyTorch 2.13 patch releases and newer stable MLX releases. With `SGLANG_USE_MLX=1`, SGLang validates both framework versions and Metal availability during argument initialization and stops before resolving or downloading a model when the runtime is incompatible. Building the optional native Metal kernels in `sgl-kernel` requires the Metal shader compiler from the full Xcode application. The standalone Xcode Command Line Tools are not sufficient. After installing Xcode, select it with: ```bash theme={null} sudo xcode-select -s /Applications/Xcode.app/Contents/Developer ``` Verify the compiler with `xcrun -sdk macosx metal --version`. ## Install SGLang You can install SGLang using one of the methods below. ### Install from Source ```bash theme={null} # Use the default branch git clone https://github.com/sgl-project/sglang.git cd sglang # Create and activate a virtual environment uv venv -p 3.12 sglang-metal source sglang-metal/bin/activate # (Optional) Compile sgl-kernel uv pip install --upgrade pip uv run python/sglang/kernels/aot/setup_metal.py install # Install sglang python package along with diffusion support rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml uv pip install -e "python[all_mps]" ``` ## Launch of the Serving Engine Launch the server with: ```bash theme={null} SGLANG_USE_MLX=1 python -m sglang.launch_server \ --model \ --disable-cuda-graph \ --host 0.0.0.0 ``` **Key Parameters Explained:** 1. `SGLANG_USE_MLX=1` - Enables the use of MLX as the SGLang runtime backend (if disabled, SGLang will fall back to `torch.mps`, which has less support) 2. `--disable-cuda-graph` - Disables usage of CUDA graph, which is not relevant for Apple Metal. 3. `--disable-overlap-schedule` - Disables overlap scheduling (enabled/not present by default) achieved using MLX's `async_eval()` 4. `SGLANG_MLX_USE_CUSTOM_ROPE=1` - Enables the optional custom Metal RoPE kernel. It is disabled by default, so the MLX backend uses the standard RoPE path unless you opt in for A/B testing. 5. `SGLANG_MLX_FUSE_SWIGLU=1` - Enables the use of fused Swish-Gated Linear Unit kernel (disabled by default) 6. `SGLANG_MLX_CLEAR_CACHE_STEPS=256` - Sets the number of decode steps before clearing the MLX cache (256 by default) ## Quantization The MLX backend supports two quantization paths on Apple Silicon: 1. **Pre-quantized HF repos.** Any `mlx-community/-4bit` (or `-8bit`) repo loads directly through `mlx_lm.load(...)` — no extra flag needed. ```bash theme={null} SGLANG_USE_MLX=1 python -m sglang.launch_server \ --model-path mlx-community/Qwen3-0.6B-4bit \ --disable-cuda-graph ``` 2. **On-the-fly quantization.** For any fp16 model, pass `--quantization mlx_q4` or `--quantization mlx_q8` to have sglang quantize the weights at load time via `mlx_lm.utils.quantize_model` (group size 64, the mlx-community default). The quantized weights stay in process memory; the on-disk model is untouched. ```bash theme={null} SGLANG_USE_MLX=1 python -m sglang.launch_server \ --model-path Qwen/Qwen3-0.6B \ --quantization mlx_q4 \ --disable-cuda-graph ``` Expected log line: ``` Quantizing MLX model on-the-fly: bits=4 group_size=64 (preset=mlx_q4) Quantization complete in 0.13s — active mem: 1.11 GB -> 0.31 GB (71.9% reduction) ``` The MLX backend silently ignores `--quantization mlx_q4` when the model is already quantized in its HF config (path 1), so the same flag is safe to pass either way. ## Benchmarking with Requests `sglang.benchmark.one_batch` calls the synchronous prefill/decode methods directly without going through the scheduler and the overlap code path. `sglang.benchmark.offline_throughput` can toggle overlap scheduling as it uses the scheduler and the overlap code path by using the flag `--disable-overlap-schedule`. ### Throughput Testing Basic synchronous one batch throughput: ```bash theme={null} SGLANG_USE_MLX=1 python -m sglang.benchmark.one_batch \ --model-path \ --disable-cuda-graph \ --tp-size 1 \ --batch-size 1 \ --input-len 60 \ --output-len 10 ``` Synchronous offline throughput: ```bash theme={null} SGLANG_USE_MLX=1 python -m sglang.benchmark.offline_throughput \ --model-path \ --disable-cuda-graph \ --num-prompts 1 \ --disable-overlap-schedule ``` Asynchronous offline throughput: ```bash theme={null} SGLANG_USE_MLX=1 python -m sglang.benchmark.offline_throughput \ --model-path \ --disable-cuda-graph \ --num-prompts 1 ``` # Contribution Guide Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/development/contribution_guide Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you’re fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process. ## Install SGLang from Source ### Prepare Environment Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](../getting-started/installation) to install the necessary dependencies. We recommend [using docker](../getting-started/installation#method-2-using-docker-image) to build the environment. ### Fork and clone the repository **Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally. ```bash theme={null} git clone https://github.com//sglang.git # if you are using docker, the environment is already set up. cd sglang export PYTHONPATH=$PWD/python:$PYTHONPATH ``` ## Format code with pre-commit We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run: ```bash theme={null} pip3 install pre-commit pre-commit install pre-commit run --all-files ``` * **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request. * **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch. * Link checking with lychee is **enforced in CI**. By default, it is not blocking local commits. * To run local link checks manually, use: `pre-commit run --hook-stage manual lychee --all-files`. ## Run and add tests All NPU tests are end-to-end (E2E) and require launching a server with real model weights. Tests live under [`test/registered/npu/`](https://github.com/sgl-project/sglang/tree/main/test/registered/npu), organized by model type and functionality: ``` ascend/ ├── llm_models/ # Per-model inference accuracy ├── vlm_models/ # Vision-language models ├── embedding_models/ # Embedding model tests ├── rerank_models/ # Reranker model tests ├── reward_models/ # Reward model tests ├── interface/ # API correctness, function calling ├── basic_function/ # Cache, sampling, quantization, etc. └── test_npu_memory_consumption.py ``` ### Adding a test See [`test_npu_sampling_backend.py`](https://github.com/sgl-project/sglang/tree/main/test/registered/npu/basic_function/backends/test_npu_sampling_backend.py) for a complete example. Key steps: 1. Place your test file in the appropriate directory under `test/registered/npu/`. 2. Extend `CustomTestCase` (from `sglang.test.test_utils`) for CI retry support. 3. Launch server with `popen_launch_server()` in `setUpClass` and clean up with `kill_process_tree()` in `tearDownClass`. 4. Register your test with `register_npu_ci()`: ```python theme={null} from sglang.test.ci.ci_register import register_npu_ci register_npu_ci(est_time=400, suite="stage-b-test-1-npu-a3", nightly=False) register_npu_ci(est_time=400, suite="nightly-1-npu-a3", nightly=True) ``` ### Running tests locally ```bash theme={null} pytest test/registered/npu/llm_models/test_npu_qwen3_0_6b.py -v ``` For detailed instructions, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md). ### Register models for CI If you need to use model which is not in `python/sglang/test/ascend/test_ascend_utils.py` list. Follow these steps: 1. Register account and upload your model to [modelscope](https://modelscope.cn/models). 2. Make sure your model is pre-cached on the CI server and is at the path "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models//". If this is not the case, use following command on CI server: ```bash theme={null} modelscope download \ --model {your_model_repo}/{your_model} \ --local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model} ``` > Note: If you don’t have access to CI server, please ask maintainers ([zl19940307@163.com](mailto:zl19940307@163.com)) to download your model. 3. Add model to `python/sglang/test/ascend/test_ascend_utils.py` (use docker `"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"` path). ## Write documentation We recommend new contributors start by writing documentation, which helps you quickly understand SGLang codebase. For more details, please refer to [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md). ## Test the accuracy If your code changes the model output, run the accuracy tests. A quick sanity check is GSM8K through sgl-eval. ```bash theme={null} # Launch a server python3 -m sglang.launch_server --model Qwen/Qwen2-7B-Instruct # Evaluate. --base-url must point at the server launched above. # The default SGLang server port is 30000; change it if you launched # the server with a different --port. sgl-eval run gsm8k \ --base-url http://localhost:30000/v1 \ --num-examples 200 ``` Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test. This test can have significant variance (1%–5%) in accuracy due to batching and the non-deterministic nature of the inference engine. Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test. GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests. You can find additional accuracy eval examples in: * [test\_eval\_accuracy\_large.py](https://github.com/sgl-project/sglang/blob/main/test/manual/eval/test_eval_accuracy_large.py) ## Benchmark the speed Refer to [Benchmark and Profiling](../../../developer_guide/benchmark_and_profiling). ## Requesting a review for merge You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md). You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals. Then your PR can be merged. ## How to Trigger CI Tests We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests. Users with permission are listed in the [CI\_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) **PR authors** can always use `/rerun-failed-ci` on their own PRs, even if they are not listed in `CI_PERMISSIONS.json`. For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands: * `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`). * `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, flaky, skipped, cancelled, or timed out**. * `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`). * `/rerun-test [ ...]`: Reruns one or more specific tests directly, bypassing stage boundaries. Each `` is pytest-style `::[.]` (the `::TestClass` and `.` parts are optional). The handler resolves each spec, groups specs by their registered runner-label, and dispatches one [Rerun Test workflow](https://github.com/sgl-project/sglang/actions/workflows/rerun-test.yml) per group. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py` (multiple at once). If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302). To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`). If you don’t have permission, please ask maintainers to trigger CI for you. ### CI rate limits Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests. We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources. Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter: ```yaml theme={null} cool-down-minutes: description: "Cooldown period in minutes for low-permission users; 0 disables rate limiting" type: number default: 120 ``` Users listed in [CI\_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval. ## Code style guidance * Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function. * Minimize device synchronization. Reduce expensive CPU-NPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code. * Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code. * A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value in `__init__` whenever possible. * Make functions as pure as possible. Avoid in-place modification of arguments. * Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_pp_mixin.py`) * In a file, put core data structures at the top of the file. Put utility functions at the bottom of the file. * Keep tests run fast. * If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`). * If a single job in a github workflow runs longer than 30 min, split it into smaller jobs/steps. * Reuse server launches in your unit tests to make tests run faster. * Never use `pickle.loads()`, `pickle.load()`, or `recv_pyobj()` to deserialize untrusted or network-received data. Python’s [pickle module is not secure](https://docs.python.org/3/library/pickle.html) — it can execute arbitrary code during deserialization. Use safe serialization formats such as [msgpack](https://github.com/jcrist/msgspec) or JSON instead. * When supporting new hardware or features, follow these guidelines: * Do not drastically change existing code. * Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_npu.py`). * If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch. ## How to update sgl-kernel-npu Sgl-kernel-npu is the separate kernel package for Ascend NPU, containing both Ascend C and Triton operators. It is maintained in the [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository. For detailed guidance on developing and integrating operators (Ascend C directory structure, PyTorch op registration, build, test, and code style), see the [Ascend NPU Operator Development Guide](./operator_development). ### Multi-PR workflow Since SGLang and sgl-kernel-npu are separate Python packages, dependency updates require a multi-PR workflow: 1. **Submit sgl-kernel-npu PR**: Add or modify operators in the sgl-kernel-npu repository following the operator development guide. Ensure all tests pass. 2. **Bump sgl-kernel-npu version**: Update the version number. Merging triggers an automatic PyPI release. If not urgent, wait for a regular release (typically within one week). 3. **Reference the new version in SGLang**: * Update the `SGLANG_KERNEL_NPU_TAG` argument in [`docker/npu.Dockerfile`](https://github.com/sgl-project/sglang/blob/main/docker/npu.Dockerfile) to the new sgl-kernel-npu release tag. * Use the new operator in SGLang code. ## Tips for newcomers If you want to contribute but don’t have a specific idea in mind, pick issues labeled ["good first issue" or "help wanted"](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase. Also check out the following materials as startup guide: * [Mini-SGLang](https://github.com/sgl-project/mini-sglang) for a quick overview on the structure of sglang. * [Code Walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow. * [GTC-2026 Training Lab](https://drive.google.com/file/d/1mwOZEtipNLJzrflCTodj34KhuOZEoEw5/view?usp=drive_link) for hands-on practices of how to do optimization, benchmarking, or profiling on a launched SGLang instance. If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io). Thank you for your interest in SGLang. Happy coding! # Operator Development Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/development/operator_development How to develop custom operators (Ascend C / Triton) for Ascend NPU and integrate them into the SGLang inference engine. ## Overview [SGL-Kernel-NPU](https://github.com/sgl-project/sgl-kernel-npu) is the official operator library provided by the SGLang framework for Ascend NPU. It includes two types of operator implementations: 1. **Ascend C operators**: High-performance C++ kernels written in Ascend C, compiled into `libsgl_kernel_npu.so`, and registered through PyTorch's custom operator mechanism (`TORCH_LIBRARY_FRAGMENT`). Called in SGLang via `torch.ops.npu.()`. 2. **Triton operators**: Python kernels written in Triton, adapted for Ascend NPU. Called directly via `from sgl_kernel_npu.xxx import ...`. When SGLang detects an NPU device, it automatically loads `sgl_kernel_npu` and uses its operators in place of GPU counterparts, providing optimized inference on Ascend hardware. ## Directory Structure The identifiers `sgl_kenel_npu_ops.h`, `KernalHelloworld`, and `retrive_*` (e.g., `retrive_index`, `retrive_next_token`, `retrive_next_sibling`) in this guide match the spelling used in the upstream [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository and are kept verbatim for consistency. ```text theme={null} sgl-kernel-npu/ ├── csrc/ # Ascend C operator C++ sources │ ├── CMakeLists.txt # Build configuration │ ├── pytorch_extensions.cpp # PyTorch op registration (core integration file) │ └── / # One directory per operator │ ├── op_host/ # Host-side code (validation, tiling, launch) │ │ ├── .cpp │ │ └── tiling/ # Optional: tiling data │ └── op_kernel/ # Device-side code (Ascend C kernel on AI Core) │ └── _kernel.cpp ├── include/ │ └── sgl_kenel_npu_ops.h # C++ interface declarations ├── python/ │ └── sgl_kernel_npu/ │ └── sgl_kernel_npu/ │ ├── __init__.py # Loads libsgl_kernel_npu.so │ ├── attention/ # Triton attention kernels │ ├── norm/ # Triton normalization kernels │ ├── activation/ # Triton activation kernels │ ├── fla/ # Triton linear attention kernels │ ├── mamba/ # Triton Mamba kernels │ ├── moe/ # Triton MoE kernels │ └── sample/ # Triton speculative decoding kernels ├── tests/ │ └── python/sgl_kernel_npu/ # One test file per operator ├── build.sh # Build script └── CMakeLists.txt # Root CMake configuration ``` ## Developing Ascend C Operators A complete Ascend C operator consists of two parts: * **Device part**: Kernel code running on the NPU AI Core, responsible for actual computation. Written using the Ascend C API. * **Host part**: Code running on the CPU, responsible for parameter validation, data pre-processing, tiling, and kernel launch. We recommend starting with the [helloworld](https://github.com/sgl-project/sgl-kernel-npu/tree/main/csrc/helloworld) example, a simple operator that performs element-wise addition on two tensors. ### Step 1: Create the operator directory and files Create a new operator directory under `csrc/`, following the `op_host/` + `op_kernel/` structure: ```text theme={null} csrc// ├── op_host/ │ └── .cpp └── op_kernel/ └── _kernel.cpp ``` ### Step 2: Write the Device-side Kernel (op\_kernel) Device-side code runs on AI Core and follows the Ascend C programming model. The core structure is a class with `Init()` and `Process()` methods, plus an `extern "C"` entry function. Using helloworld as an example: ```cpp theme={null} // csrc/helloworld/op_kernel/kernel_helloworld.cpp #include "kernel_operator.h" constexpr int32_t BUFFER_NUM = 2; class KernalHelloworld { public: __aicore__ inline KernalHelloworld() {} __aicore__ inline void Init(GM_ADDR x, GM_ADDR y, GM_ADDR z, uint32_t totalLength) { // Compute workload for current block this->blockLength = totalLength / AscendC::GetBlockNum(); this->tileNum = 8; this->tileLength = this->blockLength / this->tileNum / BUFFER_NUM; // Set global memory buffers xGm.SetGlobalBuffer((__gm__ half *)x + this->blockLength * AscendC::GetBlockIdx(), this->blockLength); yGm.SetGlobalBuffer((__gm__ half *)y + this->blockLength * AscendC::GetBlockIdx(), this->blockLength); zGm.SetGlobalBuffer((__gm__ half *)z + this->blockLength * AscendC::GetBlockIdx(), this->blockLength); // Initialize pipeline queues pipe.InitBuffer(inQueueX, BUFFER_NUM, this->tileLength * sizeof(half)); pipe.InitBuffer(inQueueY, BUFFER_NUM, this->tileLength * sizeof(half)); pipe.InitBuffer(outQueueZ, BUFFER_NUM, this->tileLength * sizeof(half)); } __aicore__ inline void Process() { int32_t loopCount = this->tileNum * BUFFER_NUM; for (int32_t i = 0; i < loopCount; i++) { CopyIn(i); // Move data from Global Memory to Local Memory Compute(i); // Compute on Local Memory CopyOut(i); // Move results back to Global Memory } } private: __aicore__ inline void CopyIn(int32_t progress) { /* data copy-in... */ } __aicore__ inline void Compute(int32_t progress) { /* core computation... */ } __aicore__ inline void CopyOut(int32_t progress) { /* data copy-out... */ } private: AscendC::TPipe pipe; AscendC::TQue inQueueX, inQueueY; AscendC::TQue outQueueZ; AscendC::GlobalTensor xGm, yGm, zGm; uint32_t blockLength, tileNum, tileLength; }; // Entry function: the compile tool auto-generates aclrtlaunch_.h from this name extern "C" __global__ __aicore__ void helloworld( GM_ADDR x, GM_ADDR y, GM_ADDR z, uint32_t totalLength) { KernalHelloworld op; op.Init(x, y, z, totalLength); op.Process(); } ``` **Key points:** * Class methods must be marked with `__aicore__`, indicating they run on AI Core. * Use `AscendC::TPipe` + `AscendC::TQue` to build a pipeline that overlaps data movement and computation. * The entry function must be declared `extern "C" __global__ __aicore__`. The compile tool generates a host-callable launch header `aclrtlaunch_.h` from the function name. * Simple operators (e.g., helloworld, cache\_assign, lora) do not need extra workspace memory. Complex operators (e.g., mla\_preprocess, alloc\_extend, build\_tree) require temporary workspace memory and are compiled separately in `CMakeLists.txt`. For more in-depth Ascend C programming knowledge, refer to the [Ascend C Kernel Development Guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/opdevg/Ascendcopdevg/atlas_ascendc_10_0001.html). ### Step 3: Write the Host-side Code (op\_host) Host-side code is responsible for passing PyTorch Tensors to the kernel and launching it. The key macro is `EXEC_KERNEL_CMD` (located in `csrc/utils/torch_helper.h`). ```cpp theme={null} // csrc/helloworld/op_host/helloworld.cpp #include "defines.h" // Provides HOST_API macro #include "torch_helper.h" // Provides EXEC_KERNEL_CMD macro #include "aclrtlaunch_helloworld.h" // Auto-generated by compile tool namespace sglang { namespace npu_kernel { HOST_API at::Tensor helloworld(const at::Tensor &x, const at::Tensor &y) { // Create output tensor at::Tensor z = at::empty_like(x); // Define block count uint32_t blockDim = 8; // Compute total element count uint32_t totalLength = 1; for (uint32_t size : x.sizes()) { totalLength *= size; } // Launch kernel via EXEC_KERNEL_CMD macro EXEC_KERNEL_CMD(helloworld, blockDim, x, y, z, totalLength); return z; } } // namespace npu_kernel } // namespace sglang ``` **Key points:** * The namespace must be `sglang::npu_kernel`. * Function signatures follow the pattern `at::Tensor (const at::Tensor &input, ...)`. * For operators with multiple outputs, use `std::tuple` or non-const reference parameters. ### Step 4: Declare the C++ Interface (include/sgl\_kenel\_npu\_ops.h) Add the operator function declaration in `include/sgl_kenel_npu_ops.h`: ```cpp theme={null} // include/sgl_kenel_npu_ops.h namespace sglang { namespace npu_kernel { at::Tensor helloworld(const at::Tensor &x, const at::Tensor &y); } // namespace npu_kernel } // namespace sglang ``` ### Step 5: Register the PyTorch Custom Operator (pytorch\_extensions.cpp) Register the operator in `csrc/pytorch_extensions.cpp` in two steps: define the schema and bind the implementation. ```cpp theme={null} // csrc/pytorch_extensions.cpp namespace { // 1. Define operator schema (used by torch.compile, etc.) TORCH_LIBRARY_FRAGMENT(npu, m) { m.def("helloworld(Tensor x, Tensor y) -> Tensor"); // ... other operator schemas ... } // 2. Bind implementation for the PrivateUse1 device (i.e., NPU) TORCH_LIBRARY_IMPL(npu, PrivateUse1, m) { m.impl("helloworld", TORCH_FN(sglang::npu_kernel::helloworld)); // ... other operator implementations ... } } // namespace ``` **Schema conventions:** * The namespace is fixed to `npu`. In SGLang, operators are called via `torch.ops.npu.()`. * Output tensor parameters use the `Tensor(a!)` mutating annotation. * Optional parameters use the `Tensor?` annotation, with `c10::optional` handling in the impl. * For detailed schema syntax, see the [PyTorch Schema Reference](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/README.md#func). **Implementation binding rules:** * The device name is fixed to `PrivateUse1` (PyTorch NPU backend identifier). * Use the `TORCH_FN` macro to bind to the implementation function. * For complex operators with optional parameters, use lambda expressions to unpack the arguments. ### Step 6: Update the Build Configuration (csrc/CMakeLists.txt) Add the new operator's source files to `csrc/CMakeLists.txt`: **For operators not requiring workspace** (simple operators), add kernel source to `no_workspace_kernel`: ```cmake theme={null} ascendc_library(no_workspace_kernel STATIC # ... existing kernel files ... ${PROJECT_OP_SRC_BASE}//op_kernel/_kernel.cpp ) ``` **For operators requiring workspace** (complex operators), add kernel source to `workspace_kernel` with the `-DHAVE_WORKSPACE -DHAVE_TILING` compile flags: ```cmake theme={null} ascendc_library(workspace_kernel STATIC # ... existing kernel files ... ${PROJECT_OP_SRC_BASE}//op_kernel/_kernel.cpp ) ``` **Add host source files to `OP_SRCS`:** ```cmake theme={null} FILE(GLOB OP_SRCS # ... existing host files ... ${PROJECT_OP_SRC_BASE}//op_host/.cpp ) ``` ### Step 7: Build Build following the steps in the [python/sgl\_kernel\_npu/README.md](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md): ```bash theme={null} cd sgl-kernel-npu # Build all modules bash build.sh # Install the sgl_kernel_npu wheel pip install output/sgl_kernel_npu*.whl ``` The compiled `libsgl_kernel_npu.so` is copied into `python/sgl_kernel_npu/sgl_kernel_npu/lib/` and loaded by the Python package. ## Developing Triton Operators Triton operators are located under `python/sgl_kernel_npu/sgl_kernel_npu/`, organized by function category: ```text theme={null} python/sgl_kernel_npu/sgl_kernel_npu/ ├── attention/ # Attention (decode_attention, sinks_attention) ├── norm/ # Normalization (rmsnorm, fused_qk_norm, l1_norm) ├── activation/ # Activation (swiglu_oai, swiglu_quant) ├── fla/ # Linear attention (chunk, cumsum, wy_fast) ├── mamba/ # Mamba-related (causal_conv1d, state_update) ├── moe/ # MoE-related (mul_add, zero_experts) └── sample/ # Speculative decoding (verify_tree_greedy) ``` **Development steps:** 1. Create a new `.py` file in the appropriate category subdirectory. 2. Write the kernel using the Triton language, using existing operators in the same directory as templates. 3. Export functions in the corresponding `__init__.py` if needed. 4. Write tests under `tests/python/sgl_kernel_npu/`. **Note:** Many Triton operators are adapted from SGLang's GPU Triton kernels (e.g., comments in `fla/utils.py` note the original source). Pay special attention to differences between NPU and GPU when adapting. ## Integrating Operators into SGLang ### Ascend C Operator Integration After completing the [Steps 1-7](#developing-ascend-c-operators) above (writing the kernel, registering the torch op, building), and installing the `sgl-kernel-npu` wheel, call the operator in SGLang as follows: ```python theme={null} import sgl_kernel_npu # Loading the library auto-triggers libsgl_kernel_npu.so loading # Call the operator result = torch.ops.npu.helloworld(x, y) ``` Real-world usage in SGLang (from `sglang/srt/speculative/eagle_utils.py`): ```python theme={null} torch.ops.npu.build_tree_kernel_efficient( parent_list, selected_index, verified_seq_len, tree_mask, positions, retrive_index, retrive_next_token, retrive_next_sibling, topk, depth, draft_token_num, tree_mask_mode ) ``` ### Triton Operator Integration Import and call directly via Python: ```python theme={null} from sgl_kernel_npu.attention.decode_attention import decode_attention_fwd from sgl_kernel_npu.norm.rmsnorm_bias import rmsnorm_bias from sgl_kernel_npu.mamba.causal_conv1d import causal_conv1d_fwd # Direct function call output = decode_attention_fwd(q, k, v, ...) ``` Real-world usage in SGLang (from `sglang/srt/models/llama.py`): ```python theme={null} from sgl_kernel_npu.norm.split_qkv_rmsnorm_rope import split_qkv_rmsnorm_rope ``` ### sgl-kernel-npu Wheel Update Process Since SGLang and sgl-kernel-npu are separate Python packages, dependency updates require a multi-PR workflow: 1. **Submit sgl-kernel-npu PR**: Add/modify operators in the sgl-kernel-npu repository, ensuring all tests pass. 2. **Bump sgl-kernel-npu version**: Update the version number in sgl-kernel-npu. Merging triggers an automatic PyPI release. 3. **Reference the new version in SGLang**: * Update the `sgl-kernel-npu` version requirement in SGLang's `python/pyproject.toml`. * Use the new operator in SGLang code. If not urgent, you can wait for a regular release (typically within one week). ## Writing Unit Tests Each operator needs a corresponding unit test under `tests/python/sgl_kernel_npu/`, using Python's `unittest` framework. Test file naming convention: `test_.py` ```python theme={null} # tests/python/sgl_kernel_npu/test_helloworld.py import unittest import torch import sgl_kernel_npu class TestHelloworld(unittest.TestCase): def test_helloworld_basic(self): x = torch.randn(2048, dtype=torch.float16, device="npu") y = torch.randn(2048, dtype=torch.float16, device="npu") z = torch.ops.npu.helloworld(x, y) expected = x + y torch.testing.assert_close(z, expected) if __name__ == "__main__": unittest.main() ``` Run tests: ```bash theme={null} python tests/python/sgl_kernel_npu/test_helloworld.py ``` **Testing checklist:** * Cover typical input shapes (power-of-2 sizes and non-standard sizes). * Cover different data types (bf16 / fp16, etc.). * For operators with in-place behavior, verify correctness of output tensors. * Compare against PyTorch native computation to verify accuracy. ## Code Style ### Pre-commit Checks sgl-kernel-npu uses pre-commit for consistent code style: ```bash theme={null} pip3 install pre-commit cd sgl-kernel-npu pre-commit install pre-commit run --all-files ``` **Note:** If `pre-commit run --all-files` fails the first time, run it again to ensure all lint errors are auto-fixed. All code must pass checks before submitting a PR. ### C++ Code Style * Use the C++17 standard. * Place all operator implementations under the `sglang::npu_kernel` namespace. * Follow existing code style; format using `.clang-format`. * Use `TORCH_CHECK` for error checking (not the standard GE `OP_ADD` macros). * Do not include unnecessary GE registration code (e.g., `OP_ADD()` macros). ### Python Code Style * Follow PEP 8. * Use snake\_case for file and function names. * When adapting from SGLang GPU code, note the original source in the file header. ### General Principles * Avoid code duplication: extract shared functions for any repeated code blocks over 5 lines. * Minimize device synchronization: reduce CPU-NPU sync operations like `tensor.item()` or `tensor.cpu()`. * Keep functions pure: avoid in-place argument modification. * Keep files concise: split files exceeding 2,000 lines. ## Submitting a PR 1. **Fork the repo**: Fork [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) on GitHub, then clone locally. 2. **Create a branch**: Create a new branch from `main`, e.g., `feature/add-my-op`. 3. **Develop and test**: Develop the operator and write tests following the steps above. Ensure all tests pass. 4. **Run pre-commit**: Ensure code formatting compliance. 5. **Commit and push**: ```bash theme={null} git add . git commit -m "feat: add operator" git push origin feature/add-my-op ``` 6. **Create a PR**: Open a Pull Request on GitHub from your branch to `sgl-project/sgl-kernel-npu:main`. 7. **Wait for CI and review**: CI checks include linting, compilation, and operator tests. After passing, wait for maintainer review and merge. ## References * [SGL-Kernel-NPU Official Repository](https://github.com/sgl-project/sgl-kernel-npu) * [SGL-Kernel-NPU Contribution Guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/docs/developer_guide/contribution_guide.md) * [Ascend C Kernel Development Guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/opdevg/Ascendcopdevg/atlas_ascendc_10_0001.html) * [PyTorch Custom Ops Schema Reference](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/README.md#func) * [helloworld Example Operator](https://github.com/sgl-project/sgl-kernel-npu/tree/main/csrc/helloworld) * [SGLang Contribution Guide](/docs/hardware-platforms/ascend-npus/development/contribution_guide) # Operator Performance Optimization Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/development/operator_performance_optimizing ## Performance\_benchmark ### Obtaining Performance Data Before optimizing the performance, you need to obtain accurate performance data, understand the current performance status, and analyze the next optimization direction based on the performance status. MindStudio provides realistic methods for testing the performance of Triton operators. #### Device-end The msProf tool is used to collect and analyze key performance indicators of operators running on the Ascend AI Processor. You can use the output performance data to quickly locate the software and hardware performance bottlenecks of operators and improve the efficiency of operator performance analysis. ```bash theme={null} msprof op python3 test_xxxxx.py ``` The following is a case of using msprof for data collection. | Attribute | Value | | - | - | | Name | DequantSwigluQuant\_int32\_high\_performance\_100000000 | | Type | DequantSwigluQuant | | OP State | static | | Accelerator Core | AI\_VECTOR\_CORE | | Start Time(us) | 1774489226717521.715 | | Duration(us) | 102.824 | | Wait Time(us) | 0 | | Block Dim | 36 | | Mix Block Dim | 0 | | HF32 Eligible | NO | | Input Shapes | 163840,1024;128,1024;163840;;;;128 | | Input Data Types | INT32;FLOAT;FLOAT;DT\_UNDEFINED;DT\_UNDEFINED;DT\_UNDEFINED;INT64 | | Input Formats | ND;ND;ND;NULL;NULL;NULL;ND | | Output Shapes | 163840,512;163840 | | Output Data Types | INT8;FLOAT | | Output Formats | ND;ND | | Context ID | N/A | | aicore\_time(us) | 0 | | aic\_total\_cycles | 0 | | aic\_mac\_time(us) | 0 | | aic\_mac\_ratio | 0 | | aic\_scalar\_time(us) | 0 | | aic\_scalar\_ratio | 0 | | aic\_mte1\_time(us) | 0 | | aic\_mte1\_ratio | 0 | | aic\_mte2\_time(us) | 0 | | aic\_mte2\_ratio | 0 | | aic\_fixpipe\_time(us) | 0 | | aic\_fixpipe\_ratio | 0 | | aic\_icache\_miss\_rate | 0 | | aiv\_time(us) | 59.128 | | aiv\_total\_cycles | 3512188 | | aiv\_vec\_time(us) | 36.708 | | aiv\_vec\_ratio | 0.621 | | aiv\_scalar\_time(us) | 41.403 | | aiv\_scalar\_ratio | 0.7 | | aiv\_mte2\_time(us) | 11.975 | | aiv\_mte2\_ratio | 0.203 | | aiv\_mte3\_time(us) | 9.738 | | aiv\_mte3\_ratio | 0.165 | | aiv\_icache\_miss\_rate | 0.005 | | cube\_utilization(%) | 0 | Below is the field-by-field breakdown of the operator performance record, aligned with the official specification. ### 1. Basic Identification Fields | Field | Value | Definition (per official docs) | | - | - | - | | Name | DequantSwigluQuant\_int32\_high\_performance\_100000000 | **Op Name**: Name of the fused operator (dequantization + SwiGLU activation + quantization), with an int32 high-performance implementation suffix. | | Type | DequantSwigluQuant | **OP Type**: Functional category of the operator. | | OP State | static | **OP State**: Indicates a static operator whose shape and scheduling logic are determined at compile time. | | Accelerator Core | AI\_VECTOR\_CORE | **Task Type**: The operator runs on the AI Vector Core; other common types include AI\_CORE (matrix computation core) and AI\_CPU. | ### 2. Timing & Scheduling Fields | Field | Value | Definition (per official docs) | | - | - | - | | Start Time(us) | 1774489226717521.715 | **Task Start Time**: Absolute start timestamp of the operator task on the device side, in microseconds. | | Duration(us) | 102.824 | **Task Duration**: End-to-end total latency of the operator, including dispatch time, accelerator execution time, and completion response time, in microseconds. | | Wait Time(us) | 0 | **Task Wait Time**: Time interval between the end of the previous task and the start of the current task. A value of 0 means no idle wait between task dispatches. | ### 3. Core Configuration & Precision Fields | Field | Value | Definition (per official docs) | | - | - | - | | Block Dim | 36 | **Block Num**: Number of parallel thread blocks for the operator task, corresponding to Block Dim in the SIMT programming model. One AI Vector Core executes only one thread block at a time, so this value reflects the scale of occupied parallel compute resources. | | Mix Block Dim | 0 | **Mix Block Num**: Number of blocks on the secondary accelerator if the operator runs on both AI Core and Vector Core. A value of 0 means the operator runs exclusively on AI\_VECTOR\_CORE with no hybrid core scheduling. | | HF32 Eligible | NO | **HF32 Eligible**: Indicates whether the HF32 high-precision floating-point format is enabled; `NO` means it is not used. This field is reported only at the `--task-time=l1` collection level. | ### 4. Input & Output Information | Field | Value | Definition (per official docs) | | - | - | - | | Input Shapes | 163840,1024;128,1024;163840;;;;128 | **Input Shapes**: Dimensions of each input tensor, separated by semicolons; empty values represent scalar inputs. Breakdown: 7 inputs with shapes `[163840,1024]`, `[128,1024]`, `[163840]`, 3 scalars, and `[128]`. | | Input Data Types | INT32;FLOAT;FLOAT;DT\_UNDEFINED;DT\_UNDEFINED;DT\_UNDEFINED;INT64 | **Input Data Types**: Data types of inputs, in the same order as input shapes. | | Input Formats | ND;ND;ND;NULL;NULL;NULL;ND | **Input Formats**: Memory layout of inputs; `ND` stands for N-dimensional tensor format, `NULL` corresponds to scalar/undefined inputs. | | Output Shapes | 163840,512;163840 | **Output Shapes**: Dimensions of two output tensors, separated by semicolons: `[163840,512]` and `[163840]`. | | Output Data Types | INT8;FLOAT | **Output Data Types**: Output 1 is INT8 (quantized result), output 2 is FLOAT. | | Output Formats | ND;ND | **Output Formats**: Both outputs use standard ND layout. | | Context ID | N/A | **Context ID**: Identifier for sub-tasks at Sub Task granularity; N/A means no sub-task splitting for this operator. | ### 5. AI Core Performance Metrics (aic\_\* series) | Field | Value | Definition (per official docs) | | - | - | - | | aicore\_time(us) | 0 | Theoretical execution time on AI Core, in microseconds. | | aic\_total\_cycles | 0 | Total execution cycles on AI Core. | | aic\_mac\_time(us) / aic\_mac\_ratio | 0 / 0 | Latency and cycle ratio of cube (matrix multiplication) instructions. | | aic\_scalar\_time(us) / aic\_scalar\_ratio | 0 / 0 | Latency and cycle ratio of scalar instructions. | | aic\_mte1\_time(us) / aic\_mte1\_ratio | 0 / 0 | Latency and cycle ratio of L1→L0A/L0B data move instructions. | | aic\_mte2\_time(us) / aic\_mte2\_ratio | 0 / 0 | Latency and cycle ratio of DDR→AICORE read move instructions. | | aic\_fixpipe\_time(us) / aic\_fixpipe\_ratio | 0 / 0 | Latency and cycle ratio of L0C→OUT/L1 move instructions. | | aic\_icache\_miss\_rate | 0 | Instruction cache miss rate of AI Core. | ### 6. AI Vector Core Performance Metrics (aiv\_\* series) | Field | Value | Definition & Interpretation | | - | - | - | | aiv\_time(us) | 59.128 | **aiv\_time**: Theoretical execution time on Vector Core under ideal conditions (all blocks scheduled simultaneously with equal duration). In practice, this value is slightly smaller than real execution time due to staggered block startup. | | aiv\_total\_cycles | 3512188 | **aiv\_total\_cycles**: Total cycles executed on Vector Core, summed across all blocks. | | aiv\_vec\_time(us) / aiv\_vec\_ratio | 36.708 / 0.621 | Latency (us) and cycle ratio (62.1%) of vector computation instructions. Vector operations are the core compute workload of this operator. | | aiv\_scalar\_time(us) / aiv\_scalar\_ratio | 41.403 / 0.7 | Latency (us) and cycle ratio (70%) of scalar instructions. The sum exceeds 100% because scalar and vector pipelines run in parallel with independent cycle counters. | | aiv\_mte2\_time(us) / aiv\_mte2\_ratio | 11.975 / 0.203 | Latency and cycle ratio (20.3%) of read memory move instructions (DDR/on-chip memory → Vector Core). | | aiv\_mte3\_time(us) / aiv\_mte3\_ratio | 9.738 / 0.165 | Latency and cycle ratio (16.5%) of write memory move instructions (Vector Core → DDR/on-chip memory). | | aiv\_icache\_miss\_rate | 0.005 | Vector Core instruction cache miss rate of 0.5%, extremely low and indicates efficient instruction fetch. | ### 7. Utilization Metrics | Field | Value | Definition (per official docs) | | - | - | - | | cube\_utilization(%) | 0 | **cube\_utilization**: Utilization rate of the matrix multiplication unit. The value is 0 because the operator is purely vector-based. | ## Optimization ### Specification #### 1. Ascend core compute units * AI Core: the core that actually performs matrix/vector computation * Vector Unit: responsible for SIMD computation (similar to CUDA Core) * Scalar Unit: responsible for control/loop * L0/L1/L2 cache: The smaller the size, the faster the speed. L0 is only 64KB, L1 is 256KB, and L2 is shared. #### 2. Ascend memory hierarchy (from fastest to slowest) * Register → Fastest * L0/L1 cache → Very fast * On-chip cache (L2) → Fast * DDR (host memory) → Slowest ### 3. Characteristics of Ascend instructions Good at accessing large contiguous memory blocks Dislikes discrete access, stride access, and random access Must be 128-bit/256-bit aligned Must be vectorized. ### Tips 1. A2/A3 Series products usually have only 40 or 48 vector cores. If the number of grids exceeds 40 or 48 vector cores, the grids will be delivered in a queue, resulting in a long waiting time. Therefore, the number of cores for high-performance implementation does not exceed the number of vector cores. 2. Try to use up all the UB as much as possible. Move a large block size at a time to ensure that the bound is in the MTE. No Redundant Copy. 3. If the offset is a negative number, the current triton-ascend considers it as a discrete memory access scenario. As a result, the performance severely deteriorates, and the data is read from the entire DMA block instead of being read in scalar mode. 4. The UB of the Ascend hardware requires that the size of the tail axis of the tensor can be exactly divided by 32bytes. If the length of the tail axis is insufficient, the length of the tail axis is automatically supplemented. For example, the performance deteriorates exponentially due to automatic supplementation for the Tensor whose shape is (2048, 3). In this situation, you can perform the transposition operation to change the alignment axis to a lower dimension. In addition, the transposition operation is affected by the automatic supplement rule. Therefore, special skills are also required to avoid supplementation. 5. Use Double Buffer, parallelizes computation and data transfer. While computing one block of data, another block of data is being transferred to L1. 6. If hostbound behavior is severe, core binding can be used to address it. # How to Support New Models Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/development/support_new_models This document explains how to add support for new language models and multimodal large language models (MLLMs) in SGLang. It also covers how to test new models and register external implementations. ## How to Support a New Language Model To support a new model in SGLang, you only need to add a single file under the [SGLang Models Directory](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/models). You can learn from existing model implementations and create a new file for your model. For most models, you should be able to find a similar model to start with (e.g., starting from Llama). Also refer to how to [port a Model from vLLM to SGLang](#port-a-model-from-vllm-to-sglang) NPU adaptations are embedded in existing model files (e.g., `llama.py`, `qwen3_vl.py`) through `_is_npu` conditional branches. The NPU hardware backend lives at `sglang/srt/hardware_backend/npu/`. Some ops may need to use `torch_npu` APIs in place of CUDA equivalents. ## How to Support a New Multimodal Large Language Model To support a new multimodal large language model (MLLM) in SGLang, there are several key components in addition to the standard LLM support: 1. **Register your new model as multimodal**: Extend `is_multimodal_model` in [model\_config.py](https://github.com/sgl-project/sglang/blob/0ab3f437aba729b348a683ab32b35b214456efc7/python/sglang/srt/configs/model_config.py#L561) to return `True` for your model. 2. **Register a new chat-template**: Only when your default chat-template is unable to accept images as input: Register a new chat template in [conversation.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/parser/conversation.py) and the corresponding matching function. 3. **Multimodal Data Processor**: Define a new `Processor` class that inherits from `BaseMultimodalProcessor` and register this processor as your model’s dedicated processor. See [Multimodal Processors](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/multimodal/processors) for more details. 4. **Handle Multimodal Tokens**: Implement a `pad_input_ids` function for your new model. In this function, multimodal tokens in the prompt should be expanded (if necessary) and padded with multimodal-data-hashes so that SGLang can recognize different multimodal data with `RadixAttention`. 5. **Handle Image Feature Extraction**: Implement a `get_image_feature` function for your new model, which extracts image features from raw image data and converts them into the embeddings used by the language model. 6. **Adapt to Vision Attention**: Adapt the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`. You can refer to [Qwen2VL](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/qwen2_vl.py) or other mllm implementations. These models demonstrate how to correctly handle both multimodal and textual inputs. On Ascend NPU, ensure vision processors and image feature extraction are compatible with the `torch_npu` backend. Refer to `vit_npu_graph_runner.py` under `hardware_backend/npu/graph_runner/` and `qwen_vl_processor.py` under `hardware_backend/npu/modules/` for NPU vision adaptation patterns. ## Testing and Debugging Please note all your testing and benchmarking results in PR description. ### Benchmark * **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow the [benchmark and profiling guide](/docs/developer_guide/benchmark_and_profiling) to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer). * **(Optional) Other evals**: If you ran other evals, please note the results in PR description. For NPU-adapted models: add the corresponding test under `test/registered/npu/` and verify correctness on Ascend NPU hardware; run benchmarks on the NPU device and report performance metrics (TTFT, throughput), comparing against SGLang GPU results as the primary baseline. Fall back to HF Transformer comparison when no GPU adaptation is available. ## Port a Model from vLLM to SGLang The [vLLM Models Directory](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models) is a valuable resource, as vLLM covers many models. SGLang reuses vLLM’s interface and some layers, making it easier to port models from vLLM to SGLang. To port a model from vLLM to SGLang: * Compare these two files for guidance: * [SGLang Llama Implementation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/llama.py) * [vLLM Llama Implementation](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py) * The major differences include: * **Replace vLLM’s `Attention` with `RadixAttention`** (ensure you pass `layer_id` to `RadixAttention`). * **Replace vLLM’s `LogitsProcessor` with SGLang’s `LogitsProcessor`.** * **(For multimodal models) Replace the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.** * **Replace other vLLM layers** (such as `RMSNorm`, `SiluAndMul`) with SGLang layers. * **Change the `forward()` functions** to accept a `forward_batch: ForwardBatch` argument carrying the full batch state (positions, `input_ids`, KV cache indices, sampling info, etc.). SGLang’s top-level model `forward()` is the entry point and returns the final `LogitsProcessorOutput` directly — it does both the backbone forward and the logits computation, unlike vLLM which splits these into `forward()` + `compute_logits()`. * **Add `EntryClass`** at the end. * **Ensure that the new implementation uses only SGLang components** and does not rely on any vLLM components. * **For Ascend NPU**: Reference existing NPU-adapted models (e.g., `llama.py`, `deepseek_v2.py`) for NPU-specific patterns, such as replacing CUDA kernels with `torch_npu` equivalents. The NPU backend is at `sglang/srt/hardware_backend/npu/`. Note: make sure you add your new model to the supported models list in the [supported models documentation](https://github.com/sgl-project/sglang/blob/main/docs/docs/hardware-platforms/ascend-npus/reference/support_models.mdx). ## Registering an External Model Implementation In addition to the methods above, you can register your new model with the `ModelRegistry` before launching the server. This allows you to integrate your model without modifying the source code. For example: ```python Register Model theme={null} from sglang.srt.models.registry import ModelRegistry from sglang.srt.entrypoints.http_server import launch_server # For a single model, add it to the registry: ModelRegistry.models[model_name] = model_class # For multiple models, you can imitate the import_model_classes() function: from functools import lru_cache @lru_cache() def import_new_model_classes(): model_arch_name_to_cls = {} # Populate model_arch_name_to_cls with your new model classes. ... return model_arch_name_to_cls ModelRegistry.models.update(import_new_model_classes()) # Launch the server with your server arguments: launch_server(server_args) ``` ## Example: Implementing and Serving a Llama Wrapper Model Below is an introductory, step-by-step walkthrough on how to implement a new model end-to-end in SGLang and then run it via the [Offline Engine](/docs/basic_usage/offline_engine_api). ### Implementing Our Model To keep things simple, this new model will be a simple wrapper around [Llama 3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), and our goal will be just to bias the output logits for each `forward` call by taking the square root of each individual logit. Let's start by defining our model in a file called `llama_wrapper.py`. The first step is to import the necessary libraries from SRT, which is SGLang's internal backend. ```python Example theme={null} # In the file `llama_wrapper.py` import torch from transformers import LlamaConfig from typing import Optional from sglang.srt.layers.logits_processor import LogitsProcessorOutput from sglang.srt.layers.quantization.base_config import QuantizationConfig from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors from sglang.srt.models.llama import LlamaForCausalLM ``` Next, we declare a new `class` for our model and have it inherit from `LlamaForCausalLM`, which allows our model to access `LlamaForCausalLM`'s predefined modules and layers, such as `LlamaAttention` and `LlamaMLP`. Note that almost all model implementations take in `config` and `quant_config` as arguments for their `__init__` method; `config` and `quant_config` are passed in via [`model_loader/loader.py`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_loader/loader.py#L219). Because we have inherited from `LlamaForCausalLM`, we can pass our parameters directly to its constructor, which will set the member variables for us. ```python Class Definition theme={null} class LlamaWrapper(LlamaForCausalLM): def __init__( self, config: LlamaConfig, quant_config: Optional[QuantizationConfig] = None, prefix: str = "", ) -> None: super().__init__(config=config, quant_config=quant_config, prefix=prefix) ``` Now, we want to define the `forward` method, which is what will be called at inference time. Note that the signature for `forward` is essentially the same for any model; you can take a look at the other models defined in the [`models` directory](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/) for references. To see where exactly `forward` is called in the SGLang runtime's internals, take a look at [`forward_decode`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1705) and [`forward_extend`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1724) in the [`ModelRunner` class](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/model_executor/model_runner.py). ```python Forward Method Signature theme={null} @torch.no_grad() def forward( self, input_ids: torch.Tensor, positions: torch.Tensor, forward_batch: ForwardBatch, pp_proxy_tensors: Optional[PPProxyTensors] = None, input_embeds: Optional[torch.Tensor] = None, get_embedding: bool = False, ) -> LogitsProcessorOutput: ``` We now call the `__call__` method for `self.model` (which is a member variable that `LlamaForCausalLM` defines in its `__init__` method), which eventually calls `LlamaForCausalLM`'s `forward` method. After that, we feed the `hidden_states` into our model's `LogitsProcessor` (again defined in `LlamaForCausalLM`). ```python Call Model and LogitsProcessor theme={null} hidden_states = self.model( input_ids, positions, forward_batch, input_embeds, pp_proxy_tensors=pp_proxy_tensors, ) res: LogitsProcessorOutput = self.logits_processor( input_ids, hidden_states, self.lm_head, forward_batch, ) ``` After receiving the logits for the next token, we can finally perform our biasing step. ```python Logit Biasing theme={null} orig_logits = res.next_token_logits res.next_token_logits = torch.where( orig_logits > 0, orig_logits.sqrt(), orig_logits ) return res ``` Now, our `LlamaWrapper` model is created and ready to be served! ### Serving Our Model Via SGLang's Offline Engine The next step of this walkthrough involves hosting our new model offline, so that it can be served locally and without an HTTP server. First, create a new file called `run.py`. Now, we must ensure that SGLang's `ModelRegistry` can find our model. To do this, we first download the model's configuration and weights from Huggingface. ```python Example theme={null} # In the file `run.py` import asyncio from functools import lru_cache from huggingface_hub import snapshot_download from llama_wrapper import LlamaWrapper # Make sure to import our new model! import sglang as sgl from sglang.srt.models.registry import ModelRegistry # Make sure to request access to this model on Huggingface, then export your # `HF_TOKEN` to download the model snapshot llama_dir = snapshot_download( repo_id="meta-llama/Llama-3.1-8B-Instruct", local_dir="./llama_ckpt", ) ``` Now that we have our model on disk, we want to point it to `LlamaWrapper` by changing the `architectures` field in `./llama_ckpt/config.json` to be `LlamaWrapper`. That way, when we pass in the path of our model checkpoint to SGLang, it will know that we want to use "LlamaWrapper" instead of "LlamaForCausalLM" as our model. ```json Example theme={null} { "architectures": [ "LlamaWrapper" ], ... } ``` However, if we don't link our `LlamaWrapper` class to the "LlamaWrapper" registry keyword, then SGLang won't be able to find our model. Thus, to register our `LlamaWrapper`, we want to follow the steps in the above section titled "Registering an External Model Implementation". ```python Register LlamaWrapper theme={null} @lru_cache() def import_new_model_classes(): model_arch_name_to_cls = {"LlamaWrapper": LlamaWrapper} return model_arch_name_to_cls ModelRegistry.models.update(import_new_model_classes()) ``` Lastly, when we create our `Engine`, we just pass in the path to the local model directory. Then, our `LlamaWrapper` is ready to be served; for this walkthrough, we will use SGLang `Engine`'s non-streaming asynchronous generation endpoint. ```python Example theme={null} def main(): llm = sgl.Engine(model_path="./llama_ckpt") sampling_params = {"temperature": 0.2, "top_k": 5} prompts = [ "Write a short, neutral self-introduction for a fictional character. Hello, my name is", "Provide a concise factual statement about France’s capital city. The capital of France is", "Explain possible future trends in artificial intelligence. The future of AI is", ] asyncio.run(run_llm(llm, sampling_params, prompts)) llm.shutdown() async def run_llm( llm, sampling_params, prompts, ) -> None: outputs = await llm.async_generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print(f"\nPrompt: {prompt}") print(f"Generated text: {output['text']}") if __name__ == "__main__": main() ``` Now, when we call `python run.py`, we will get the outputs of our newly created model! ## Serving External Models via the Standard CLI The previous sections show how to register a model programmatically via `ModelRegistry` and serve it through the Offline Engine. Similar to vLLM model plugin, there is an alternative that lets you keep using the standard `python -m sglang.launch_server` CLI without modifying any SGLang source code: you can register your model using the `SGLANG_EXTERNAL_MODEL_PACKAGE` environment variable. On Ascend NPU, `--device` and `--attention-backend` are auto-detected and can be omitted from the launch command. SGLang sets the device to `npu` and attention backend to `ascend` automatically when `torch.npu.is_available()`. ### The `EntryClass` Variable When SGLang scans a model package, it looks for the variable `EntryClass` at the module level of your Python file. The [model registry](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/registry.py) imports your file, checks for `EntryClass`, and registers the class assigned to it. If you are using a model based on HuggingFace, the name of this class needs to match the `"architectures"` field in your model's `config.json`. For example, if you are implementing a Llama wrapper, add this line at the end of your model file: ```python Example theme={null} # This is what "Add EntryClass at the end" means EntryClass = LlamaWrapper ``` ### Example: Text-Only Model Using the same Llama wrapper from the previous section, here is how to package and serve it via the CLI. 1. Create your project ```text theme={null} sglang_custom_project/ |----setup.py |----custom_llm/ |----__init__.py |----llama_wrapper.py ``` Write the `setup.py`: ```python Example theme={null} # sglang_custom_project/setup.py from setuptools import setup, find_packages setup( name="sglang-custom-plugins", version="0.1", packages=find_packages(), ) ``` 2. Write your model code Inside `llama_wrapper.py`, write your model and include `EntryClass`: ```python Example theme={null} # sglang_custom_project/custom_llm/llama_wrapper.py import torch from typing import Optional from sglang.srt.layers.logits_processor import LogitsProcessorOutput from sglang.srt.layers.quantization.base_config import QuantizationConfig from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors from sglang.srt.models.llama import LlamaForCausalLM class LlamaWrapper(LlamaForCausalLM): def __init__(self, config, quant_config: Optional[QuantizationConfig] = None, prefix: str = "") -> None: super().__init__(config=config, quant_config=quant_config, prefix=prefix) @torch.no_grad() def forward(self, input_ids, positions, forward_batch, pp_proxy_tensors=None, input_embeds=None, get_embedding=False): hidden_states = self.model( input_ids, positions, forward_batch, input_embeds, pp_proxy_tensors=pp_proxy_tensors, ) res: LogitsProcessorOutput = self.logits_processor( input_ids, hidden_states, self.lm_head, forward_batch, ) orig = res.next_token_logits res.next_token_logits = torch.where(orig > 0, orig.sqrt(), orig) return res # Don't forget to add EntryClass EntryClass = LlamaWrapper ``` 3. Install your package Run this inside your `sglang_custom_project` directory to install your code into the active Python environment: ```bash Command theme={null} pip install -e . ``` 4. Update your `config.json` Update the `config.json` under your HuggingFace model checkpoint directory so the `architectures` field matches your class name: ```json Config theme={null} { "architectures": ["LlamaWrapper"], ... } ``` 5. Launch the server Set the environment variable before running the CLI: ```bash Command theme={null} export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_llm python -m sglang.launch_server \ --model-path /path/to/Llama-3.1-8B-Instruct \ --port 8000 ``` The `SGLANG_EXTERNAL_MODEL_PACKAGE` should be the parent folder name containing your model-related code. In this example, it should be `custom_llm`. ### Example: Multimodal Model If you are working with multimodal models, setting `SGLANG_EXTERNAL_MODEL_PACKAGE` alone is not enough. SGLang also needs to recognize your architecture as multimodal to enable the image/video processing pipelines, and it needs a custom processor. You can handle this by setting two additional environment variables: * `SGLANG_EXTERNAL_MM_MODEL_ARCH`: Adds your architecture name to SGLang's internal list of multimodal models. * `SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE`: Tells SGLang where to find your custom processor class. For example, let's build a custom model based on Qwen2-VL-Instruct that takes the square root of the logits. Create the project: ```text theme={null} sglang_custom_project_vl/ |----setup.py |----custom_vlm/ |----__init__.py |----qwenvl_wrapper.py ``` Write `setup.py`: ```python Example theme={null} # sglang_custom_project_vl/setup.py from setuptools import setup, find_packages setup( name="sglang-custom-plugins-vl", version="0.1", packages=find_packages(), ) ``` Write the model in `qwenvl_wrapper.py`: ```python Example theme={null} # sglang_custom_project_vl/custom_vlm/qwenvl_wrapper.py import torch from sglang.srt.models.qwen2_vl import Qwen2VLForConditionalGeneration from sglang.srt.multimodal.processors.qwen_vl import QwenVLImageProcessor class CustomQwen2VL(Qwen2VLForConditionalGeneration): def forward(self, input_ids, positions, forward_batch, input_embeds=None, get_embedding=False): res = super().forward( input_ids, positions, forward_batch, input_embeds=input_embeds, get_embedding=get_embedding ) if not get_embedding: orig = res.next_token_logits res.next_token_logits = torch.where(orig > 0, orig.sqrt(), orig) return res class CustomQwen2VLProcessor(QwenVLImageProcessor): models = [CustomQwen2VL] def __init__(self, hf_config, server_args, _processor, *args, **kwargs): super().__init__(hf_config, server_args, _processor, *args, **kwargs) EntryClass = CustomQwen2VL ``` **Note:** you don't need a separate `EntryClass` for the custom processor as long as you associate the processor with the specific model class. Install the package, update `config.json`, and launch: ```bash Command theme={null} pip install -e . ``` ```json Config theme={null} { "architectures": ["CustomQwen2VL"], ... } ``` ```bash Command theme={null} export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_vlm export SGLANG_EXTERNAL_MM_MODEL_ARCH=CustomQwen2VL export SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE=custom_vlm python -m sglang.launch_server \ --model-path /path/to/Qwen2-VL-2B-Instruct \ --port 8000 \ --enable-multimodal ``` ## Documentation Add to table of supported models in [generative\_models.mdx](/docs/supported-models/generative_models) or [multimodal\_language\_models.mdx](/docs/supported-models/multimodal_language_models) For NPU-adapted models, also add entries to the NPU support models table in [reference/support\_models.mdx](../reference/support_models). *** By following these guidelines, you can add support for new language models and multimodal large language models in SGLang and ensure they are thoroughly tested and easily integrated into the system. # Accuracy Evaluation Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation # Ascend NPU Accuracy Evaluation This document describes how to perform accuracy evaluation for SGLang models running on Ascend NPU using a tool: **EvalScope**. The following scenarios are covered: * **Online Testing**: Evaluate via API interface after starting SGLang server * **Text Models**: Using Qwen2.5-7B-Instruct as example * **Multimodal Models**: Using Qwen2.5-VL-7B-Instruct as example *** ## Environment Setup Ensure sufficient disk space before proceeding. The Docker image requires at least **30GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. First, launch the SGLang environment using the provided container image: ```shell Command theme={null} export IMAGE=quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \ --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \ --device=/dev/davinci_manager \ --device=/dev/hisi_hdc \ --volume /usr/local/sbin:/usr/local/sbin \ --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --volume /etc/ascend_install.info:/etc/ascend_install.info \ --volume /var/queue_schedule:/var/queue_schedule \ --volume ~/.cache/:/root/.cache/ \ --entrypoint=bash \ $IMAGE ``` ```shell Command theme={null} export IMAGE=quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device=/dev/davinci_manager \ --device=/dev/hisi_hdc \ --volume /usr/local/sbin:/usr/local/sbin \ --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --volume /etc/ascend_install.info:/etc/ascend_install.info \ --volume /var/queue_schedule:/var/queue_schedule \ --volume ~/.cache/:/root/.cache/ \ --entrypoint=bash \ $IMAGE ``` *** ## Using EvalScope [EvalScope](https://github.com/modelscope/evalscope) is a comprehensive model evaluation framework from ModelScope, supporting both accuracy evaluation and performance stress testing. ### Install EvalScope ```shell Command theme={null} # Method 1: Installing via pip pip install evalscope # Method 2: Installing from source git clone https://github.com/modelscope/evalscope.git cd evalscope/ pip install -e . ``` ### Online Text Model Testing This section covers online evaluation scenarios where the SGLang server is already running. #### Start SGLang Server ```shell Command theme={null} # Set HuggingFace mirror (if network access is restricted) export HF_ENDPOINT=https://hf-mirror.com # Start text model server sglang serve --model-path /home/weights/Qwen2.5-7B-Instruct --attention-backend ascend --host 0.0.0.0 --port 30000 & ``` For more details of SGLang server, refer to the [Ascend NPU Quick Start](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) #### Execute Accuracy Evaluation Use sgl-eval for GSM8K and MMLU. It owns the datasets, chat prompts and graders. EvalScope remains available for the other benchmarks below: ```shell Command theme={null} pip install sgl-eval sgl-eval run gsm8k \ --model /home/weights/Qwen2.5-7B-Instruct \ --base-url http://localhost:30000/v1 \ --num-examples 10 --max-tokens 2048 ``` The run directory contains `metrics.json` and per-example predictions. Use the model name reported by `/v1/models`. #### Common Datasets for Online Evaluation ```shell Command theme={null} # MMLU sgl-eval run mmlu \ --model /home/weights/Qwen2.5-7B-Instruct \ --base-url http://localhost:30000/v1 --max-tokens 2048 # CEval (Chinese evaluation) evalscope eval \ --model /home/weights/Qwen2.5-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets ceval # MATH-500 evalscope eval \ --model /home/weights/Qwen2.5-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets math_500 # HumanEval (code generation) evalscope eval \ --model /home/weights/Qwen2.5-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets humaneval ``` ### Online Multimodal Model Testing #### Start Multimodal Model Server ```shell Command theme={null} # Start multimodal model server (Qwen2.5-VL-7B-Instruct) # Multimodal models require both --attention-backend and --mm-attention-backend sglang serve --model-path /home/weights/Qwen2.5-VL-7B-Instruct \ --attention-backend ascend \ --mm-attention-backend ascend_attn \ --host 0.0.0.0 --port 30000 & ``` #### Execute Multimodal Accuracy Evaluation ```shell Command theme={null} # MMBench (multimodal evaluation) evalscope eval \ --model /home/weights/Qwen2.5-VL-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets mm_bench # MMMU (multimodal comprehensive understanding) evalscope eval \ --model /home/weights/Qwen2.5-VL-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets mmmu # HallusionBench (hallucination evaluation) evalscope eval \ --model /home/weights/Qwen2.5-VL-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets hallusion_bench ``` For more details, refer to the [EvalScope documentation](https://evalscope.readthedocs.io/). *** ## Troubleshooting ### SGLang Server Startup Failure 1. Verify device mapping: A2 Series uses `davinci[0-7]`, A3 Series uses `davinci[0-15]` 2. Confirm image tag matches device type: A2 Series uses `...-910b`, A3 Series uses `...-a3` 3. Check NPU status with `npu-smi info` 4. First run requires model download; set `HF_ENDPOINT=https://hf-mirror.com` if network access is restricted ### EvalScope Connection Failure to Server 1. Confirm SGLang server started successfully (look for `Application startup complete` in logs) 2. Verify `--api-url` points to the correct port (SGLang defaults to `30000`) 3. Ensure URL ends with `/v1`, e.g., `http://localhost:30000/v1` ### EvalScope SSL certificate verification failed When using EvalScope commands without specifying a dataset or model path, it will attempt to download automatically, which may encounter an SSL certificate verification error: ```text theme={null} File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py", line 605, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py", line 592, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py", line 706, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/adapters.py", line 676, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='www.modelscope.cn', port=443): Max retries exceeded with url: /api/v1/datasets/AI-ModelScope/ceval (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1016)'))) [ERROR] 2026-05-13-02:20:01 (PID:876, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception ``` **Temporary workaround (test only):** Navigate to `/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py`, find the `class Session` definition, and set `self.verify = False`. This **disables TLS certificate validation globally** for the Python `requests` library. Use it **only as a temporary diagnostic step** in isolated test environments — never in production. **Stable solution:** The error is caused by a corporate TLS proxy injecting a self-signed certificate. Point `requests` to the proxy's CA bundle: ```shell theme={null} # Obtain the CA certificate from your network administrator # Then set the environment variable: export REQUESTS_CA_BUNDLE=/path/to/your-proxy-ca-bundle.crt ``` This is a common workaround for corporate proxy environments. If it does not resolve your issue, consult your IT department — proxy configurations vary across organizations. If you cannot obtain the CA certificate, download datasets manually as shown in [Download Dataset Error](#download-dataset-error) below. ### EvalScope Request Retry Timeout If EvalScope keeps retrying requests with errors like: ``` 2026-06-22 03:09:03 - evalscope - WARNING: Attempt 4 / 5 failed: ....... Retrying... 2026-06-22 03:09:14 - evalscope - INFO: Evaluating[ceval] 0%| 0/520 [Elapsed: 02:00 < Remaining: ?, ?it/s] 2026-06-22 03:09:19,557 - openai._base_client - INFO: Retrying request to /chat/completions in 0.447260 seconds 2026-06-22 03:09:26,088 - openai._base_client - INFO: Retrying request to /chat/completions in 0.992551 seconds ``` This is usually caused by the HTTP proxy intercepting requests to the local SGLang server. Disable the proxy with: ```shell Command theme={null} unset http_proxy unset https_proxy unset HTTP_PROXY unset HTTPS_PROXY ``` ### Download Dataset Error For this error ```text theme={null} root@localhost:/home/# wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --2026-05-12 12:08:01-- https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv Connecting to :... connected. ERROR: cannot verify www.modelscope.cn's certificate, issued by ‘’: Self-signed certificate encountered. To connect to www.modelscope.cn insecurely, use `--no-check-certificate`. ``` You can add `--no-check-certificate` ```bash theme={null} wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --no-check-certificate ``` For additional assistance, refer to [SGLang GitHub Issues](https://github.com/sgl-project/sglang/issues). # Performance Testing Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/evaluation/performance_testing This page walks through performance testing your SGLang deployment on Ascend NPUs. We cover three model types — text generation (`Qwen/Qwen2.5-7B-Instruct`), multimodal vision (`Qwen/Qwen2.5-VL-7B-Instruct`), and embedding (`Qwen/Qwen3-Embedding-8B`) — in both online and offline serving modes. You can use [Evalscope](https://evalscope.readthedocs.io/en/latest/), [AISBench](https://ais-bench-benchmark.readthedocs.io/en/latest/), or SGLang's built-in benchmarking tools. The benchmark output examples in this guide are for illustration only. Actual performance depends on your hardware (e.g., Ascend A2 Series vs Ascend A3 Series), model version, SGLang version, and deployment configuration. Always run benchmarks on your own hardware to obtain accurate performance data. ## 1. Prepare ### 1.1 Start SGLang server Launch the server with the appropriate flags for each model type. Make sure SGLang is installed first — see [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) for environment setup. ```shell Command theme={null} # The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded. sglang serve --model-path Qwen/Qwen2.5-7B-Instruct ``` ```shell Command theme={null} # The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded. sglang serve --model-path Qwen/Qwen2.5-VL-7B-Instruct --mm-attention-backend ascend_attn ``` ```shell Command theme={null} # The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded. sglang serve --model-path Qwen/Qwen3-Embedding-8B --is-embedding ``` Add `&` at the end of the command to run the server in the background, or open a new terminal to run the benchmark commands in the following sections. The server binds to `http://127.0.0.1:30000` by default. All online benchmarks below assume the server is running at that address. The `--is-embedding` flag is required for embedding models. ### 1.2 Install benchmarking tools `bench_serving` and `bench_offline_throughput` are built into SGLang and require no extra installation. For Evalscope and AISBench, set up each in its own virtual environment: ```shell Command theme={null} python3 -m venv .evalscope_venv source .evalscope_venv/bin/activate pip install evalscope[perf] -U ``` ```shell Command theme={null} python3 -m venv .aisbench_venv source .aisbench_venv/bin/activate git clone https://github.com/AISBench/benchmark.git cd benchmark/ pip3 install -e ./ --use-pep517 pip3 install -r requirements/api.txt pip3 install -r requirements/extra.txt ``` Run `ais_bench -h` to verify. AISBench requires Python 3.10-3.12. After installation, all AISBench commands must be run from the `benchmark/` directory (the cloned repo root). Set `stream=True` and `ignore_eos=True` in the model config for accurate results. ## 2. Online Service: Text Generation Model Test `Qwen/Qwen2.5-7B-Instruct` via the online serving endpoint. Before running any benchmark in this section, make sure the SGLang text-generation server is running at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) for the launch command. For performance testing, prefer random datasets (`--dataset random`, `--dataset-name random`) over real datasets. Random datasets let you pin `--min-prompt-length` / `--max-prompt-length` and `--min-tokens` / `--max-tokens` to fixed values, producing consistent, repeatable results. Real datasets (ShareGPT, openqa, etc.) have variable input lengths that add noise and make cross-run comparisons unreliable. ### 2.1 Using Evalscope Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang server running at `http://127.0.0.1:30000`. Run the following command to run a performance test against the server: ```shell Command theme={null} evalscope perf \ --parallel 10 \ --number 20 \ --model Qwen/Qwen2.5-7B-Instruct \ --url http://127.0.0.1:30000/v1/chat/completions \ --api openai \ --dataset random \ --max-tokens 1024 \ --min-tokens 1024 \ --prefix-length 0 \ --min-prompt-length 1024 \ --max-prompt-length 1024 \ --tokenizer-path Qwen/Qwen2.5-7B-Instruct \ --extra-args '{"ignore_eos": true}' ``` If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id. Example output (for illustration only — actual results depend on your hardware and configuration): ```text theme={null} Benchmarking summary: ┌────────────────────────────┬─────────────┐ │ Metric │ Value │ ├────────────────────────────┼─────────────┤ │ ── General ── │ │ │ Test Duration (s) │ 89.34 │ │ Concurrency │ 10 │ │ Request Rate (req/s) │ -1.00 │ │ Total / Success / Failed │ 20 / 20 / 0 │ │ Req Throughput (req/s) │ 0.22 │ │ ── Latency ── │ │ │ Avg Latency (s) │ 44.67 │ │ TTFT (ms) │ 578.51 │ │ TPOT (ms) │ 43.10 │ │ ITL (ms) │ 43.12 │ │ ── Tokens ── │ │ │ Avg Input Tokens │ 1024.00 │ │ Avg Output Tokens │ 1024.00 │ │ Output Throughput (tok/s) │ 229.24 │ │ Total Throughput (tok/s) │ 458.49 │ │ ── Speculative Decoding ── │ │ │ Decoded Tok/Iter │ 1.00 │ │ Spec. Accept Rate │ 0.00 │ └────────────────────────────┴─────────────┘ Percentile results: ┌────────────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐ │ Metric │ 1% │ 5% │ 10% │ 25% │ 50% │ 75% │ 90% │ 95% │ 99% │ ├────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤ │ Latency (s) │ 44.47 │ 44.47 │ 44.47 │ 44.47 │ 44.86 │ 44.86 │ 44.86 │ 44.86 │ 44.86 │ │ TTFT (ms) │ 138.12 │ 142.07 │ 426.17 │ 426.87 │ 783.67 │ 785.26 │ 786.85 │ 787.97 │ 787.97 │ │ ITL (ms) │ 41.84 │ 42.14 │ 42.22 │ 42.36 │ 42.57 │ 42.80 │ 42.99 │ 49.24 │ 49.84 │ │ TPOT (ms) │ 42.71 │ 42.71 │ 42.71 │ 43.05 │ 43.08 │ 43.43 │ 43.43 │ 43.71 │ 43.71 │ │ Input tokens │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ │ Output tokens │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ │ Output (tok/s) │ 22.83 │ 22.83 │ 22.83 │ 22.83 │ 23.02 │ 23.03 │ 23.03 │ 23.03 │ 23.03 │ │ Total (tok/s) │ 45.65 │ 45.65 │ 45.65 │ 45.65 │ 46.05 │ 46.05 │ 46.05 │ 46.05 │ 46.05 │ │ Decode (tok/s) │ 22.88 │ 23.03 │ 23.03 │ 23.07 │ 23.21 │ 23.42 │ 23.42 │ 23.42 │ 23.42 │ └────────────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘ ... ``` See the [Evalscope Performance Testing Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html) for full details. ### 2.2 Using AISBench Prerequisites: [AISBench installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .aisbench_venv/bin/activate`). All commands must be run from the `benchmark/` directory. SGLang server running at `http://127.0.0.1:30000`. Set `stream=True` and `ignore_eos=True` in the model config for accurate results. Two files need to be configured for performance testing. First, describe the model and server settings in `ais_bench/benchmark/configs/models/vllm_api/vllm_api_stream_chat.py`: ```python vllm_api_stream_chat.py theme={null} # more details: https://ais-bench-benchmark.readthedocs.io/en/latest/base_tutorials/scenes_intro/performance_benchmark.html from ais_bench.benchmark.models import VLLMCustomAPIChat from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content models = [ dict( attr="service", type=VLLMCustomAPIChat, abbr="vllm-api-stream-chat", path="Qwen/Qwen2.5-7B-Instruct", model="Qwen/Qwen2.5-7B-Instruct", stream=True, request_rate=0, use_timestamp=False, retry=2, api_key="", host_ip="127.0.0.1", host_port=30000, url="", max_out_len=512, batch_size=32, trust_remote_code=False, generation_kwargs=dict( temperature=0.01, ignore_eos=True, ), pred_postprocessor=dict(type=extract_non_reasoning_content), ) ] ``` If the model has already been downloaded, point `path` to the local model path instead of the model id. Second, configure random prompt lengths in `ais_bench/datasets/synthetic/synthetic_config.py`: ```python synthetic_config.py theme={null} # more details: https://ais-bench-benchmark.readthedocs.io/en/latest/advanced_tutorials/synthetic_dataset.html synthetic_config = { "Type":"tokenid", "RequestCount": 10, "TrustRemoteCode": False, "StringConfig" : { "Input" : { "Method": "uniform", "Params": {"MinValue": 1, "MaxValue": 200} }, "Output" : { "Method": "gaussian", "Params": {"Mean": 100, "Var": 200, "MinValue": 1, "MaxValue": 100} } }, "TokenIdConfig" : { "RequestSize": 10, "PrefixLen": 0 } } ``` Run with a synthetic dataset: ```shell Command theme={null} ais_bench --models vllm_api_stream_chat --datasets synthetic_gen_string -m perf ``` Example output (for illustration only — actual results depend on your hardware and configuration): ```text theme={null} ╒══════════════════════════╤═════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════╕ │ Performance Parameters │ Stage │ Average │ Min │ Max │ Median │ P75 │ P90 │ P99 │ N │ ╞══════════════════════════╪═════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════╡ │ E2EL │ total │ 3896.4 ms │ 3081.6 ms │ 4175.3 ms │ 4013.8 ms │ 4123.4 ms │ 4137.1 ms │ 4171.5 ms │ 10 │ ├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤ │ TTFT │ total │ 411.6 ms │ 346.7 ms │ 439.7 ms │ 416.3 ms │ 426.6 ms │ 434.4 ms │ 439.2 ms │ 10 │ ├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤ │ TPOT │ total │ 38.3 ms │ 37.4 ms │ 39.0 ms │ 38.3 ms │ 38.7 ms │ 38.9 ms │ 39.0 ms │ 10 │ ├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤ │ ITL │ total │ 38.7 ms │ 0.0 ms │ 156.5 ms │ 38.9 ms │ 39.0 ms │ 39.2 ms │ 117.1 ms │ 10 │ ├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤ │ InputTokens │ total │ 123.4 │ 34.0 │ 228.0 │ 130.5 │ 170.5 │ 217.2 │ 226.92 │ 10 │ ├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤ │ OutputTokens │ total │ 92.1 │ 69.0 │ 100.0 │ 95.0 │ 99.75 │ 100.0 │ 100.0 │ 10 │ ├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤ │ OutputTokenThroughput │ total │ 23.5937 token/s │ 22.3912 token/s │ 24.2616 token/s │ 23.7399 token/s │ 23.9919 token/s │ 24.2027 token/s │ 24.2557 token/s │ 10 │ ╘══════════════════════════╧═════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════╛ ╒══════════════════════════╤═════════╤══════════════════╕ │ Common Metric │ Stage │ Value │ ╞══════════════════════════╪═════════╪══════════════════╡ │ Benchmark Duration │ total │ 4175.4485 ms │ ├──────────────────────────┼─────────┼──────────────────┤ │ Total Requests │ total │ 10 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Failed Requests │ total │ 0 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Success Requests │ total │ 10 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Concurrency │ total │ 9.3317 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Max Concurrency │ total │ 32 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Request Throughput │ total │ 2.395 req/s │ ├──────────────────────────┼─────────┼──────────────────┤ │ Total Input Tokens │ total │ 1234 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Prefill Token Throughput │ total │ 299.8329 token/s │ ├──────────────────────────┼─────────┼──────────────────┤ │ Total Generated Tokens │ total │ 921 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Input Token Throughput │ total │ 295.5371 token/s │ ├──────────────────────────┼─────────┼──────────────────┤ │ Output Token Throughput │ total │ 220.5751 token/s │ ├──────────────────────────┼─────────┼──────────────────┤ │ Total Token Throughput │ total │ 516.1122 token/s │ ╘══════════════════════════╧═════════╧══════════════════╛ ``` See the [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) for details. ### 2.3 Using bench\_serving SGLang's built-in `bench_serving` requires no extra installation. Make sure the server is running at `http://127.0.0.1:30000` before running the benchmark. See the [Bench Serving Guide](/docs/developer_guide/bench_serving) for all backends, datasets, and advanced options. ```shell Command theme={null} python -m sglang.bench_serving \ --backend sglang-oai \ --base-url http://127.0.0.1:30000 \ --model Qwen/Qwen2.5-7B-Instruct \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 512 \ --random-range-ratio 1 \ --num-prompts 100 \ --max-concurrency 32 ``` `--dataset-name random` samples token IDs from the ShareGPT dataset to generate realistic input; the first run downloads ShareGPT from Hugging Face automatically. 1. If you have network issues, set `export HF_ENDPOINT=https://hf-mirror.com` to use domestic mirror. 2. If downloading still fails, manually download the dataset file `ShareGPT_V3_unfiltered_cleaned_split.json` locally, upload it to your server, then specify the file directory via `--dataset-path` to run offline. Set `--random-range-ratio 1` for fixed input/output lengths (recommended for consistent comparisons) or `0` (default) for uniform distribution. Add `--request-rate` to control the request rate. For all backends, datasets, and advanced options, see the full [Bench Serving Guide](/docs/developer_guide/bench_serving). Example output (for illustration only — actual results depend on your hardware and configuration): ```text theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai Traffic request rate: inf Max request concurrency: 32 Successful requests: 100 Benchmark duration (s): 47.51 Total input tokens: 102400 Total input text tokens: 102400 Total generated tokens: 51200 Total generated tokens (retokenized): 51195 Request throughput (req/s): 2.10 Input token throughput (tok/s): 2155.35 Output token throughput (tok/s): 1077.68 Peak output token throughput (tok/s): 1587.00 Peak concurrent requests: 64 Total token throughput (tok/s): 3233.03 Concurrency: 26.93 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 12793.49 Median E2E Latency (ms): 12940.17 P90 E2E Latency (ms): 13049.86 P99 E2E Latency (ms): 13051.61 ---------------Time to First Token---------------- Mean TTFT (ms): 1423.99 Median TTFT (ms): 1489.29 P99 TTFT (ms): 2325.56 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 22.25 Median TPOT (ms): 22.22 P99 TPOT (ms): 25.08 ---------------Inter-Token Latency---------------- Mean ITL (ms): 22.26 Median ITL (ms): 20.74 P95 ITL (ms): 21.40 P99 ITL (ms): 23.62 Max ITL (ms): 2229.30 ================================================== ``` #### SGLang Serving Benchmark Result — Complete Reference The output format is **hardcoded in `bench_serving.py`**. All formatting decisions — including column widths, alignment, and decimal precision — are statically defined in the source and cannot be changed via command-line arguments. ##### Test Configuration
Parameter Description
Backend The serving backend under test (e.g., sglang, vllm).
Traffic request rate Request generation rate in req/s. inf means maximum rate (concurrency-bounded). trace indicates trace timestamp mode. A fixed value enforces constant inter-arrival time.
Max request concurrency Maximum number of concurrent requests from the client side. Displays not set when unspecified.
##### Core Statistics & Throughput Metrics
Parameter Description Format Specification
Successful requestsTotal number of successfully completed requests (HTTP 200, no generation errors).Integer, no decimal places
Benchmark duration (s)Total elapsed time from first request sent to last response fully received (seconds).2 decimal places
Total input tokensTotal number of input (prompt) tokens across all requests, counted by server-side tokenizer.Integer, no decimal places
Total input text tokensSame as Total input tokens. For multimodal inputs, this may differ.Integer, no decimal places
Total generated tokensTotal number of output tokens actually generated by the server (server-side tokenizer count).Integer, no decimal places
Total generated tokens (retokenized)Output text re-tokenized by the client using its own tokenizer. A large discrepancy indicates tokenizer mismatch or special tokens in output.Integer, no decimal places
Request throughput (req/s)Number of successful requests processed per second. Formula: Successful requests / Benchmark duration (s).2 decimal places
Input token throughput (tok/s)Number of input tokens processed per second. Formula: Total input tokens / Benchmark duration (s).2 decimal places
Output token throughput (tok/s)Number of output tokens generated per second. Formula: Total generated tokens / Benchmark duration (s).2 decimal places
Peak output token throughput (tok/s)Observed instantaneous peak output token generation rate during the test (computed over a sliding window).2 decimal places
Peak concurrent requestsMaximum number of requests being processed simultaneously on the server side. May exceed client-side Max request concurrency due to queueing.Integer, no decimal places
Total token throughput (tok/s)Sum of input and output token throughputs. Formula: Input token throughput + Output token throughput.2 decimal places
ConcurrencyAverage number of concurrent requests during the test (Little's Law). Formula: Sum of all E2E latencies / Benchmark duration.2 decimal places
##### End-to-End Latency (E2E Latency)
StatisticDescriptionFormat
Mean E2E Latency (ms)Arithmetic mean2 decimal places
Median E2E Latency (ms)50th percentile2 decimal places
P90 E2E Latency (ms)90th percentile (90% of requests have latency ≤ this value)2 decimal places
P99 E2E Latency (ms)99th percentile2 decimal places
##### Time to First Token (TTFT)
StatisticDescriptionFormat
Mean TTFT (ms)Arithmetic mean2 decimal places
Median TTFT (ms)50th percentile2 decimal places
P99 TTFT (ms)99th percentile2 decimal places
##### Time per Output Token (TPOT) – Excluding First Token Formula: (E2E Latency - TTFT) / (Number of output tokens - 1)
StatisticDescriptionFormat
Mean TPOT (ms)Arithmetic mean2 decimal places
Median TPOT (ms)50th percentile2 decimal places
P99 TPOT (ms)99th percentile2 decimal places
##### Inter-Token Latency (ITL)
StatisticDescriptionFormat
Mean ITL (ms)Average inter-token interval2 decimal places
Median ITL (ms)50th percentile inter-token interval2 decimal places
P95 ITL (ms)95th percentile (used to detect stalls)2 decimal places
P99 ITL (ms)99th percentile2 decimal places
Max ITL (ms)Maximum observed inter-token interval; useful for identifying severe blocking events2 decimal places
## 3. Online Service: Multimodal Model Test `Qwen/Qwen2.5-VL-7B-Instruct` for vision-language tasks. Before running any benchmark in this section, make sure the SGLang multimodal server is running at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) and use the Multimodal tab for the launch command. For consistent, repeatable results, set `--random-range-ratio 1` to fix input/output lengths, or `0` (default) for uniform distribution. ### 3.1 Using Evalscope Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang multimodal server running at `http://127.0.0.1:30000`. Evalscope's `perf` tool uses the OpenAI-compatible `/v1/chat/completions` endpoint. Use `--dataset random_vl` for randomized multimodal data with image generation: ```shell Command theme={null} evalscope perf \ --parallel 10 \ --number 20 \ --model Qwen/Qwen2.5-VL-7B-Instruct \ --url http://127.0.0.1:30000/v1/chat/completions \ --api openai \ --dataset random_vl \ --min-tokens 1024 \ --max-tokens 1024 \ --prefix-length 0 \ --min-prompt-length 1024 \ --max-prompt-length 1024 \ --image-width 512 \ --image-height 512 \ --image-format RGB \ --image-num 1 \ --tokenizer-path Qwen/Qwen2.5-VL-7B-Instruct \ --extra-args '{"ignore_eos": true}' ``` If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id. ### 3.2 Using AISBench Prerequisites: [AISBench installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .aisbench_venv/bin/activate`). All commands run from the `benchmark/` directory. SGLang multimodal server running at `http://127.0.0.1:30000`. AISBench does not include a built-in multimodal dataset — you must provide your own. First, edit `ais_bench/benchmark/configs/models/vllm_api/vllm_api_stream_chat.py` to configure the vision model: ```python vllm_api_stream_chat.py theme={null} from ais_bench.benchmark.models import VLLMCustomAPIChat from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content models = [ dict( attr="service", type=VLLMCustomAPIChat, abbr="vllm-api-stream-chat", path="Qwen/Qwen2.5-VL-7B-Instruct", model="Qwen/Qwen2.5-VL-7B-Instruct", stream=True, request_rate=0, use_timestamp=False, retry=2, api_key="", host_ip="127.0.0.1", host_port=30000, url="", max_out_len=256, batch_size=16, trust_remote_code=False, generation_kwargs=dict( temperature=0.01, ignore_eos=True, ), pred_postprocessor=dict(type=extract_non_reasoning_content), ) ] ``` If the model has already been downloaded, point `path` to the local model path instead of the model id. Next, download a multimodal dataset such as mmstar: ```shell Command theme={null} # Download the mmstar dataset (from within the benchmark/ directory) cd ais_bench/datasets mkdir mmstar cd mmstar wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv ``` Run the performance test: ```shell Command theme={null} ais_bench --models vllm_api_stream_chat --datasets mmstar_gen -m perf ``` Example output (for illustration only — actual results depend on your hardware and configuration): ```text theme={null} ╒══════════════════════════╤═════════╤═════════════════╤════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤══════╕ │ Performance Parameters │ Stage │ Average │ Min │ Max │ Median │ P75 │ P90 │ P99 │ N │ ╞══════════════════════════╪═════════╪═════════════════╪════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪══════╡ │ E2EL │ total │ 6190.9 ms │ 5071.4 ms │ 8464.8 ms │ 6126.6 ms │ 6475.2 ms │ 6833.5 ms │ 7897.9 ms │ 1500 │ ├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤ │ TTFT │ total │ 693.3 ms │ 96.0 ms │ 2161.5 ms │ 747.4 ms │ 870.9 ms │ 1032.3 ms │ 1620.8 ms │ 1500 │ ├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤ │ TPOT │ total │ 21.6 ms │ 17.8 ms │ 32.1 ms │ 21.3 ms │ 23.1 ms │ 24.5 ms │ 29.1 ms │ 1500 │ ├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤ │ ITL │ total │ 25.5 ms │ 0.0 ms │ 1951.1 ms │ 18.8 ms │ 19.7 ms │ 37.3 ms │ 121.8 ms │ 1500 │ ├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤ │ InputTokens │ total │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 1500 │ ├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤ │ OutputTokens │ total │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 1500 │ ├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤ │ OutputTokenThroughput │ total │ 41.6779 token/s │ 30.243 token/s │ 50.4791 token/s │ 41.7847 token/s │ 44.6424 token/s │ 45.6484 token/s │ 46.0932 token/s │ 1500 │ ╘══════════════════════════╧═════════╧═════════════════╧════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧══════╛ ╒═════════════════════════╤═════════╤══════════════════╕ │ Common Metric │ Stage │ Value │ ╞═════════════════════════╪═════════╪══════════════════╡ │ Benchmark Duration │ total │ 582099.6816 ms │ ├─────────────────────────┼─────────┼──────────────────┤ │ Total Requests │ total │ 1500 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Failed Requests │ total │ 0 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Success Requests │ total │ 1500 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Concurrency │ total │ 15.9532 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Max Concurrency │ total │ 16 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Request Throughput │ total │ 2.5769 req/s │ ├─────────────────────────┼─────────┼──────────────────┤ │ Total Input Tokens │ total │ 0 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Total Generated Tokens │ total │ 384000 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Input Token Throughput │ total │ 0.0 token/s │ ├─────────────────────────┼─────────┼──────────────────┤ │ Output Token Throughput │ total │ 659.6808 token/s │ ├─────────────────────────┼─────────┼──────────────────┤ │ Total Token Throughput │ total │ 659.6808 token/s │ ╘═════════════════════════╧═════════╧══════════════════╛ ``` See the [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) for details. ### 3.3 Using bench\_serving (image dataset) Set `--dataset-name image` for image datasets. `bench_serving` will generate random prompts with image inputs. Make sure the server is running at `http://127.0.0.1:30000` before running the benchmark. See the [Bench Serving Guide](/docs/developer_guide/bench_serving) for the full list of image-related flags. ```shell Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --base-url http://127.0.0.1:30000 \ --model Qwen/Qwen2.5-VL-7B-Instruct \ --dataset-name image \ --random-input-len 1024 \ --random-output-len 512 \ --random-range-ratio 1 \ --num-prompts 32 \ --max-concurrency 16 \ --image-count 1 \ --image-resolution 720p ``` Example output (for illustration only — actual results depend on your hardware and configuration): ```text theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 32 Benchmark duration (s): 51.74 Total input tokens: 73464 Total input text tokens: 35128 Total input vision tokens: 38336 Total generated tokens: 16384 Total generated tokens (retokenized): 9300 Request throughput (req/s): 0.62 Input token throughput (tok/s): 1419.96 Output token throughput (tok/s): 316.68 Peak output token throughput (tok/s): 800.00 Peak concurrent requests: 32 Total token throughput (tok/s): 1736.64 Concurrency: 15.98 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 25841.84 Median E2E Latency (ms): 25842.85 P90 E2E Latency (ms): 26296.42 P99 E2E Latency (ms): 26303.13 ---------------Time to First Token---------------- Mean TTFT (ms): 12211.59 Median TTFT (ms): 14405.77 P99 TTFT (ms): 15837.60 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 26.67 Median TPOT (ms): 21.75 P99 TPOT (ms): 41.89 ---------------Inter-Token Latency---------------- Mean ITL (ms): 26.67 Median ITL (ms): 20.34 P95 ITL (ms): 20.85 P99 ITL (ms): 21.70 Max ITL (ms): 11309.91 ================================================== ``` ## 4. Online Service: Embedding Model Test `Qwen/Qwen3-Embedding-8B` on the embedding API endpoint. Before running any benchmark in this section, make sure the SGLang embedding server is running with `--is-embedding` at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) and use the Embedding tab for the launch command. AISBench does not support embedding endpoints — use `bench_serving` or Evalscope instead. ### 4.1 Using Evalscope Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang embedding server running with `--is-embedding` at `http://127.0.0.1:30000`. Evalscope supports embedding evaluation. For performance testing the embedding API directly: ```shell Command theme={null} evalscope perf \ --parallel 10 \ --number 20 \ --model Qwen/Qwen3-Embedding-8B \ --url http://127.0.0.1:30000/v1/embeddings \ --api openai_embedding \ --dataset random_embedding \ --min-prompt-length 1024 \ --max-prompt-length 1024 \ --tokenizer-path Qwen/Qwen3-Embedding-8B ``` If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id. Evalscope's embedding performance testing support may vary by version. If the `perf` command does not accept the embeddings endpoint, use [`bench_serving` with `--backend sglang-embedding`](#4-2-using-bench_serving-embedding-backend) as the primary option. ### 4.2 Using bench\_serving (embedding backend) `bench_serving` is built into SGLang. Use `--backend sglang-embedding` to target the `/v1/embeddings` endpoint. Make sure the server is running with `--is-embedding` at `http://127.0.0.1:30000`. ```shell Command theme={null} python -m sglang.bench_serving \ --backend sglang-embedding \ --base-url http://127.0.0.1:30000 \ --model Qwen/Qwen3-Embedding-8B \ --dataset-name random \ --random-input-len 512 \ --random-output-len 0 \ --num-prompts 1000 \ --max-concurrency 64 \ --request-rate 32 ``` `--dataset-name random` samples token IDs from the ShareGPT dataset; the first run downloads ShareGPT from Hugging Face automatically. Set `export HF_ENDPOINT=https://hf-mirror.com` if network is not available. Set `--random-output-len 0` for embedding benchmarks — no output tokens are generated. Example output (for illustration only — actual results depend on your hardware and configuration): ```text theme={null} ============ Serving Benchmark Result ============ Backend: sglang-embedding Traffic request rate: 32.0 Max request concurrency: 64 Successful requests: 1000 Benchmark duration (s): 31.86 Total input tokens: 257891 Total input text tokens: 257891 Request throughput (req/s): 31.39 Input token throughput (tok/s): 8094.67 Peak concurrent requests: 62 Concurrency: 6.67 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 212.34 Median E2E Latency (ms): 160.97 P90 E2E Latency (ms): 267.31 P99 E2E Latency (ms): 1445.94 ================================================== ``` ## 5. Offline Performance Testing SGLang's `Engine` API runs inference in-process, without an HTTP server, letting you measure maximum throughput. `bench_offline_throughput` is built into SGLang and requires no extra installation or running server. `bench_offline_throughput` currently only supports text-generation (LLM) benchmarks. Multimodal and embedding models are not supported. ### 5.1 Using bench\_offline\_throughput `bench_offline_throughput` uses the `Engine` API internally and measures pure inference throughput without HTTP overhead: ```shell Command theme={null} python -m sglang.benchmark.offline_throughput \ --model-path Qwen/Qwen2.5-7B-Instruct \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 512 \ --num-prompts 500 ``` `--dataset-name random` samples token IDs from the ShareGPT dataset; the first run downloads ShareGPT from Hugging Face automatically. Set `export HF_ENDPOINT=https://hf-mirror.com` if network is not available. `--dataset-name random` with `--random-input-len` and `--random-output-len` gives you full control over input/output token counts. Fixed-length random data eliminates variance from real datasets, making throughput comparisons across runs deterministic and reliable. ## See also * [Bench Serving Guide](/docs/developer_guide/bench_serving) — all backends, datasets, and advanced options for `bench_serving` * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — environment setup for Ascend NPUs * [Evalscope Performance Testing Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html) — full Evalscope documentation * [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) — full AISBench documentation # Troubleshooting and FAQ Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/faq Contributions are welcome. Feel free to add more. ## 1. Context corruption with GLOO op.preamble.length \<= op.nbytes in PD disaggregation ### Error message ```text highlight=2,5-7 theme={null} [2026-04-07 13:24:13 TP0] Decode batch, #running-req: 10, #token: 485248, token usage: 0.94, pre-allocated usage: 0.51, #prealloc-req: 1, #transfer-req: 12, #retracted-req: 0, npu graph: True, gen throughput (token/s): 259.82, #queue-req: 0 [2026-04-07 13:24:13 TP0] Context corruption detected: Request 3b5dcfe1575d4e1f9b18c953de878a93 (bootstrap_room=7451500070298748792) received metadata from bootstrap_room=4125156593077881415. Metadata buffer index: 1. This indicates metadata buffer index collision. [2026-04-07 13:24:13] INFO: 127.0.0.1:59272 - "POST /v1/chat/completions HTTP/1.1" 200 OK [2026-04-07 13:24:13] INFO: 127.0.0.1:34000 - "POST /v1/chat/completions HTTP/1.1" 200 OK terminate called after throwing an instance of 'gloo::EnforceNotMet' what(): [enforce fail at /pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:456] op.preamble.length <= op.nbytes. 4 vs 3 Fatal Python error: Aborted Thread 0x0000fff873f6f120 (most recent call first): File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 1499 in heartbeat_checker ``` ### Cause (Possibly, not precisely located) High-concurrency long sequences fill up the transfer buffer, causing buffer index collision and data corruption. ### Solution 1. Disable overlap on the Prefill node with `--disable-overlap-schedule`. The Prefill node in PD disaggregation must not enable overlap, otherwise it causes timing issues that lead to out-of-order reception on the Decode node. 2. Even with overlap disabled, multi-Prefill node high-concurrency long-sequence scenarios may still encounter this issue with low probability. This is a known issue pending resolution. ## 2. Graph mode aclnnInplaceFillScalar error ### Error message ```text highlight=2 theme={null} (SGLangEngine pid=3872176) [rank0]:[E414 12:14:41.204711510 compiler_depend.ts:444] operator():build/CMakeFiles/torch_npu.dir/compiler_depend.ts:26 NPU function error: call aclnnInplaceFillScalar failed, error code is 507000 (SGLangEngine pid=3872176) [ERROR] 2026-04-14-12:14:41 (PID:3874122, Device:0, RankID:-1) ERR00100 PTA call acl api failed (SGLangEngine pid=3872176) [Error]: An internal error occurs in the runtime module on the host. (SGLangEngine pid=3872176) Rectify the fault based on the error information in the ascend log. (SGLangEngine pid=3872176) [PID: 3874122] 2026-04-14-12:14:41.897.548 AclNN_Runtime_Error(EZ9903): aclrtLaunchKernelWithHostArgs failed, return: 507000 (SGLangEngine pid=3872176) Solution: In this scenario, collect the plog when the fault occurs and locate the fault based on the plog. (SGLangEngine pid=3872176) TraceBack (most recent call last): (SGLangEngine pid=3872176) Check kernel task failed, stream_id=2028, task_id=48, retCode=0x7080005.[FUNC:LaunchKernel][FILE:context.cc][LINE:1585] (SGLangEngine pid=3872176) rtsLaunchKernelWithHostArgs execution failed, reason=kernel type error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:61] (SGLangEngine pid=3872176) rtsLaunchKernelWithHostArgs failed, runtime result = 507000.[FUNC:ReportCallError][FILE:Log_inner.cpp][LINE:148] (SGLangEngine pid=3872176) aclrtLaunchKernelWithHostArgs failed, return: 507000 (SGLangEngine pid=3872176) Launch kernel failed. (SGLangEngine pid=3872176) #### KernelLaunch failed: /home/850b160/cann-8.5.8/opp/built-in/op_impl/ai_core/tbe//kernel/ascend910_93/ops_legacy/fill/Fill_41dadce325bOf810d03359af2a38990b_high_performance.o (SGLangEngine pid=3872176) Kernel Run failed. opType: 18, Fill (SGLangEngine pid=3872176) launch failed for Fill, errno:361001. (SGLangEngine pid=3872176) (SGLangEngine pid=3872176) Exception raised from operator() at build/CMakeFiles/torch_npu.dir/compiler depend.ts:26 (most recent call first): (SGLangEngine pid=3872176) frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >)+ 0xb0 (0xffff806848c0 in /root/anaconda3/envs/slime_re/Lib/python3.11/site-packages/torch/lib/libc10.so) (SGLangEngine pid=3872176) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0x68(0xffff8062c140 in /root/anaconda3/envs/slime_re/Tib/puthon3.11/site-packages/torch/lib/libc10.so) (SGLangEngine pid=3872176) frame #2: + 0x110e2b4 (0xffff6d44e2b4 in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so) (SGLangEngine pid=3872176) frame #3: + 0x29f0894(0xffff6ed30894 in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so) (SGLangEngine pid=3872176) frame #4: + 0x9cc708(0xffff6cd0c700 in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so) (SGLangEngine pid=3872176) frame #5: + 0x9cd2dc (0xffff6cd0d2dc in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so) ``` ### Cause Too many captured graphs cause a conflict in the graph mode update stream. Each graph is placed on a separate stream, but the number of streams is limited. If too many graphs are captured, conflicts occur. ### Solution * CANN 8.5 + TorchNPU 2.8 should have resolved this issue. * If your versions do not match, reduce the number of captured graphs to 10 or fewer. ## 3. alloc\_extend\_kernel error ### Error message ```text theme={null} [root@os-node-created-6z9tp pd_7p1d_tp2_20260414_030537]# grep -nR -E "aivec error|ACL" *,log prefill 1.log:5739:EZ9999[PID: 164243] 2026-04-14-06:40:35.033.745 (EZ9999): The error from device(chipId:0, dieId:1), serial number is 1, there is an exception of aivec error, core id is 23, error code = 0, dump info: pc start: 0x12420156a000, current: 0x12420156ad74, vec error info: 0xdb1751f50e, mte error info: 0x98f6388707, ifu error info: 0x212c93fc00000, ccu error info: 0x998e981c00000000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100dccc00.[FUNC:PrintCoreInfo][FILE:device error core proc.cc][LINE:347] prefill_1.log:5747:EZ9999[PID: 164242] 2026-04-14-06:40:35.034.456 (EZ9999): The error from device(chipId:0, dieId:0), serial number is 1, there is an exception of aivec error, core id is 45, error code = O, dump info: pc start: 0x12400156a000, current: 0x12400156ad74, vec error info: 0x7304583407, mte error info: 0x27e2b1c765, ifu error info: 0x212c93fc00000, ccu error info: 0x6e0836f300000000, cube error info: 0, biu error info: 0O, aic error mask: 0x6500020bd00028c, para base: 0x12c100dccc0O.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347] prefill_1.Log:5784:RuntimeError: ACL stream synchronize failed, error code:507035 prefill_1.Log:5816:RuntimeError: ACL stream synchronize failed, error code:507035 prefill 2.log:2566:EZ9999[PID: 163475] 2026-04-14-05:28:43.282.016 (EZ9999): The error from device(chipId:1, dieId:1), serial number is 1, there is an exception of aivec error, core id is 20, error code = 0, dump info: pc start: 0x124601550000, current: 0x124601550d74, vec error info: 0x9b07e496e3, mte error info: 0xc2a6806828, ifu error info: 0x212c93f200000, ccu error info: 0xaa64401100000000, cube error info: O, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100c72400.[FUNC:PrintCoreInfo][FILE:device_error_core proc.cc][LINE:347] prefill 2.log:2574:EZ9999[PID: 163474] 2026-04-14-05:28:43.282.809 (EZ9999): The error from device(chipId:1, dieId:0), serial number is 1, there is an exception of aivec error, core id is 22, error code = 0, dump info: pc start: 0x124401550000, current: 0x124401550d74, vec error info: 0xa80496049a, mte error info: 0xa0770e2bf2, ifu error info: 0x212c93f200000, ccu error info: 0x1083000000000000, cube error info: O, biu error info: 0O, aic error mask: 0x6500020bd00028c, para base: 0x12c100c72400.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347] prefill 2.log:2611:RuntimeError: ACL stream synchronize failed, error code:507035 prefill 2.log:2644:RuntimeError: ACL stream synchronize failed, error code:507035 prefill 4.log:1323:EZ9999[PID: 163478] 2026-04-14-05:08:15.363.509 (EZ9999): The error from device(chipId:3, dieId:0), serial number is 1, there is an exception of aivec error, core id is 5, error code = 0, dump info: pc start: 0x124c00dff000, current: 0x124c00dff860, vec error info: 0xf311fb8727, mte error info: 0x45418486a, ifu error info: 0x212c93fa00000, ccu error info: 0x17a9996f00000000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100d40c00.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347] prefill 4.log:1331:EZ9999[PID: 163479] 2026-04-14-05:08:15.364.016 (EZ9999): The error from device(chipId:3, dieId:1), serial number is 1, there is an exception of aivec error, core id is 40, error code = 0, dump info: pc start: 0x124e00dff000, current: 0x124e00dff860, vec error info: 0xf11a9cf188, mte error info: 0xfb7710514a, ifu error info: 0x212c93fa00000, ccu error info: 0x1f5828a900000000, cube error info: 0, biu error info: O, aic error mask: 0x6500020bd00028c, para base: 0x12c100d40c00.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347] prefill 4.Log:1368:RuntimeError: ACL stream synchronize failed, error code:507035 prefill 4.log:1400:RuntimeError: ACL stream synchronize failed, error code:507035 prefill 7.log:3017:EZ9999[PID: 164628] 2026-04-14-05:34:12.369.755 (EZ9999): The error from device(chipId:6, dieId:0), serial number is 1, there is an exception of aivec error, core id is 33, error code = 0, dump info: pc start: 0x1258015ae000, current: 0x1258015aed74, vec error info: 0x740053222c, mte error info: 0x97103df5a0, ifu error info: 0x212c93f400000, ccu error info: 0x391c89ab00000000, cube error info: 0, biu error info: 0O, aic error mask: 0x6500020bd00028c, para base: 0x12c10Ocd6400.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347] prefill 7.log:3025:EZ9999[PID: 164629] 2026-04-14-05:34:12.369.742 (EZ9999): The error from device(chipId:6, dieId:1), serial number is 1, there is an exception of aivec error, core id is 42, error code = 0, dump info: pc start: 0x125a015ae000, current: 0x125a015aed74, vec error info: 0xb91ae07b35, mte error info: 0xfc000670ef, ifu error info: 0x212c93f400000, ccu error info: 0x57c069b000000000, cube error info: 0, biu error info: O, aic error mask: 0x6500020bd00028c, para base: 0x12c10Ocd6400.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347] prefill 7.log:3062:RuntimeError: ACL stream synchronize failed, error code:507035 prefill 7.log:3094:RuntimeError: ACL stream synchronize failed, error code:507035 ``` ### Error plog ```text highlight=21 theme={null} [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.353.461 [stars engine.cc:1534]170327 ProcLogicCaReport:Task run failed, device id=13, stream id=43, task id=13145, sqe type=0(ffts), errType=0x1(task exception), sqSwStatus=0 [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.720 device error core proc.cc:3211170327 AddExceptionReqInfo:add error register: core id=42, stream id=43, task id=13145 [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.730 device error core proc.cc:3471170327 PrintCorelnfo:The error from device(chipld:6, dield:1), serial number is 1, there is an exception of aivec error, core id is 42, error code = O, dump info: pc start: 0x125a015ae000,current: 0x125a015aed74, vec error info: 0xb91ae07b35, mte error info: 0xfc000670ef, ifu error info: 0x212c93f400000, ccu error info: 0x57c069b000000000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100cd6400. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.774 device error core proc.cc:3601170327 PrintCorelnto:The extend into: errcode:(0, 0x8000, 0) errorStr: When the D-cache reads and writes data to the UB, the response value returned bv the bus is a non-zero value. fixp_error0 info: 0x670ef, fixp error1 info: 0xfc, fsmId:0, tslot:2, thread:0, ctxid:0, blk:2, sublk:0, subErrType:4. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.787 device error core proc.cc:4341170327 ProcessStarsCoreErrorInfo:devId=13, streamId=43, taskId=13145, MTE errorCode=0. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.795 davinci task.cc:2011170327 SetStarsResultForDavinciTask:AIV Kernel happen error, retCode=0x31. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.644 davinci kernel task.cc:15821170327 PreCheckTaskErr:Kernel task happen error retCode=0x31, vector core exception1. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.705 davinci kernel task.cc:14241170327 GetArasInfo:[AIC INFO] aras(0 to 9) after execute:0x3fffffb9000, 0, 0, 0x12c9323ff600, 0x12c93231ee00, 0x12c93f1d8800,0x12c93f3ff600,0x12c958200000,0x100000003, 0xaaaa00000001. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.710 [davinci kernel task.cc:1427]170327 GetArgsInfo:tilingKey = 0, print 1 Times totalLen=(10*8), argsSize=80, blockDim=3 [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.717 [davinci kernel task.cc:1468]170327 PrintErrorInfoForDavinciTask:[AIC INFO] after execute:arqs print end [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.751 davinci kernel task.cc:14981170327 PrintErrorInfoForDavinciTask:[DFX INFO]Aicore kernel execute failed, device id=13, stream id=43, report stream id=43, task id=13145, flip num=56, fault kernel_name=alloc_extend_kernel_18, fault kernel info ext=alloc_extend_kernel, program id=141, hash=14069671779787989248. [ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.823 [dump manager.cpp:41][tid:170327] An exception callback message is received. [ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.971 [kernel info collector.cpp:384][tid:170327] Get error register information. coreNum=0 [ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.981 kernel info collector.cpp:4771tid:1703271 It is Non-SuperKernel. functionCount=1, qlobalCount=1 [ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.987 [dump args.cpp:668][tid:170327] In arqAddr[0x12c100cd6400]|arqSize[80]dfxAddr[(nil)]|dfxSize[0] has invalid attribute. [ERROR] IDEDD(164629,):2026-04-14-05:34:12.383.894 [dump_printf.cpp:1118][tid:170327] infoAddr is null [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.908 [stream.cc:1332]170327 GetError:Stream Synchronize failed, stream id=43, retCode=0x31, [vector core exception]. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.911 [stream.cc:1335]170327 GetError:AIV Kernel happen error, retCode=0x31. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.929 [stream.cc:1335]170327 GetError:[AIC_INFO] after execute:args print end [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.936 stream.cc:13351170327 GetError: DFX INFO1Aicore kernel execute failed, device id=13, stream id=43, report stream id=43, task id=13145, tlip num=56, fault kernel_name=alloc_extend_kernel_18, fault kernel info ext=alloc_extend_kernel, program id=141, hash=14069671779787989248. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.943 [stream.cc:3549]170327 EnterFailureAbort:stream id=43 enter failure abort. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.973 [stars_engine.cc:1427]170327 StarsResumeRtsa:stop scheduling in abort failure mode: stream id=43, sq id=6,sq head=801, task id=13145, taskType=66. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.972 [stream.cc:1463]164629 SynchronizeExecutedTask:context is abort, status=0x715005e. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.978 [stream.cc:1516]164629 Synchronizelmpl:failed, stream_id=43, error=0x715005e [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.982 [api error.cc:1015]164629 StreamSynchronize:Stream synchronize failed, stream_id=43, timeout=-1ms. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.991 [apic stream.cc:154]164629 rtStreamSynchronize:ErrCode=507035, desc=[vector core exception], InnerCode=0x715005e [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.997 [error_message_manage.cc:61]164629 FuncErrorReason:rtStreamSynchronize execution failed, reason=vector core exception [ERROR] ASCENDCL(164629,):2026-04-14-05:34:12.384.269 [stream.cpp:140]164629 acIrtSynchronizeStreamImpl:synchronize stream failed, runtime result = 507035 ``` ### Cause The `alloc_extend_kernel` operator appears to have a memory allocation issue. Pending resolution. ### Solution Modify `sglang/srt/hardware_backend/npu/allocator_npu.py` to comment out the affected branch and use the else branch instead. ```python highlight=15-36 theme={null} def alloc_extend( self, prefix_lens: torch.Tensor, prefix_lens_cpu: torch.Tensor, seq_lens: torch.Tensor, seq_lens_cpu: torch.Tensor, last_loc: torch.Tensor, extend_num_tokens: int, num_new_pages: int = None, ): ... if num_new_pages_item > len(self.free_pages): return None # if num_new_pages_item < 200: # from sgl_kernel_npu.mem_cache.allocator import alloc_extend_kernel # out_indices = torch.empty( # (extend_num_tokens,), # dtype=torch.int64, # device=self.device, # ) # max_num_extend_tokens = next_power_of_2(extend_num_tokens) # bs = prefix_lens.shape[0] # alloc_extend_kernel[(bs,)]( # prefix_lens, # seq_lens, # last_loc, # self.free_pages, # out_indices, # next_power_of_2(bs), # self.page_size, # max_num_extend_tokens, # ) # else: out_indices = torch.empty( (extend_num_tokens,), dtype=torch.int32, device=self.device, ) ... ``` ## 4. Out of NPU memory ```text highlight=7 theme={null} File "/home/code/sglang/python/sglang/srt/model_executor/pool_configurator.py", line 175, in calculate_pool_sizes return MemoryPoolConfig(max_total_num_tokens=max_total_num_tokens) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 8, in __init__ File "/home/code/sglang/python/sglang/srt/model_executor/pool_configurator.py", line 44, in __post_init__ raise RuntimeError(msg) RuntimeError: Not enough memory. Please try to increase --mem-fraction-static. ``` ### Solution First, use the `npu-smi info` command to check the NPU memory usage. If the NPUs are occupied by other processes, use `--base-gpu-id` to specify the starting device index. If the NPUs are not occupied, you can use `--tp` to deploy across multiple devices, or reduce the KV cache memory usage by increasing the `--mem-fraction-static` value. For detailed tuning guidance, see [Hyperparameter Tuning](/docs/advanced_features/hyperparameter_tuning). ## 5. How to update sgl-kernel-npu ### Solution ```bash theme={null} git clone https://github.com/sgl-project/sgl-kernel-npu.git source /usr/local/Ascend/ascend-toolkit/set_env.sh cd sgl-kernel-npu # Building Project bash build.sh pip install output/sgl_kernel_npu*.whl --force-reinstall # (Optional) Confirm whether the import can be successfully python -c "import sgl_kernel_npu; print(sgl_kernel_npu.__path__)" rm -rf sgl-kernel-npu ``` ## 6. `[Errno 101] Network is unreachable` when downloading HuggingFace datasets ### Error message ```text highlight=1-2 theme={null} '[Errno 101] Network is unreachable' thrown while requesting HEAD https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json Retrying in 1s [Retry 1/5]. Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 2353, in run_benchmark(args) File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 1848, in run_benchmark input_requests = get_dataset(args, tokenizer, model_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/__init__.py", line 44, in get_dataset return dataset.load(tokenizer=tokenizer, model_id=model_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 45, in load return sample_random_requests( ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 89, in sample_random_requests dataset_path = download_and_cache_hf_file( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/utils.py", line 98, in download_and_cache_hf_file return hf_hub_download(repo_id=repo_id, filename=filename, repo_type=repo_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ... RuntimeError: Cannot send a request, as the client has been closed. [ERROR] 2026-05-18-11:58:31 (PID:215, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception ``` ### Cause The machine cannot directly access the HuggingFace server due to network restrictions (firewall, proxy, or regional access limitations). ### Solution * **Use an HF mirror site** — set the `HF_ENDPOINT` environment variable to a mirror (e.g., hf-mirror.com): ```bash theme={null} export HF_ENDPOINT=https://hf-mirror.com ``` * **Use a proxy** — if you have an HTTP proxy available: ```bash theme={null} export http_proxy=http://your-proxy:port export https_proxy=http://your-proxy:port ``` * **Download the dataset manually** — use a machine with network access to download the file, then transfer it to the target machine. Use `--dataset-path` to specify the local file path: ```bash theme={null} python -m sglang.bench_serving \ --dataset-name random \ --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \ ... ``` ## 7. Unexpected type fp8 ### Cause FP8 model is not supported ### Solution Replace model weights, e.g., switch Qwen/Qwen3.5-27B-FP8 to Eco-Tech/Qwen3.5-27B-w8a8-mtp. ## 8. Docker image versions: stable release vs. daily build Docker images for Ascend NPU are available in two types: * **Stable release** — validated version with a specific tag, e.g., `quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16`. Recommended for production deployments. * **Daily build** — includes the latest development changes, e.g., `quay.io/ascend/sglang:main-cann8.5.0-a3`. Use this if you need the latest features or bug fixes that have not yet been included in a stable release. If you encounter issues with a stable release, try switching to a daily build to see if the issue has been resolved in the latest development version. # Installation Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/getting-started/installation Complete installation guide for SGLang on Ascend NPUs, including component version mapping, environment setup, and launching inference services. You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are operating at optimal performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems. ## Component Version Mapping For SGLang Each CANN version requires its own matched set of component versions — do not mix versions across sets.
Component Version Obtain Method
HDK 25.5.2 link
CANN 9.0.0 Obtain Images
TorchNPU 26.0.0 link
MemFabric 1.1.4 `pip install memfabric-hybrid==1.1.4`
Triton 3.2.1.dev20260530 `pip install triton-ascend==3.2.1.dev20260530 \`
`--extra-index-url=https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly \`
`--trusted-host mirrors.huaweicloud.com`
SGLang NPU Kernel 2026.9.0 link
MemFabric-zbal 1.1.2 `pip install memfabric-zbal==1.1.2`
Component Version Obtain Method
HDK 25.5.2 link
CANN 9.1.0 Obtain Images
TorchNPU 26.1.0 link
MemFabric 1.1.4 `pip install memfabric-hybrid==1.1.4`
Triton 3.2.2 `pip install https://github.com/triton-lang/triton-ascend/releases/download/v3.2.2/triton_ascend-3.2.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl`
`(aarch64 host: replace x86_64 with aarch64)`
SGLang NPU Kernel 20260821 link
MemFabric-zbal 1.1.3 `pip install memfabric-zbal==1.1.3`
SGLang Docker images for the CANN 9.1.0 set have not been published yet. If you deploy with the prebuilt SGLang images ([Method 2](#method-2-using-docker-image)), use the CANN 9.0.0 set for now. To use the CANN 9.1.0 set, install SGLang from source ([Method 1](#method-1-installing-from-source-with-prerequisites), with the component versions in this tab) or start from a CANN 9.1.0 base image (see [Obtain CANN Image](#obtain-cann-image)). ### Obtain CANN Image Ensure sufficient disk space before pulling images. Each Docker image requires at least **30GB** of free space. You can obtain the dependency of a specified version of CANN through an image. ```bash Command theme={null} # CANN 9.0.0 docker pull quay.io/ascend/cann:9.0.0-a3-ubuntu22.04-py3.11 # CANN 9.1.0 docker pull quay.io/ascend/cann:9.1.0-a3-ubuntu22.04-py3.12 ``` ```bash Command theme={null} # CANN 9.0.0 docker pull quay.io/ascend/cann:9.0.0-910b-ubuntu22.04-py3.11 # CANN 9.1.0 docker pull quay.io/ascend/cann:9.1.0-910b-ubuntu22.04-py3.12 ``` ## Preparing the Running Environment ### Method 1: Installing from source with prerequisites The steps in this method install the **CANN 9.0.0** set. To install the CANN 9.1.0 set from source, follow the same steps with the component versions from the [version mapping table](#component-version-mapping-for-sglang) (python 3.12 is required for the CANN 9.1.0 set). #### Python Version **The CANN 9.0.0 set requires `python==3.11`, and the CANN 9.1.0 set uses `python==3.12`.** If you don't want to break system pre-installed python, try installing with [conda](https://github.com/conda/conda). ```bash Command theme={null} conda create --name sglang_npu python=3.11 conda activate sglang_npu ``` **Note on Anaconda repository restrictions** If you encounter an error like “Terms of Service have not been accepted” during the conda create step, the default Anaconda repository is blocking package downloads. To resolve this, configure a mirror (e.g., Tsinghua Open Source Mirror): ```bash Command theme={null} # Add Tsinghua mirrors conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/ conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/ conda config --set show_channel_urls yes conda config --remove channels defaults ``` Edit the system-level conda config to remove any hardcoded defaults, e.g., `vi ~/miniconda3/.condarc`. Then remove the failed environment and recreate it: ```bash Command theme={null} conda clean -i conda env remove -n sglang_npu conda create --name sglang_npu python=3.11 conda activate sglang_npu ``` #### CANN Prior to starting work with SGLang on Ascend, you need to install CANN Toolkit, Kernels operator package and NNAL version 9.0.0, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/900/softwareinst/instg/instg_0008.html?OS=openEuler\&InstallType=local) #### MemFabric-Hybrid If you want to use PD disaggregation mode, you need to install MemFabric-Hybrid. MemFabric-Hybrid is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters. ```bash Command theme={null} pip install memfabric-hybrid==1.0.8 ``` #### MemFabric-zbal MemFabric-zbal is a Zero Buffer Acceleration Library of high-performance operators for LLM inference and training on Ascend, accelerating computation by eliminating intermediate memory buffers; it is required only on **aarch64** clusters and is installed in addition to MemFabric-Hybrid. ```bash Command theme={null} # Only needed on aarch64 (arm64) hosts pip install memfabric-zbal==1.1.1 ``` #### PyTorch and PyTorch Framework Adapter on Ascend ```bash Command theme={null} PYTORCH_VERSION=2.10.0 TORCHVISION_VERSION=0.25.0 TORCH_NPU_VERSION=2.10.0 pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu pip install torch_npu==$TORCH_NPU_VERSION ``` If you are using other versions of `torch` and install `torch_npu`, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README.md) #### Triton on Ascend We provide our own implementation of Triton for Ascend. ```bash Command theme={null} pip install triton-ascend==3.2.1.dev20260530 \ --extra-index-url=https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly \ --trusted-host mirrors.huaweicloud.com ``` For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://github.com/triton-lang/triton-ascend/blob/main/docs/en/installation_guide.md) #### SGLang Kernels NPU We provide SGL kernels for Ascend NPU, check [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md). #### DeepEP-compatible Library We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README.md). #### Some other dependencies ```bash Command theme={null} # libGL apt update apt install libgl1 libglib2.0-0 # ensure setuptools contains pkg_resources module pip install "setuptools<80" ``` #### Installing SGLang from source ```bash Command theme={null} # Use the last release branch git clone https://github.com/sgl-project/sglang.git cd sglang mv python/pyproject_npu.toml python/pyproject.toml pip install -e python[all_npu] ``` ### Method 2: Using Docker Image #### Obtain Image You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image. Ensure sufficient disk space before pulling images. Each Docker image requires at least **30GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. 1. Download SGLang image We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `cann9.0.0-a3-v0.5.16`) if you prefer a validated version, or a daily build tag (e.g., `main-cann9.0.0-a3`) if you need the latest development changes. SGLang images for the CANN 9.1.0 set have not been published yet. For now, please use the CANN 9.0.0 images below. ```bash Command theme={null} # Stable release docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 # Daily build docker pull quay.io/ascend/sglang:main-cann9.0.0-a3 ``` ```bash Command theme={null} # Stable release docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 # Daily build docker pull quay.io/ascend/sglang:main-cann9.0.0-910b ``` 2. Build an image based on Dockerfile ```bash Command theme={null} # Clone the SGLang repository git clone https://github.com/sgl-project/sglang.git cd sglang/docker # Build the docker image # Replace with the target architecture, e.g., amd64, arm64. # Optional build arguments: # --build-arg DEVICE_TYPE=910b # Required for Ascend A2 Series # --build-arg APTMIRROR= # Use a custom APT mirror to improve download speed # If there are network errors, please modify the Dockerfile to add ARG HTTP_PROXY/HTTPS_PROXY and set them as ENV. docker build --build-arg TARGETARCH= -t -f npu.Dockerfile . ``` #### Create Docker **Notice:** `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters. ```bash Command theme={null} # Create a shortcut 'drun' to launch a privileged Docker container alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \ --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \ --device=/dev/davinci_manager --device=/dev/hisi_hdc \ --volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --volume /etc/ascend_install.info:/etc/ascend_install.info \ --volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/' # Add HF_TOKEN env for downloading models by SGLang. # The container runs with the '--rm' flag, so it will be automatically removed after the command finishes (including Ctrl+C) drun --env "HF_TOKEN=" \ \ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend ``` ```bash Command theme={null} # Create a shortcut 'drun' to launch a privileged Docker container alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device=/dev/davinci_manager --device=/dev/hisi_hdc \ --volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --volume /etc/ascend_install.info:/etc/ascend_install.info \ --volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/' # Add HF_TOKEN env for downloading models by SGLang. # The container runs with the '--rm' flag, so it will be automatically removed after the command finishes (including Ctrl+C) drun --env "HF_TOKEN=" \ \ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend ``` SGLang will serve on `http://127.0.0.1:30000` by default. You can change the host and port by `--host` and `--port` parameters. ## System Settings ### CPU performance power scheme The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended. ```bash Command theme={null} echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor # Make sure changes are applied successfully cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance ``` ### Disable NUMA balancing ```bash Command theme={null} sudo sysctl -w kernel.numa_balancing=0 # Check cat /proc/sys/kernel/numa_balancing # shows 0 ``` ### Prevent swapping out system memory ```bash Command theme={null} sudo sysctl -w vm.swappiness=10 # Check cat /proc/sys/vm/swappiness # shows 10 ``` ## Running SGLang Service ### Running Service For Large Language Models #### PD Mixed Scene ```bash Command theme={null} # Enabling CPU Affinity export SGLANG_SET_CPU_AFFINITY=1 python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --attention-backend ascend \ --host 127.0.0.1 \ --port 30000 ``` #### PD Disaggregation Scene 1. Launch Prefill Server ```bash Command theme={null} # Enabling CPU Affinity export SGLANG_SET_CPU_AFFINITY=1 # PREFILL_IP: IP address of the first Prefill Server # FREE_PORT: any available port # all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT" python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode prefill \ --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8995 \ --attention-backend ascend \ --device npu \ --base-gpu-id 0 \ --tp-size 1 \ --host 127.0.0.1 \ --port 30001 ``` ```bash Command theme={null} # Enabling CPU Affinity export SGLANG_SET_CPU_AFFINITY=1 # PREFILL_IP: IP address of the first Prefill Server # FREE_PORT: any available port # all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT" export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma" python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode prefill \ --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8995 \ --attention-backend ascend \ --device npu \ --base-gpu-id 0 \ --tp-size 1 \ --host 127.0.0.1 \ --port 30001 ``` 2. Launch Decode Server ```bash Command theme={null} # PREFILL_IP: IP address of the first Prefill Server # FREE_PORT: any available port # all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT" python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode decode \ --disaggregation-transfer-backend ascend \ --attention-backend ascend \ --device npu \ --base-gpu-id 1 \ --tp-size 1 \ --host 127.0.0.1 \ --port 30002 ``` ```bash Command theme={null} # PREFILL_IP: IP address of the first Prefill Server # FREE_PORT: any available port # all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT" export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma" python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode decode \ --disaggregation-transfer-backend ascend \ --attention-backend ascend \ --device npu \ --base-gpu-id 1 \ --tp-size 1 \ --host 127.0.0.1 \ --port 30002 ``` 3. Launch Router ```bash Command theme={null} python3 -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://127.0.0.1:30001 8995 \ --decode http://127.0.0.1:30002 \ --host 127.0.0.1 \ --port 30000 ``` The `8995` in command script is the disaggregation bootstrap port. It must match the `--disaggregation-bootstrap-port` value set on the prefill server in step 1. ### Running Service For Multimodal Language Models #### PD Mixed Scene ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-30B-A3B-Instruct \ --host 127.0.0.1 \ --port 30000 \ --tp 4 \ --device npu \ --attention-backend ascend \ --mm-attention-backend ascend_attn \ --disable-radix-cache \ --trust-remote-code \ --enable-multimodal \ --sampling-backend ascend ``` ## Testing the Service Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. ### Which port to send requests to The port you use depends on your deployment mode: | Scenario | Where to send requests | | - | - | | Non-PD (single server) | The server's `--port` (e.g., `30000` in the examples above) | | Non-PD (multi-node) | The primary node's (`--node-rank 0`) `--port`; do **not** send requests to worker nodes | | PD disaggregation | The router's `--port` (e.g., `30000` in the examples above); do **not** send requests directly to prefill or decode servers | SGLang defaults to port `30000` when `--port` is not specified. The examples in this guide use explicit ports for clarity. ### Health Check ```bash Command theme={null} curl http://127.0.0.1:30000/health ``` A successful response returns HTTP 200 with an empty body. ### Generate (Native Endpoint) ```bash Command theme={null} curl http://127.0.0.1:30000/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": {"temperature": 0, "max_new_tokens": 128} }' ``` The expected output should contain "Paris". ### Chat Completions (OpenAI-Compatible) ```bash Command theme={null} curl http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "What is the capital of France?"}] }' ``` Some models return responses accompanied by thinking process content. To disable this output, configure parameters as follows: ```bash Command theme={null} curl http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Eco-Tech/Qwen3.5-27B-w8a8-mtp", "messages": [{"role": "user", "content": "What is the capital of France?"}], "chat_template_kwargs": {"enable_thinking": false} }' ``` The expected output should contain "Paris". ### Multimodal Chat Completions The image URL in the example below references an external resource (`raw.githubusercontent.com`). Make sure the server has internet access so the image can be downloaded at inference time. Alternatively, you can use a locally accessible URL or base64-encoded image data. ```bash Command theme={null} curl http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3-VL-30B-A3B-Instruct", "messages": [{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image."} ] }] }' ``` # Quickstart Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/getting-started/quick_start This page covers only the simplest deployment flow using the official container image. For the complete installation guide across all scenarios (source install, Docker build, system settings, version mapping, etc.), see [SGLang installation with NPUs support](/docs/hardware-platforms/ascend-npus/getting-started/installation). ## Prerequisites ### Supported Devices * Ascend A2 Series Products * Ascend A3 Series Products To identify your device, run `npu-smi info -l`: A3 Series reports `Chip Count: 2` per NPU, while A2 Series reports `Chip Count: 1` per NPU. For hardware details, see the [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ### Docker Ensure Docker is installed and the Docker daemon is running on your host machine. Verify with: ```bash theme={null} docker --version && docker info ``` If Docker is not installed, follow the [official Docker installation guide](https://docs.docker.com/engine/install/) for your operating system. ## Setup environment using container Ensure sufficient disk space before proceeding. Run `df -h` to check the available disk space. The Docker image requires at least **30GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `cann9.0.0-a3-v0.5.16`) if you prefer a validated version, or a daily build tag (e.g., `main-cann9.0.0-a3`) if you need the latest development changes. If you have already downloaded model weights to a local path (e.g., `/path/to/model`), mount the path into the container by adding `--volume /path/to/model:/path/to/model` to the `docker run` command below. ```shell Command theme={null} # Choose one (uncomment the line you want): export IMAGE=quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 # Stable release # export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-a3 # Daily build docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \ --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \ --device=/dev/davinci_manager \ --device=/dev/hisi_hdc \ --volume /usr/local/sbin:/usr/local/sbin \ --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --volume /etc/ascend_install.info:/etc/ascend_install.info \ --volume /var/queue_schedule:/var/queue_schedule \ --volume ~/.cache/:/root/.cache/ \ --entrypoint=bash \ $IMAGE ``` ```shell Command theme={null} # Choose one (uncomment the line you want): export IMAGE=quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 # Stable release # export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-910b # Daily build docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device=/dev/davinci_manager \ --device=/dev/hisi_hdc \ --volume /usr/local/sbin:/usr/local/sbin \ --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --volume /etc/ascend_install.info:/etc/ascend_install.info \ --volume /var/queue_schedule:/var/queue_schedule \ --volume ~/.cache/:/root/.cache/ \ --entrypoint=bash \ $IMAGE ``` ## Usage The SGLang server is installed in the container by default. You can use `pip show sglang` to check the version. ### Start SGLang server SGLang will automatically download the model from Hugging Face. If the model is already downloaded to a local path (and has been mounted into the container), use that path directly like `--model-path /path/to/model`. ```shell Command theme={null} # Set HF_ENDPOINT to a mirror site if network is not available export HF_ENDPOINT=https://hf-mirror.com # Set your own HF_TOKEN to download restricted models export HF_TOKEN= # Start SGLang server # It may take several minutes to download the model on the first run sglang serve --model-path Qwen/Qwen2.5-7B-Instruct --attention-backend ascend & ``` Server startup may take several minutes. Once you see output like the following, the server is running. ```log Output theme={null} INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:30000 (Press CTRL+C to quit) The server is fired up and ready to roll! ``` ### Send a test request You can do inference using the server: ```shell Command theme={null} curl -X POST http://localhost:30000/generate \ -H "Content-Type: application/json" \ -d '{ "text": "The capital of France is", "sampling_params": { "temperature": 0, "max_new_tokens": 16 } }' ``` If the "text" field in the response contains "Paris", the server is working as expected. ### Stop server and exit container The SGLang server is running as a background process. You can send a `SIGINT` signal to stop it. ```shell Command theme={null} SGLANG_PID=$(pgrep -f "sglang serve") kill -SIGINT $SGLANG_PID ``` Wait a moment for the server to shut down gracefully. The output should be like the following: ```log Output theme={null} INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [] ``` The server has now stopped. You can verify it with `ps -ef | grep sglang` — the expected output is nothing (no matching process), then exit the container by pressing `Ctrl+D`. # Mindspore backend Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/mindspore_backend ## Introduction MindSpore is a high-performance AI framework optimized for Ascend NPUs. This doc guides users to run MindSpore models in SGLang. ## Requirements MindSpore currently only supports Ascend NPU devices. Users need to first install Ascend CANN software packages. The CANN software packages can be downloaded from the [Ascend Official Website](https://www.hiascend.com). The recommended version is 8.3.RC2. ## Supported Models Currently, the following models are supported: * **Qwen3**: Dense and MoE models * **DeepSeek V3/R1** * *More models coming soon...* ## Installation Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](./getting-started/installation) and then install `sgl-mindspore`: ```shell Install theme={null} git clone https://github.com/mindspore-lab/sgl-mindspore.git cd sgl-mindspore pip install -e . ``` ## Run Model Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example. ### Offline Inference Use the following script for offline inference: ```python Offline Inference theme={null} import sglang as sgl # Initialize the engine with MindSpore backend llm = sgl.Engine( model_path="/path/to/your/model", # Local model path device="npu", # Use NPU device model_impl="mindspore", # MindSpore implementation attention_backend="ascend", # Attention backend tp_size=1, # Tensor parallelism size dp_size=1 # Data parallelism size ) # Generate text prompts = [ "Hello, my name is", "The capital of France is", "The future of AI is" ] sampling_params = {"temperature": 0, "top_p": 0.9} outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print(f"Prompt: {prompt}") print(f"Generated: {output['text']}") print("---") ``` ### Start server Launch a server with MindSpore backend: ```bash Launch Server theme={null} # Basic server startup python3 -m sglang.launch_server \ --model-path /path/to/your/model \ --host 0.0.0.0 \ --device npu \ --model-impl mindspore \ --attention-backend ascend \ --tp-size 1 \ --dp-size 1 ``` For distributed server with multiple nodes: ```bash Multi-node Distributed theme={null} # Multi-node distributed server python3 -m sglang.launch_server \ --model-path /path/to/your/model \ --host 0.0.0.0 \ --device npu \ --model-impl mindspore \ --attention-backend ascend \ --dist-init-addr 127.0.0.1:29500 \ --nnodes 2 \ --node-rank 0 \ --tp-size 4 \ --dp-size 2 ``` ## Troubleshooting ### Debug Mode Enable sglang debug logging by log-level argument. ```bash Debug Mode theme={null} python3 -m sglang.launch_server \ --model-path /path/to/your/model \ --host 0.0.0.0 \ --device npu \ --model-impl mindspore \ --attention-backend ascend \ --log-level DEBUG ``` Enable mindspore info and debug logging by setting environments. ```bash Set Log Level theme={null} export GLOG_v=1 # INFO export GLOG_v=0 # DEBUG ``` ### Explicitly select devices Use the following environment variable to explicitly select the devices to use. ```shell Select Devices theme={null} export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7 # to set device ``` ### Some communication environment issues In case of some environment with special communication environment, users need set some environment variables. ```shell Disable LCCL theme={null} export MS_ENABLE_LCCL=off # current not support LCCL communication mode in SGLang-MindSpore ``` ### Some dependencies of protobuf In case of some environment with special protobuf version, users need set some environment variables to avoid binary version mismatch. ```shell Fix Protobuf theme={null} export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python # to avoid protobuf binary version mismatch ``` ## Support For MindSpore-specific issues: * Refer to the [MindSpore documentation](https://www.mindspore.cn/) # DeepSeek-R1 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1 This page focuses on optimal configuration and benchmark results for DeepSeek-R1 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [DeepSeek-R1 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-20ms) | | DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 3.5k+1k | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k-19-0ms) | | DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 3.9k+1k | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k9-out1k-19-0ms) | | DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 6k+1.6k | 20.5ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in6k-out1k6-20-5ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | DeepSeek-R1 | Ascend A3 Series Products | 16 | PD Disaggregation | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-1p1d-16p-in3k5-out1k5-50ms) | | DeepSeek-R1 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-8p-in3k5-out1k5-50ms) | | DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-50ms) | ## Optimal Configuration ### DeepSeek-R1 W4A8 1P1D 16P IN3K5 OUT1K5 50ms **Model**: DeepSeek-R1 **Hardware**: Ascend A3 Series Products **Cards**: 16 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ENABLE_MOE_NZ=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_NPU_USE_MLAPO=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=4000 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --nnodes 1 \ --tp-size 16 \ --mem-fraction-static 0.8 \ --quantization modelslim \ --max-running-requests 32 \ --context-length 8192 \ --disable-radix-cache \ --chunked-prefill-size 20480 \ --max-prefill-tokens 20480 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --attn-dp-size 8 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --max-total-tokens 80000 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=800 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --nnodes 1 \ --tp-size 16 \ --attn-dp-size 16 \ --mem-fraction-static 0.9 \ --max-running-requests 416 \ --quantization modelslim \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --enable-dp-lm-head \ --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --tokenizer-worker-num 4 \ --load-balance-method round_robin \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --max-total-tokens 145792 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --request-rate 24 \ --max-concurrency 416 \ --num-prompts 1664 \ --random-input-len 3584 \ --random-output-len 1536 \ --random-range-ratio 1 \ --seed 1 ``` ### DeepSeek-R1 W4A8 8P IN3K5 OUT1K5 50ms **Model**: DeepSeek-R1 **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEPEP_HCCL_BUFFSIZE=2000 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_LONG_SEQ_ROUND=10 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=72 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 16 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --watchdog-timeout 9000 \ --cuda-graph-bs-decode 4 8 12 14 \ --mem-fraction-static 0.9 \ --max-running-requests 224 \ --context-length 8188 \ --disable-radix-cache \ --chunked-prefill-size 65536 \ --max-prefill-tokens 3000 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --attn-dp-size 16 \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --dtype bfloat16 \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --max-total-tokens 100000 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 224 \ --num-prompts 896 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` ### DeepSeek-R1 W8A8 2P1D 32P IN3K5 OUT1K5 20ms **Model**: DeepSeek-R1 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_NPU_USE_MLAPO=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1536 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port $((8998 + $i)) \ --node-rank 0 \ --nnodes 1 \ --tp-size 16 \ --mem-fraction-static 0.81 \ --quantization modelslim \ --max-running-requests 4 \ --context-length 8192 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 28680 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --attn-dp-size 2 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=650 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp-size 32 \ --attn-dp-size 16 \ --mem-fraction-static 0.75 \ --max-running-requests 32 \ --quantization modelslim \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --cuda-graph-bs-decode 2 4 6 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tokenizer-worker-num 4 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ export SGLANG_DP_ROUND_ROBIN=1 python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --request-rate 16 \ --max-concurrency 32 \ --num-prompts 32 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 42 ``` ### DeepSeek-R1 W8A8 2P1D 32P IN3K5 OUT1K5 50ms **Model**: DeepSeek-R1 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=800 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=131072 export SGLANG_NPU_FUSED_MOE_MODE=2 export SGLANG_USE_AG_AFTER_QLORA=1 export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port $((8998 + $i)) \ --node-rank 0 \ --nnodes 1 \ --tp-size 16 \ --mem-fraction-static 0.778 \ --quantization modelslim \ --max-running-requests 16 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 60000 \ --moe-a2a-backend ascend_fuseep \ --deepep-mode normal \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --attn-dp-size 4 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=600 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_LM_HEAD_TP=8 export SGLANG_NPU_FUSED_MOE_MODE=1 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp-size 32 \ --attn-dp-size 32 \ --mem-fraction-static 0.82 \ --max-running-requests 1024 \ --quantization modelslim \ --moe-a2a-backend ascend_fuseep \ --deepep-mode low_latency \ --moe-dense-tp 1 \ --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --tokenizer-worker-num 4 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ export SGLANG_DP_ROUND_ROBIN=1 python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --request-rate 40 \ --max-concurrency 1024 \ --num-prompts 7168 \ --random-input-len 3584 \ --random-output-len 1536 \ --random-range-ratio 1 \ --seed 1 ``` ### DeepSeek-R1 W8A8 2P1D 32P IN3K5 OUT1K 19.0ms **Model**: DeepSeek-R1 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1k **TPOT**: 19.0ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_NPU_USE_MLAPO=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1536 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port $((8998 + $i)) \ --node-rank 0 \ --nnodes 1 \ --tp-size 16 \ --mem-fraction-static 0.81 \ --quantization modelslim \ --max-running-requests 4 \ --context-length 8192 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 28680 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --attn-dp-size 2 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=650 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp-size 32 \ --attn-dp-size 16 \ --mem-fraction-static 0.75 \ --max-running-requests 32 \ --quantization modelslim \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --cuda-graph-bs-decode 2 4 6 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tokenizer-worker-num 4 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ export SGLANG_DP_ROUND_ROBIN=1 python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --request-rate 16 \ --max-concurrency 32 \ --num-prompts 32 \ --random-input-len 3500 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 ``` ### DeepSeek-R1 W8A8 2P1D 32P IN3K9 OUT1K 19.0ms **Model**: DeepSeek-R1 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 3.9k+1k **TPOT**: 19.0ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_NPU_USE_MLAPO=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1536 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port $((8998 + $i)) \ --node-rank 0 \ --nnodes 1 \ --tp-size 16 \ --mem-fraction-static 0.81 \ --quantization modelslim \ --max-running-requests 4 \ --context-length 8192 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 28680 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --attn-dp-size 2 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=650 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp-size 32 \ --attn-dp-size 16 \ --mem-fraction-static 0.75 \ --max-running-requests 32 \ --quantization modelslim \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --cuda-graph-bs-decode 2 4 6 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tokenizer-worker-num 4 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ export SGLANG_DP_ROUND_ROBIN=1 python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --request-rate 16 \ --max-concurrency 32 \ --num-prompts 32 \ --random-input-len 3900 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 ``` ### DeepSeek-R1 W8A8 2P1D 32P IN6K OUT1K6 20.5ms **Model**: DeepSeek-R1 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 6k+1.6k **TPOT**: 20.5ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_NPU_USE_MLAPO=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1536 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port $((8998 + $i)) \ --node-rank 0 \ --nnodes 1 \ --tp-size 16 \ --mem-fraction-static 0.81 \ --quantization modelslim \ --max-running-requests 4 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 28680 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --attn-dp-size 2 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=650 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp-size 32 \ --attn-dp-size 8 \ --mem-fraction-static 0.75 \ --max-running-requests 32 \ --quantization modelslim \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --cuda-graph-bs-decode 2 4 6 \ --watchdog-timeout 9000 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tokenizer-worker-num 4 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ export SGLANG_DP_ROUND_ROBIN=1 python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --request-rate 16 \ --max-concurrency 32 \ --num-prompts 32 \ --random-input-len 6000 \ --random-output-len 1600 \ --random-range-ratio 1 \ --seed 1 ``` # DeepSeek-V3.2 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2 This page focuses on optimal configuration and benchmark results for DeepSeek-V3.2 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [DeepSeek-V3.2 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | DeepSeek-V3.2 | Ascend A3 Series Products | 32 | PD Disaggregation | 128k+1k | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-26ms) | | DeepSeek-V3.2 | Ascend A3 Series Products | 32 | PD Disaggregation | 128k+1k | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs8) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | DeepSeek-V3.2 | Ascend A3 Series Products | 32 | PD Disaggregation | 128k+1k | 107ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs16) | ## Optimal Configuration ### DeepSeek-V3.2 W8A8 1P1D 32P IN128K OUT1K 26ms **Model**: DeepSeek-V3.2 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 128k+1k **TPOT**: 26ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --nnodes 2 \ --tp 32 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.73 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 68000 \ --max-running-requests 1 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --disable-cuda-graph \ --moe-dense-tp-size 1 \ --enable-prefill-cp \ --cp-strategy zigzag \ --attn-cp-size 32 \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --reasoning-parser deepseek-v3 \ --tool-call-parser deepseekv32 \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=400 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp 32 \ --attn-dp-size 8 \ --ep 32 \ --moe-dense-tp-size 1 \ --enable-dp-lm-head \ --watchdog-timeout 9000 \ --mem-fraction-static 0.79 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 68000 \ --max-running-requests 32 \ --cuda-graph-max-bs-decode 4 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --quantization modelslim \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --disaggregation-transfer-backend ascend \ --reasoning-parser deepseek-v3 \ --tool-call-parser deepseekv32 \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 8 \ --num-prompts 8 \ --random-input-len 131072 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 ``` ### DeepSeek-V3.2 W8A8 1P1D 32P IN128K OUT1K BS16 **Model**: DeepSeek-V3.2 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 128k+1k **TPOT**: 107ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --nnodes 2 \ --tp 32 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.73 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 68000 \ --max-running-requests 1 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --disable-cuda-graph \ --moe-dense-tp-size 1 \ --enable-prefill-cp \ --cp-strategy zigzag \ --attn-cp-size 32 \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --reasoning-parser deepseek-v3 \ --tool-call-parser deepseekv32 \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=400 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp 32 \ --attn-dp-size 8 \ --ep 32 \ --moe-dense-tp-size 1 \ --enable-dp-lm-head \ --watchdog-timeout 9000 \ --mem-fraction-static 0.79 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 68000 \ --max-running-requests 32 \ --cuda-graph-max-bs-decode 4 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --quantization modelslim \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --disaggregation-transfer-backend ascend \ --reasoning-parser deepseek-v3 \ --tool-call-parser deepseekv32 \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 16 \ --num-prompts 16 \ --random-input-len 131072 \ --random-output-len 1024 \ --random-range-ratio 1 ``` ### DeepSeek-V3.2 W8A8 1P1D 32P IN128K OUT1K BS8 **Model**: DeepSeek-V3.2 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 128k+1k **TPOT**: 26ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --nnodes 2 \ --tp 32 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.73 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 68000 \ --max-running-requests 1 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --disable-cuda-graph \ --moe-dense-tp-size 1 \ --enable-prefill-cp \ --cp-strategy zigzag \ --attn-cp-size 32 \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --reasoning-parser deepseek-v3 \ --tool-call-parser deepseekv32 \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=400 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp 32 \ --attn-dp-size 8 \ --ep 32 \ --moe-dense-tp-size 1 \ --enable-dp-lm-head \ --watchdog-timeout 9000 \ --mem-fraction-static 0.79 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 68000 \ --max-running-requests 32 \ --cuda-graph-max-bs-decode 4 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --quantization modelslim \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --disaggregation-transfer-backend ascend \ --reasoning-parser deepseek-v3 \ --tool-call-parser deepseekv32 \ --trust-remote-code \ --attention-backend ascend \ --device npu break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 8 \ --num-prompts 8 \ --random-input-len 131072 \ --random-output-len 1024 \ --random-range-ratio 1 ``` # DeepSeek-V4-Flash Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash This page focuses on optimal configuration and benchmark results for DeepSeek-V4-Flash on the Ascend NPU. On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | DeepSeek-V4-Flash | Ascend A3 Series Products | 16 | PD Disaggregation | 8k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v4-flash-w8a8-1p1d-16p-in8k-out1k-50ms) | | DeepSeek-V4-Flash | Ascend A3 Series Products | 8 | PD Mixed | 32k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v4-flash-w8a8-8p-in32k-out1k-50ms) | | DeepSeek-V4-Flash | Ascend A3 Series Products | 8 | PD Mixed | 8k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v4-flash-w8a8-8p-in8k-out1k-50ms) | ## Optimal Configuration ### DeepSeek-V4-Flash W8A8 1P1D 16P IN8K OUT1K 50ms **Model**: DeepSeek-V4-Flash **Hardware**: Ascend A3 Series Products **Cards**: 16 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 8k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export FORCE_DRAFT_MODEL_NON_QUANT=1 export HCCL_OP_EXPANSION_MODE=AIV export INF_NAN_MODE_FORCE_DISABLE=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DSV4_FP4_EXPERTS=False export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_OPT_BF16_FP32_GEMM_ALGO=torch export SGLANG_OPT_DEEPGEMM_HC_PRENORM=False export SGLANG_OPT_FP8_WO_A_GEMM=0 export SGLANG_OPT_FUSE_WQA_WKV=0 export SGLANG_OPT_USE_FUSED_HASH_TOPK=False export SGLANG_OPT_USE_OVERLAP_STORE_CACHE=False export SGLANG_OPT_USE_TILELANG_MHC_POST=False export SGLANG_OPT_USE_TILELANG_MHC_PRE=False export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=8 export HCCL_SOCKET_IFNAME= export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=62084 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --page-size 128 \ --tp-size 16 \ --trust-remote-code \ --device npu \ --attention-backend dsv4 \ --watchdog-timeout 9000 \ --disaggregation-transfer-backend ascend \ --mem-fraction-static 0.62 \ --prefill-max-requests 6 \ --max-prefill-tokens 70000 \ --chunked-prefill-size -1 \ --max-running-requests 112 \ --attn-dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --quantization modelslim \ --enable-dp-lm-head \ --kv-cache-dtype bfloat16 \ --disable-cuda-graph \ --disable-radix-cache \ --load-balance-method round_robin \ --ep-dispatch-algorithm static break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 export DEEPEP_NORMAL_LONG_SEQ_ROUND=8 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --page-size 128 \ --tp-size 16 \ --trust-remote-code \ --device npu \ --attention-backend dsv4 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.75 \ --prefill-max-requests 1 \ --disable-radix-cache \ --chunked-prefill-size 32768 \ --disaggregation-transfer-backend ascend \ --max-running-requests 896 \ --attn-dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --quantization modelslim \ --enable-dp-lm-head \ --kv-cache-dtype bfloat16 \ --cuda-graph-bs-decode 1 2 4 8 16 24 36 40 48 56 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy cache_aware ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 1600 \ --max-concurrency 800 \ --random-range-ratio 1 \ --warmup-requests 0 \ --request-rate inf \ --seed 1 ``` ### DeepSeek-V4-Flash W8A8 8P IN32K OUT1K 50ms **Model**: DeepSeek-V4-Flash **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 32k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export FORCE_DRAFT_MODEL_NON_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=8 export HCCL_SOCKET_IFNAME= export INF_NAN_MODE_FORCE_DISABLE=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_DISABLE_DRAFT_EXTEND_GRAPH=1 export SGLANG_DSV4_FP4_EXPERTS=False export SGLANG_DSV4_NPU_FUSED_COMPRESSOR=1 export SGLANG_DSV4_NPU_FUSED_COMPRESSOR_PREFILL=0 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_OPT_BF16_FP32_GEMM_ALGO=torch export SGLANG_OPT_DEEPGEMM_HC_PRENORM=False export SGLANG_OPT_FP8_WO_A_GEMM=0 export SGLANG_OPT_FUSE_WQA_WKV=0 export SGLANG_OPT_USE_FUSED_HASH_TOPK=False export SGLANG_OPT_USE_OVERLAP_STORE_CACHE=False export SGLANG_OPT_USE_TILELANG_MHC_POST=False export SGLANG_OPT_USE_TILELANG_MHC_PRE=False export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=61000 export STREAMS_PER_DEVICE=32 export USE_FUSED_HC_PRE_ASCENDC=1 export USE_NPU_MOE_GATING_TOP_K=1 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --page-size 128 \ --tp-size 16 \ --trust-remote-code \ --device npu \ --prefill-max-requests 32 \ --attention-backend dsv4 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.7 \ --chunked-prefill-size 131072 \ --max-running-requests 64 \ --attn-dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --quantization modelslim \ --enable-dp-lm-head \ --kv-cache-dtype auto \ --skip-server-warmup \ --cuda-graph-bs-decode 1 2 4 8 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 \ --ep-size 16 \ --disable-radix-cache ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --random-input-len 32000 \ --random-output-len 1000 \ --num-prompts 64 \ --max-concurrency 64 \ --random-range-ratio 1 \ --warmup-requests 0 \ --request-rate inf \ --seed 1 \ --max-attempts 3 ``` ### DeepSeek-V4-Flash W8A8 8P IN8K OUT1K 50ms **Model**: DeepSeek-V4-Flash **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 8k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export FORCE_DRAFT_MODEL_NON_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=8 export HCCL_SOCKET_IFNAME= export INF_NAN_MODE_FORCE_DISABLE=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_DISABLE_DRAFT_EXTEND_GRAPH=1 export SGLANG_DSV4_FP4_EXPERTS=False export SGLANG_DSV4_NPU_FUSED_COMPRESSOR=1 export SGLANG_DSV4_NPU_FUSED_COMPRESSOR_PREFILL=1 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_OPT_BF16_FP32_GEMM_ALGO=torch export SGLANG_OPT_DEEPGEMM_HC_PRENORM=False export SGLANG_OPT_FP8_WO_A_GEMM=0 export SGLANG_OPT_FUSE_WQA_WKV=0 export SGLANG_OPT_USE_FUSED_HASH_TOPK=False export SGLANG_OPT_USE_OVERLAP_STORE_CACHE=False export SGLANG_OPT_USE_TILELANG_MHC_POST=False export SGLANG_OPT_USE_TILELANG_MHC_PRE=False export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=61000 export STREAMS_PER_DEVICE=32 export USE_FUSED_HC_PRE_ASCENDC=1 export USE_NPU_MOE_GATING_TOP_K=1 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --page-size 128 \ --tp-size 16 \ --trust-remote-code \ --device npu \ --prefill-max-requests 160 \ --attention-backend dsv4 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.7 \ --chunked-prefill-size 131072 \ --max-running-requests 160 \ --attn-dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --quantization modelslim \ --enable-dp-lm-head \ --kv-cache-dtype auto \ --skip-server-warmup \ --cuda-graph-bs-decode 1 2 4 8 10 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 \ --ep-size 16 \ --disable-radix-cache ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 160 \ --random-range-ratio 1 \ --warmup-requests 0 \ --request-rate inf \ --seed 1 \ --max-attempts 3 ``` # GLM-5.1 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1 This page focuses on optimal configuration and benchmark results for GLM-5.1 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [GLM-5.1 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | - | - | - | - | - | - | - | - | - | | GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 65k+1.5k (90% prefix cache hit rate) | 25ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in65k-out1k5-prefix90-25ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | - | - | - | - | - | - | - | - | - | | GLM-5.1 | Ascend A3 Series Products | 16 | PD Mixed | 3.5k+1.5k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-16p-in3k5-out1k5-50ms) | | GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 128k+1k | 56.4ms | 13.1s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in128k-out1k-56-4ms) | | GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 16k+1k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in16k-out1k-50ms) | | GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 64k+1k | 55.2ms | 7.58s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-55-2ms) | | GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 64k+1k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-50ms) | | GLM-5.1 | Ascend A3 Series Products | 48 | PD Disaggregation | 128k+1k (90% prefix cache hit rate) | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-2p1d-48p-in128k-out1k-prefix90-50ms) | | GLM-5.1 | Ascend A3 Series Products | 48 | PD Disaggregation | 64k+1k (90% prefix cache hit rate) | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-4p1d-48p-in64k-out1k-prefix90-50ms) | ## Optimal Configuration ### GLM-5.1 W4A8 16P IN3K5 OUT1K5 50ms **Model**: GLM-5.1 **Hardware**: Ascend A3 Series Products **Cards**: 16 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # NODE_IPS: IP addresses of each node in the cluster # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights NODE_IPS=('' '') echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEPEP_HCCL_BUFFSIZE=2500 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" for i in "${!NODE_IPS[@]}"; do if [[ "$LOCAL_HOST1" == "${NODE_IPS[$i]}" || "$LOCAL_HOST2" == "${NODE_IPS[$i]}" ]]; then echo "${NODE_IPS[$i]}" python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host ${NODE_IPS[$i]} --port 6688 \ --nnodes 2 \ --dist-init-addr ${NODE_IPS[0]}:5000 \ --node-rank $i \ --attention-backend ascend \ --device npu \ --tp-size 32 \ --attn-dp-size 16 \ --chunked-prefill-size 65536 \ --max-prefill-tokens 280000 \ --trust-remote-code \ --mem-fraction-static 0.65 \ --served-model-name glm-5 \ --cuda-graph-max-bs-decode 16 \ --max-running-requests 256 \ --quantization modelslim \ --speculative-draft-model-quantization unquant \ --moe-a2a-backend deepep \ --deepep-mode auto \ --load-balance-method round_robin \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser glm45 \ --tool-call-parser glm47 break fi done ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 128 \ --num-prompts 128 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` ### GLM-5.1 W4A8 1P1D 32P IN128K OUT1K 56.4ms **Model**: GLM-5.1 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 128k+1k **TPOT**: 56.4ms **TTFT**: 13.1s #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699 export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184 export TASK_QUEUE_ENABLE=2 export ZBAL_ENABLE_GRAPH=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --tp-size 4 \ --nnodes 2 \ --mem-fraction-static 0.72 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 16 \ --served-model-name glm-5 \ --chunked-prefill-size 32768 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 4 \ --disable-radix-cache \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --pp-size 8 \ --trust-remote-code break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=200 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --trust-remote-code \ --tp-size 32 \ --nnodes 2 \ --attn-dp-size 32 \ --ep-size 32 \ --mem-fraction-static 0.85 \ --max-running-requests 32 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs-decode 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 16 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 32 \ --num-prompts 32 \ --random-input-len 131072 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 ``` ### GLM-5.1 W4A8 1P1D 32P IN16K OUT1K 50ms **Model**: GLM-5.1 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 16k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --mem-fraction-static 0.75 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 64 \ --served-model-name glm-5 \ --chunked-prefill-size 524288 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --attn-dp-size 4 \ --load-balance-method round_robin \ --enable-prefill-cp \ --cp-strategy zigzag \ --attn-cp-size 8 \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=650 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --attn-dp-size 32 \ --ep-size 32 \ --mem-fraction-static 0.87 \ --max-running-requests 96 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs-decode 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 4 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 128 \ --num-prompts 512 \ --random-input-len 16384 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 ``` ### GLM-5.1 W4A8 1P1D 32P IN64K OUT1K 55.2ms **Model**: GLM-5.1 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 64k+1k **TPOT**: 55.2ms **TTFT**: 7.58s #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699 export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184 export TASK_QUEUE_ENABLE=2 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --trust-remote-code \ --tp-size 4 \ --nnodes 2 \ --mem-fraction-static 0.72 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 16 \ --served-model-name glm-5 \ --chunked-prefill-size 32768 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 4 \ --disable-radix-cache \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --pp-size 8 break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=200 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --attn-dp-size 32 \ --ep-size 32 \ --mem-fraction-static 0.85 \ --max-running-requests 32 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs-decode 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 16 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --trust-remote-code break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 32 \ --num-prompts 32 \ --random-input-len 65536 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 ``` ### GLM-5.1 W4A8 1P1D 32P IN64K OUT1K 50ms **Model**: GLM-5.1 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 64k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --tp-size 4 \ --nnodes 2 \ --mem-fraction-static 0.72 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 16 \ --served-model-name glm-5 \ --chunked-prefill-size 16384 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 4 \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --pp-size 8 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=200 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --attn-dp-size 32 \ --ep-size 32 \ --mem-fraction-static 0.85 \ --max-running-requests 32 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs-decode 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 16 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --num-prompts 1 \ --random-input-len 65536 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 ``` ### GLM-5.1 W4A8 1P1D 32P IN65K OUT1K5 PREFIX90 25ms **Model**: GLM-5.1 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 65k+1.5k (90% prefix cache hit rate) **TPOT**: 25ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --mem-fraction-static 0.75 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 64 \ --served-model-name glm-5 \ --chunked-prefill-size 53248 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --attn-dp-size 4 \ --load-balance-method round_robin \ --enable-prefill-cp \ --cp-strategy zigzag \ --attn-cp-size 8 \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=650 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --attn-dp-size 32 \ --ep-size 32 \ --mem-fraction-static 0.87 \ --max-running-requests 96 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs-decode 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 4 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 59904` = `round(66560 * 0.9)` is the shared prefix portion. `--gsp-question-len 6656` = `round(66560 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 480 \ --gsp-system-prompt-len 59904 \ --gsp-question-len 6656 \ --gsp-output-len 1536 \ --max-concurrency 100 \ --num-prompts 480 \ --request-rate inf ``` ### GLM-5.1 W4A8 2P1D 48P IN128K OUT1K PREFIX90 50ms **Model**: GLM-5.1 **Hardware**: Ascend A3 Series Products **Cards**: 48 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '' '' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[$(( $i / 2 * 2 ))]}:5000 \ --disaggregation-bootstrap-port $((8998 + $i / 2)) \ --node-rank $(( $i % 2 )) \ --tp-size 4 \ --nnodes 2 \ --mem-fraction-static 0.72 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 32 \ --served-model-name glm-5 \ --chunked-prefill-size 16384 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --enable-prefill-cp \ --cp-strategy zigzag \ --attn-cp-size 4 \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --pp-size 8 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=200 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=24 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --attn-dp-size 32 \ --ep-size 32 \ --mem-fraction-static 0.865 \ --max-running-requests 96 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs-decode 1 2 3 4 5 6 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 32 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 117965` = `round(131072 * 0.9)` is the shared prefix portion. `--gsp-question-len 13107` = `round(131072 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 576 \ --gsp-system-prompt-len 117965 \ --gsp-question-len 13107 \ --gsp-output-len 1024 \ --max-concurrency 144 \ --num-prompts 576 \ --request-rate inf ``` ### GLM-5.1 W4A8 4P1D 48P IN64K OUT1K PREFIX90 50ms **Model**: GLM-5.1 **Hardware**: Ascend A3 Series Products **Cards**: 48 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '' '' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port $((8998 + $i)) \ --node-rank 0 \ --tp-size 4 \ --nnodes 1 \ --mem-fraction-static 0.72 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 16 \ --served-model-name glm-5 \ --chunked-prefill-size 16384 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --enable-prefill-cp \ --cp-strategy zigzag \ --attn-cp-size 4 \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --pp-size 4 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=300 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=40 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --attn-dp-size 32 \ --ep-size 32 \ --mem-fraction-static 0.85 \ --max-running-requests 320 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs-decode 1 2 3 4 5 6 7 8 9 10 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 4 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , , , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --prefill http://:8000 9000 \ --prefill http://:8000 9001 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 58982` = `round(65536 * 0.9)` is the shared prefix portion. `--gsp-question-len 6554` = `round(65536 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 1280 \ --gsp-system-prompt-len 58982 \ --gsp-question-len 6554 \ --gsp-output-len 1024 \ --max-concurrency 320 \ --num-prompts 1280 \ --request-rate inf ``` # GLM-5.2 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_2 This page focuses on optimal configuration and benchmark results for GLM-5.2 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [GLM-5.2 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | GLM-5.2 | Ascend A3 Series Products | 32 | PD Disaggregation | 16k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#glm-5-2-w4a8-3p1d-32p-in16k-out1k-50ms) | ## Optimal Configuration ### GLM-5.2 W4A8 3P1D 32P IN16K OUT1K 50ms **Model**: GLM-5.2 **Hardware**: Ascend A3 Series Products **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 16k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 export TRANSFORMERS_VERBOSITY=error P_IP=('' '' '') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export DEEP_USE_ALLTOALL_MODE=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=128 export HCCL_SOCKET_IFNAME= export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_PP_LAYER_PARTITION=18,20,24,16 export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184 export TASK_QUEUE_ENABLE=2 export ZBAL_HCCL_OP=send,recv export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port $((8998 + $i)) \ --trust-remote-code \ --tp-size 4 \ --mem-fraction-static 0.72 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 16 \ --served-model-name glm-5 \ --chunked-prefill-size 4096 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 4 \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --pp-size 4 \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=300 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=40 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --trust-remote-code \ --tp-size 16 \ --attn-dp-size 16 \ --ep-size 16 \ --mem-fraction-static 0.895 \ --max-running-requests 128 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-max-bs-decode 8 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 8 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , , : prefill node IP addresses # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --prefill http://:8000 9000 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 64 \ --num-prompts 200 \ --random-input-len 16000 \ --random-output-len 1000 \ --random-range-ratio 1 ``` # Kimi-K2.6 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6 This page focuses on optimal configuration and benchmark results for Kimi-K2.6 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Kimi-K2.6 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | - | - | - | - | - | - | - | - | - | | Kimi-K2.6 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 20ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | - | - | - | - | - | - | - | - | - | | Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Mixed | 64k+1k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-16p-in64k-out1k-100ms) | | Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Disaggregation | 128k+1k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-100ms) | | Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Disaggregation | 128k+1k (90% prefix cache hit rate) | 100ms | 5s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-prefix90-100ms) | | Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Disaggregation | 64k+1.5k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-100ms) | | Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Disaggregation | 64k+1.5k (90% prefix cache hit rate) | 100ms | 3s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-prefix90-100ms) | | Kimi-K2.6 | Ascend A3 Series Products | 8 | PD Mixed | 1024x1024 (30)+1024 | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1024x1024-30-out1024-50ms) | | Kimi-K2.6 | Ascend A3 Series Products | 8 | PD Mixed | 1080p\_30+256 | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1080p-30-out256-50ms) | | Kimi-K2.6 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-50ms) | ## Optimal Configuration ### Kimi-K2.6 W4A8 16P IN64K OUT1K 100ms **Model**: Kimi-K2.6 **Hardware**: Ascend A3 Series Products **Cards**: 16 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 64k+1k **TPOT**: 100ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # NODE_IPS: IP addresses of each node in the cluster # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights NODE_IPS=('' '') echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEPEP_HCCL_BUFFSIZE=4400 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" for i in "${!NODE_IPS[@]}"; do if [[ "$LOCAL_HOST1" == "${NODE_IPS[$i]}" || "$LOCAL_HOST2" == "${NODE_IPS[$i]}" ]]; then echo "${NODE_IPS[$i]}" python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host ${NODE_IPS[$i]} --port 6688 \ --nnodes 2 \ --dist-init-addr ${NODE_IPS[0]}:5000 \ --node-rank $i \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --dtype bfloat16 \ --tp-size 32 \ --mem-fraction-static 0.662 \ --max-running-requests 32 \ --chunked-prefill-size 262144 \ --context-length 75000 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --attn-dp-size 32 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 \ --disable-radix-cache \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 break fi done ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 32 \ --num-prompts 32 \ --random-input-len 64000 \ --random-output-len 1000 \ --random-range-ratio 1 \ --seed 1 ``` ### Kimi-K2.6 W4A8 1P1D 16P IN128K OUT1K 100ms **Model**: Kimi-K2.6 **Hardware**: Ascend A3 Series Products **Cards**: 16 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 128k+1k **TPOT**: 100ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=8 export HCCL_SOCKET_IFNAME= export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699 export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --disable-radix-cache \ --mem-fraction-static 0.78 \ --max-running-requests 2 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --chunked-prefill-size 16384 \ --prefill-max-requests 2 \ --max-prefill-tokens 65536 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --mem-fraction-static 0.82 \ --max-running-requests 2 \ --enable-dp-lm-head \ --disable-radix-cache \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 4 6 8 16 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy cache_aware ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --num-prompts 1 \ --request-rate inf \ --random-input-len 128000 \ --random-output-len 1000 \ --random-range-ratio 1 \ --seed 1 ``` ### Kimi-K2.6 W4A8 1P1D 16P IN128K OUT1K PREFIX90 100ms **Model**: Kimi-K2.6 **Hardware**: Ascend A3 Series Products **Cards**: 16 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 100ms **TTFT**: 5s #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=8 export HCCL_SOCKET_IFNAME= export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699 export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --mem-fraction-static 0.78 \ --max-running-requests 2 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --chunked-prefill-size 16384 \ --prefill-max-requests 2 \ --max-prefill-tokens 65536 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --mem-fraction-static 0.82 \ --max-running-requests 2 \ --enable-dp-lm-head \ --disable-radix-cache \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 4 6 8 16 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy cache_aware ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 115200` = `round(128000 * 0.9)` is the shared prefix portion. `--gsp-question-len 12800` = `round(128000 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 4 \ --gsp-system-prompt-len 115200 \ --gsp-question-len 12800 \ --gsp-output-len 1000 \ --max-concurrency 1 \ --num-prompts 4 \ --request-rate inf ``` ### Kimi-K2.6 W4A8 1P1D 16P IN64K OUT1K5 100ms **Model**: Kimi-K2.6 **Hardware**: Ascend A3 Series Products **Cards**: 16 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 64k+1.5k **TPOT**: 100ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1800 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --disable-radix-cache \ --disable-cuda-graph \ --mem-fraction-static 0.78 \ --max-running-requests 1 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --chunked-prefill-size 16384 \ --prefill-max-requests 1 \ --max-prefill-tokens 65536 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --mem-fraction-static 0.82 \ --max-running-requests 16 \ --enable-dp-lm-head \ --disable-radix-cache \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 16 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --speculative-draft-model-quantization unquant break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy cache_aware ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --num-prompts 1 \ --request-rate inf \ --random-input-len 64000 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` ### Kimi-K2.6 W4A8 1P1D 16P IN64K OUT1K5 PREFIX90 100ms **Model**: Kimi-K2.6 **Hardware**: Ascend A3 Series Products **Cards**: 16 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 64k+1.5k (90% prefix cache hit rate) **TPOT**: 100ms **TTFT**: 3s #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1800 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --mem-fraction-static 0.78 \ --max-running-requests 2 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --chunked-prefill-size 16384 \ --prefill-max-requests 2 \ --max-prefill-tokens 65536 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1200 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --mem-fraction-static 0.82 \ --max-running-requests 2 \ --attn-dp-size 2 \ --enable-dp-lm-head \ --disable-radix-cache \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 4 6 8 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --speculative-draft-model-quantization unquant break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy cache_aware ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 57600` = `round(64000 * 0.9)` is the shared prefix portion. `--gsp-question-len 6400` = `round(64000 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 8 \ --gsp-system-prompt-len 57600 \ --gsp-question-len 6400 \ --gsp-output-len 1500 \ --max-concurrency 2 \ --num-prompts 8 \ --request-rate inf ``` ### Kimi-K2.6 W4A8 8P IN1024X1024 30 OUT1024 50ms **Model**: Kimi-K2.6 **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 1024x1024 (30)+1024 *Format: resolution (input tokens) + output tokens* **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEPEP_HCCL_BUFFSIZE=1280 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=112 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --quantization modelslim \ --dtype bfloat16 \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --trust-remote-code \ --device npu \ --attention-backend ascend \ --tp-size 16 \ --mem-fraction-static 0.865 \ --max-running-requests 176 \ --chunked-prefill-size 32768 \ --context-length 8192 \ --max-prefill-tokens 16384 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --attn-dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 4 8 9 10 11 \ --disable-radix-cache \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 \ --speculative-draft-model-quantization unquant \ --prefill-delayer-max-delay-passes 200 \ --enable-prefill-delayer \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 ``` #### Benchmark We tested it based on the `IMAGE` dataset with 1024x1024 resolution. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name image \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 6688 \ --image-resolution 1024x1024 \ --image-count 1 \ --max-concurrency 160 \ --num-prompts 640 \ --request-rate inf \ --random-input-len 30 \ --random-output-len 1024 \ --random-range-ratio 1 \ --warmup-requests 16 \ --seed 1 ``` ### Kimi-K2.6 W4A8 8P IN1080P 30 OUT256 50ms **Model**: Kimi-K2.6 **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 1080p\_30+256 **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEPEP_HCCL_BUFFSIZE=1024 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --quantization modelslim \ --dtype bfloat16 \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --trust-remote-code \ --device npu \ --attention-backend ascend \ --tp-size 16 \ --mem-fraction-static 0.852 \ --max-running-requests 64 \ --chunked-prefill-size 16384 \ --context-length 8192 \ --max-prefill-tokens 16384 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --attn-dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 3 4 \ --disable-radix-cache \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 \ --speculative-draft-model-quantization unquant \ --prefill-delayer-max-delay-passes 200 \ --enable-prefill-delayer \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 ``` #### Benchmark We tested it based on the `IMAGE` dataset with 1920x1080 resolution. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name image \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 6688 \ --image-resolution 1920x1080 \ --image-count 1 \ --max-concurrency 48 \ --num-prompts 196 \ --request-rate inf \ --random-input-len 30 \ --random-output-len 256 \ --random-range-ratio 1 \ --warmup-requests 16 \ --seed 1 ``` ### Kimi-K2.6 W4A8 8P IN3K5 OUT1K5 20ms **Model**: Kimi-K2.6 **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEPEP_HCCL_BUFFSIZE=1200 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --dtype bfloat16 \ --tp-size 16 \ --mem-fraction-static 0.865 \ --max-running-requests 80 \ --chunked-prefill-size 32768 \ --context-length 6144 \ --max-prefill-tokens 65536 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --attn-dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 3 4 5 \ --disable-radix-cache \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --speculative-draft-model-quantization unquant \ --prefill-delayer-max-delay-passes 200 \ --enable-prefill-delayer \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-attempts 5 \ --max-concurrency 64 \ --num-prompts 256 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --warmup-requests 0 \ --seed 1 ``` ### Kimi-K2.6 W4A8 8P IN3K5 OUT1K5 50ms **Model**: Kimi-K2.6 **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEPEP_HCCL_BUFFSIZE=1200 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=200 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --dtype bfloat16 \ --tp-size 16 \ --mem-fraction-static 0.895 \ --max-running-requests 208 \ --chunked-prefill-size 32768 \ --context-length 6144 \ --max-prefill-tokens 16384 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --attn-dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 4 8 11 12 13 \ --disable-radix-cache \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --speculative-draft-model-quantization unquant \ --prefill-delayer-max-delay-passes 50 \ --enable-prefill-delayer \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 192 \ --num-prompts 768 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --warmup-requests 0 \ --seed 1 ``` # MiMo-V2.5-Pro Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_5_pro This page focuses on the deployment of MiMo-V2.5-Pro (FP4) with DFlash speculative decoding in PD disaggregation mode on the Ascend NPU. On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Model Deployment MiMo-V2.5-Pro-FP4-DFlash is an MXFP4-quantized checkpoint with a built-in DFlash draft model (located in the `dflash/` subdirectory of the weights). The following example deploys it in 1P1D mode (1 prefill node + 1 decode node, TP8 + DP2 per node). #### Common environment setup (both nodes) ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # ASCEND_MF_STORE_URL: prefill node IP with port # HCCL_SOCKET_IFNAME / GLOO_SOCKET_IFNAME: network interface name # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 export SGLANG_SET_CPU_AFFINITY=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export HCCL_BUFFSIZE=300 export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export STREAMS_PER_DEVICE=32 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 # Use the AscendC flash attention export ASCEND_USE_FIA=1 # PD disaggregation transfer config export ASCEND_MF_STORE_URL="tcp://:24669" export ASCEND_MF_TRANSFER_PROTOCOL="device_urma" # DeepEP export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= export HCCL_HOST_SOCKET_PORT_RANGE=auto MODEL_PATH=/path/to/MiMo-V2.5-Pro-FP4-DFlash ``` #### Prefill node ```bash Command theme={null} export DEEPEP_HCCL_BUFFSIZE=2500 # Enable chunked dispatch for long sequences export DEEPEP_NORMAL_LONG_SEQ_ROUND=10 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=0 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --attention-backend ascend \ --device npu \ --tp-size 8 --nnodes 1 --node-rank 0 \ --chunked-prefill-size 8192 \ --trust-remote-code --port 10001 \ --host --max-running-requests 32 \ --mem-fraction-static 0.90 \ --swa-full-tokens-ratio 0.3 \ --disaggregation-mode prefill --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8996 \ --cuda-graph-backend-prefill disabled \ --attn-dp-size 2 --enable-dp-lm-head \ --moe-a2a-backend deepep --deepep-mode normal ``` #### Decode node ```bash Command theme={null} # Use eagle_worker_v2 and overlap plan stream to hide the draft/target preparation export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 # DFlash draft model has a longer context length than the derived value export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 export DEEPEP_HCCL_BUFFSIZE=1200 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --speculative-draft-model-path $MODEL_PATH/dflash \ --attention-backend ascend \ --device npu \ --tp-size 8 --nnodes 1 --node-rank 0 \ --trust-remote-code --port 20001 \ --host --max-running-requests 32 \ --mem-fraction-static 0.88 \ --swa-full-tokens-ratio 0.3 \ --cuda-graph-bs 1 2 4 8 12 16 \ --disaggregation-mode decode --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8996 \ --moe-a2a-backend deepep --deepep-mode low_latency \ --attn-dp-size 2 --enable-dp-lm-head \ --speculative-algorithm DFLASH \ --speculative-num-draft-tokens 8 ``` #### Router ```bash Command theme={null} python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:10001 \ --decode http://:20001 \ --host 127.0.0.1 \ --port 6688 \ --health-check-interval-secs 3600 --mini-lb ``` ### Benchmark We tested it based on the `RANDOM` dataset. #### Benchmark Prefill Node (TTFT) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --model /path/to/MiMo-V2.5-Pro-FP4-DFlash \ --dataset-name random \ --tokenize-prompt \ --random-input-len 16000 \ --random-output-len 1 \ --request-rate 0.4 \ --random-range-ratio 1 \ --num-prompts 128 \ --max-concurrency 32 ``` #### Benchmark Decode Node (TPOT) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --model /path/to/MiMo-V2.5-Pro-FP4-DFlash \ --dataset-name random \ --tokenize-prompt \ --random-input-len 16000 \ --random-output-len 1000 \ --request-rate inf \ --random-range-ratio 1 \ --num-prompts 128 \ --max-concurrency 32 ``` # MiMo-V2-Flash Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash This page focuses on optimal configuration and benchmark results for MiMo-V2-Flash on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [MiMo-V2-Flash Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | - | - | - | - | - | - | - | - | - | | MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 16k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1k-tpot-20ms) | | MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 32k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1k-tpot-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | - | - | - | - | - | - | - | - | - | | MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 16k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1-ttft-5s) | | MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 32k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1-ttft-5s) | ## Optimal Configuration ### MiMo-V2-Flash 1P1D 12P IN16K OUT1 TTFT 5s **Model**: MiMo-V2-Flash **Hardware**: Ascend A3 Series Products **Cards**: 12 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 16k+1 **TTFT**: 5s #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export HCCL_CONNECT_TIMEOUT=1800 export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_BF16_DISPATCH=0 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1024 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --tp-size 8 \ --nnodes 1 \ --chunked-prefill-size 8192 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --disaggregation-transfer-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ --cuda-graph-backend-prefill=disabled \ --dp-size 2 break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=800 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --nnodes 1 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \ --disaggregation-transfer-backend ascend \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --disable-radix-cache \ --attn-dp-size 2 \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode low_latency break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --health-check-interval-secs 3600 --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 64 \ --num-prompts 128 \ --request-rate 0.4 \ --random-input-len 16000 \ --random-output-len 1 \ --random-range-ratio 1 \ --seed 1 ``` ### MiMo-V2-Flash 1P1D 12P IN16K OUT1K TPOT 20ms **Model**: MiMo-V2-Flash **Hardware**: Ascend A3 Series Products **Cards**: 12 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 16k+1k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export HCCL_CONNECT_TIMEOUT=1800 export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_BF16_DISPATCH=0 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1024 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --tp-size 8 \ --nnodes 1 \ --chunked-prefill-size 8192 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --disaggregation-transfer-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ --cuda-graph-backend-prefill=disabled \ --dp-size 2 break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=800 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --nnodes 1 \ --trust-remote-code \ --max-running-requests 32 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --cuda-graph-bs-decode 1 2 4 8 12 16 \ --disaggregation-transfer-backend ascend \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --disable-radix-cache \ --attn-dp-size 2 \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode low_latency break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --health-check-interval-secs 3600 --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 32 \ --num-prompts 128 \ --request-rate inf \ --random-input-len 16000 \ --random-output-len 1000 \ --random-range-ratio 1 \ --seed 1 ``` ### MiMo-V2-Flash 1P1D 12P IN32K OUT1 TTFT 5s **Model**: MiMo-V2-Flash **Hardware**: Ascend A3 Series Products **Cards**: 12 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 32k+1 **TTFT**: 5s #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export HCCL_CONNECT_TIMEOUT=1800 export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_BF16_DISPATCH=0 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1024 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --tp-size 8 \ --nnodes 1 \ --chunked-prefill-size 8192 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --disaggregation-transfer-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ --cuda-graph-backend-prefill=disabled \ --dp-size 2 break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=800 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --nnodes 1 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \ --disaggregation-transfer-backend ascend \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --disable-radix-cache \ --attn-dp-size 2 \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode low_latency break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --health-check-interval-secs 3600 --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 64 \ --num-prompts 128 \ --request-rate 0.4 \ --random-input-len 32000 \ --random-output-len 1 \ --random-range-ratio 1 \ --seed 1 ``` ### MiMo-V2-Flash 1P1D 12P IN32K OUT1K TPOT 20ms **Model**: MiMo-V2-Flash **Hardware**: Ascend A3 Series Products **Cards**: 12 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 32k+1k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export HCCL_CONNECT_TIMEOUT=1800 export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_BF16_DISPATCH=0 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=1024 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --tp-size 8 \ --nnodes 1 \ --chunked-prefill-size 8192 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --disaggregation-transfer-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ --cuda-graph-backend-prefill=disabled \ --dp-size 2 break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export DEEPEP_HCCL_BUFFSIZE=800 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --nnodes 1 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \ --disaggregation-transfer-backend ascend \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --disable-radix-cache \ --attn-dp-size 2 \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode low_latency break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --health-check-interval-secs 3600 --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 48 \ --num-prompts 192 \ --request-rate inf \ --random-input-len 32000 \ --random-output-len 1000 \ --random-range-ratio 1 \ --seed 1 ``` # MiniMax-M2.5 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5 This page focuses on optimal configuration and benchmark results for MiniMax-M2.5 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [MiniMax-M2.5 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | MiniMax-M2.5 | Ascend A3 Series Products | 8 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 24.44ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in128k-out1k-prefix90-24-44ms) | | MiniMax-M2.5 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | MiniMax-M2.5 | Ascend A3 Series Products | 4 | PD Mixed | 32k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in32k-out1k-50ms) | | MiniMax-M2.5 | Ascend A3 Series Products | 4 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in64k-out1k-prefix90-50ms) | | MiniMax-M2.5 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-50ms) | ## Optimal Configuration ### MiniMax-M2.5 W8A8 4P IN32K OUT1K 50ms **Model**: MiniMax-M2.5 **Hardware**: Ascend A3 Series Products **Cards**: 4 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 32k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 export DEEPEP_NORMAL_LONG_SEQ_ROUND=64 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=128 export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=640 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_LOCAL_MEM_SIZE=60184 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 export ZBAL_ENABLE_GRAPH=1 export ZBAL_HCCL_OP=allreduce,_allgather_base,allgather,broadcast,scatter,reduce_scatter,_reduce_scatter_base,alltoall_base export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 8 \ --disable-radix-cache \ --mem-fraction-static 0.74 \ --max-running-requests 18 \ --chunked-prefill-size -1 \ --max-prefill-tokens 32768 \ --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 24 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --quantization modelslim \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --dtype bfloat16 \ --trust-remote-code \ --tokenizer-worker-num 4 \ --reasoning-parser minimax-append-think \ --tool-call-parser minimax-m2 \ --device npu ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 18 \ --num-prompts 72 \ --random-input-len 32768 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 ``` ### MiniMax-M2.5 W8A8 4P IN64K OUT1K PREFIX90 50ms **Model**: MiniMax-M2.5 **Hardware**: Ascend A3 Series Products **Cards**: 4 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_HCCL_BUFFSIZE=1024 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=140000 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 export SGLANG_NPU_FUSED_MOE_MODE=2 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 8 \ --mem-fraction-static 0.63 \ --max-running-requests 26 \ --reasoning-parser minimax-append-think \ --tool-call-parser minimax-m2 \ --enable-prefill-delayer \ --prefill-max-requests 10 \ --chunked-prefill-size 67072 \ --max-prefill-tokens 67000 \ --cuda-graph-bs-decode 2 4 8 12 16 18 20 22 24 26 \ --moe-a2a-backend ascend_fuseep \ --deepep-mode auto \ --quantization modelslim \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --dtype bfloat16 \ --trust-remote-code \ --device npu ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 58982` = `round(65536 * 0.9)` is the shared prefix portion. `--gsp-question-len 6554` = `round(65536 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 104 \ --gsp-system-prompt-len 58982 \ --gsp-question-len 6554 \ --gsp-output-len 1024 \ --max-concurrency 26 \ --num-prompts 104 \ --request-rate inf ``` ### MiniMax-M2.5 W8A8 8P IN128K OUT1K PREFIX90 24.44ms **Model**: MiniMax-M2.5 **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 24.44ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_HCCL_BUFFSIZE=1024 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=160000 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 export SGLANG_NPU_FUSED_MOE_MODE=2 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 16 \ --attn-dp-size 2 \ --mem-fraction-static 0.65 \ --max-running-requests 4 \ --reasoning-parser minimax-append-think \ --tool-call-parser minimax-m2 \ --enable-prefill-delayer \ --prefill-max-requests 4 \ --chunked-prefill-size 160000 \ --max-prefill-tokens 80000 \ --cuda-graph-bs-decode 2 4 6 8 \ --moe-a2a-backend ascend_fuseep \ --deepep-mode auto \ --quantization modelslim \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --tokenizer-worker-num 4 \ --dtype bfloat16 \ --device npu ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 117965` = `round(131072 * 0.9)` is the shared prefix portion. `--gsp-question-len 13107` = `round(131072 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 16 \ --gsp-system-prompt-len 117965 \ --gsp-question-len 13107 \ --gsp-output-len 1024 \ --max-concurrency 4 \ --num-prompts 16 \ --request-rate inf ``` ### MiniMax-M2.5 W8A8 8P IN3K5 OUT1K5 20ms **Model**: MiniMax-M2.5 **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_HCCL_BUFFSIZE=2048 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=204800 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 export SGLANG_NPU_FUSED_MOE_MODE=2 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 16 \ --attn-dp-size 16 \ --mem-fraction-static 0.53 \ --max-running-requests 96 \ --disable-radix-cache \ --reasoning-parser minimax-append-think \ --tool-call-parser minimax-m2 \ --prefill-delayer-max-delay-passes 500 \ --enable-prefill-delayer \ --prefill-max-requests 3 \ --chunked-prefill-size -1 \ --max-prefill-tokens 8192 \ --cuda-graph-bs-decode 1 2 3 4 5 6 \ --moe-a2a-backend ascend_fuseep \ --deepep-mode auto \ --quantization modelslim \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --dtype bfloat16 \ --device npu ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 112 \ --num-prompts 448 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` ### MiniMax-M2.5 W8A8 8P IN3K5 OUT1K5 50ms **Model**: MiniMax-M2.5 **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_HCCL_BUFFSIZE=1024 export GLOO_SOCKET_IFNAME= export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=204800 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 16 \ --attn-dp-size 16 \ --mem-fraction-static 0.75 \ --max-running-requests 320 \ --disable-radix-cache \ --reasoning-parser minimax-append-think \ --tool-call-parser minimax-m2 \ --prefill-delayer-max-delay-passes 500 \ --enable-prefill-delayer \ --chunked-prefill-size 196608 \ --max-prefill-tokens 8192 \ --cuda-graph-bs-decode 1 2 4 8 12 16 20 \ --moe-a2a-backend ascend_fuseep \ --fuseep-mode 2 \ --quantization modelslim \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --dtype bfloat16 \ --device npu ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 320 \ --num-prompts 1280 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` # Qwen3-235B-A22B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b This page focuses on optimal configuration and benchmark results for Qwen3-235B-A22B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-235B-A22B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3-235B-A22B | Ascend A3 Series Products | 8 | PD Mixed | 11k+1.5k | 8ms | BF16 | [Optimal Configuration](#qwen3-235b-a22b-bf16-8p-in11k-out1k5-8ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3-235B-A22B | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50.1ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-w8a8-8p-in3k5-out1k5-50-1ms) | ## Optimal Configuration ### Qwen3-235B-A22B BF16 8P IN11K OUT1K5 8ms **Model**: Qwen3-235B-A22B **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 11k+1.5k **TPOT**: 8ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEPEP_HCCL_BUFFSIZE=1600 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --max-running-requests 1 \ --dtype bfloat16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 16384 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --disable-radix-cache \ --enable-dp-lm-head \ --tp 16 \ --mem-fraction-static 0.78 \ --cuda-graph-bs-decode 1 \ --reasoning-parser qwen3 \ --tool-call-parser qwen25 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --num-prompts 1 \ --random-input-len 11000 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3-235B-A22B W8A8 8P IN3K5 OUT1K5 50.1ms **Model**: Qwen3-235B-A22B **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50.1ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEPEP_HCCL_BUFFSIZE=570 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=188416 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_FUSED_MOE_MODE=2 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 432 \ --context-length 8192 \ --dtype bfloat16 \ --chunked-prefill-size 94208 \ --max-prefill-tokens 458880 \ --sampling-backend ascend \ --ep-dispatch-algorithm static \ --disable-radix-cache \ --moe-a2a-backend ascend_fuseep \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --tp 16 \ --attn-dp-size 16 \ --enable-dp-lm-head \ --mem-fraction-static 0.8 \ --cuda-graph-bs-decode 1 2 4 8 16 20 24 26 27 \ --reasoning-parser qwen3 \ --tool-call-parser qwen25 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 432 \ --num-prompts 1728 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` # Qwen3-30B-A3B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b This page focuses on optimal configuration and benchmark results for Qwen3-30B-A3B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-30B-A3B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3-30B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 10ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-10ms) | | Qwen3-30B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 6k+1.5k | 10.25ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in6k-out1k5-bs16) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3-30B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 1k+100 | 10000ms | BF16 | [Optimal Configuration](#qwen3-30b-a3b-bf16-1p-in1k-out100) | | Qwen3-30B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-50ms) | ## Optimal Configuration ### Qwen3-30B-A3B BF16 1P IN1K OUT100 **Model**: Qwen3-30B-A3B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 1k+100 **TPOT**: 10000ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_LAUNCH_BLOCKING=0 export DP_ROUND_ROBIN=1 export GLOO_SOCKET_IFNAME= export HCCL_ALGO="level0:NA;level1:ring" export HCCL_SOCKET_IFNAME= export INF_NAN_MODE_FORCE_DISABLE=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:False export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_USE_MAX_DP_ATT=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --max-running-requests 168 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 8300 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 7 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 8 \ --tp-size 2 \ --attn-dp-size 2 \ --mem-fraction-static 0.85 \ --cuda-graph-bs-decode 1 2 4 8 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 162 \ --num-prompts 624 \ --random-input-len 1000 \ --random-output-len 100 \ --random-range-ratio 1 \ --seed 1 \ --max-attempts 4 ``` ### Qwen3-30B-A3B W8A8 1P IN3K5 OUT1K5 10ms **Model**: Qwen3-30B-A3B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 10ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_LAUNCH_BLOCKING=0 export DEEPEP_HCCL_BUFFSIZE=400 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 162 \ --disable-radix-cache \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-prefill-tokens 35000 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp-size 2 \ --mem-fraction-static 0.87 \ --cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --num-prompts 1 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3-30B-A3B W8A8 1P IN3K5 OUT1K5 50ms **Model**: Qwen3-30B-A3B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_LAUNCH_BLOCKING=0 export DEEPEP_HCCL_BUFFSIZE=400 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 162 \ --disable-radix-cache \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-prefill-tokens 35000 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp-size 2 \ --mem-fraction-static 0.87 \ --cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 160 \ --num-prompts 640 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3-30B-A3B W8A8 1P IN6K OUT1K5 BS16 **Model**: Qwen3-30B-A3B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 6k+1.5k **TPOT**: 10.25ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEPEP_HCCL_BUFFSIZE=400 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export TRANSFORMERS_VERBOSITY=error python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 16 \ --disable-radix-cache \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --chunked-prefill-size -1 \ --max-prefill-tokens 35000 \ --tp-size 2 \ --mem-fraction-static 0.6 \ --cuda-graph-bs-decode 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-attempts 5 \ --max-concurrency 16 \ --num-prompts 16 \ --random-input-len 6144 \ --random-output-len 1500 \ --random-range-ratio 1 ``` # Qwen3-32B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b This page focuses on optimal configuration and benchmark results for Qwen3-32B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-32B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3-32B | Ascend A3 Series Products | 8 | PD Mixed | 18k+4k | 6ms | BF16 | [Optimal Configuration](#qwen3-32b-bf16-8p-in18k-out4k-6ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3-32B | Ascend A3 Series Products | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-50ms) | | Qwen3-32B | Ascend A2 Series Products | 2 | PD Mixed | 3.5k+1.5k | 55ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-55ms-a2-series) | ## Optimal Configuration ### Qwen3-32B BF16 8P IN18K OUT4K 6ms **Model**: Qwen3-32B **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 18k+4k **TPOT**: 6ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --max-running-requests 1 \ --disable-radix-cache \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-prefill-tokens 65536 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --tp-size 16 \ --mem-fraction-static 0.72 \ --cuda-graph-bs-decode 1 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --num-prompts 1 \ --random-input-len 18000 \ --random-output-len 4000 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3-32B W8A8 2P IN3K5 OUT1K5 50ms **Model**: Qwen3-32B **Hardware**: Ascend A3 Series Products **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_DEEPGEMM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 101 \ --disable-radix-cache \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-prefill-tokens 35000 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp-size 4 \ --mem-fraction-static 0.845 \ --cuda-graph-bs-decode 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 100 \ --num-prompts 400 \ --random-input-len 3584 \ --random-output-len 1536 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3-32B W8A8 2P IN3K5 OUT1K5 55ms A2 Series **Model**: Qwen3-32B **Hardware**: Ascend A2 Series Products **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 55ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_DEEPGEMM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 101 \ --disable-radix-cache \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-prefill-tokens 35000 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp-size 4 \ --mem-fraction-static 0.845 \ --cuda-graph-bs-decode 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 100 \ --num-prompts 400 \ --random-input-len 3584 \ --random-output-len 1536 \ --random-range-ratio 1 \ --seed 1 ``` # Qwen3.5-397B-A17B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b This page focuses on optimal configuration and benchmark results for Qwen3.5-397B-A17B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3.5-397B-A17B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 128k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in128k-out1k-20ms) | | Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 16k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in16k-out1k-20ms) | | Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 22.2ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in3k5-out1k5-22-2ms) | | Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 64k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in64k-out1k-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 128k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in128k-out1k-50ms) | | Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in128k-out1k-prefix90-50ms) | | Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 16k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in16k-out1k-50ms) | | Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in3k5-out1k5-50ms) | | Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 64k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in64k-out1k-50ms) | | Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in64k-out1k-prefix90-50ms) | ## Optimal Configuration ### Qwen3.5-397B-A17B W4A8 8P IN128K OUT1K 20ms **Model**: Qwen3.5-397B-A17B **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 128k+1k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=60672 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 131072 \ --prefill-max-requests 1 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 16 \ --mem-fraction-static 0.6 \ --cuda-graph-bs-decode 2 3 4 5 6 8 10 12 14 16 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --warmup-requests 2 \ --max-concurrency 3 \ --num-prompts 3 \ --random-input-len 131072 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 \ --request-rate inf \ --temperature 0.6 \ --top-p 0.95 ``` ### Qwen3.5-397B-A17B W4A8 8P IN128K OUT1K 50ms **Model**: Qwen3.5-397B-A17B **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 128k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=60672 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 131072 \ --prefill-max-requests 1 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 16 \ --mem-fraction-static 0.6 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --warmup-requests 8 \ --max-concurrency 10 \ --num-prompts 10 \ --random-input-len 131072 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 \ --request-rate inf \ --temperature 0.6 \ --top-p 0.95 ``` ### Qwen3.5-397B-A17B W4A8 8P IN128K OUT1K PREFIX90 50ms **Model**: Qwen3.5-397B-A17B **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_HCCL_BUFFSIZE=2200 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 131072 \ --max-mamba-cache-size 320 \ --prefill-max-requests 10 \ --mamba-radix-cache-strategy extra_buffer \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.6 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 117965` = `round(131072 * 0.9)` is the shared prefix portion. `--gsp-question-len 13107` = `round(131072 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 40 \ --gsp-system-prompt-len 117965 \ --gsp-question-len 13107 \ --gsp-output-len 1024 \ --max-concurrency 40 \ --num-prompts 40 \ --request-rate inf \ --temperature 0.6 \ --top-p 0.95 ``` ### Qwen3.5-397B-A17B W4A8 8P IN16K OUT1K 20ms **Model**: Qwen3.5-397B-A17B **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 16k+1k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=20 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=59648 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 50000 \ --prefill-max-requests 4 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 48 \ --mem-fraction-static 0.8 \ --max-total-tokens 210000 \ --cuda-graph-bs-decode 2 4 6 8 10 12 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --attn-dp-size 4 \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --warmup-requests 32 \ --max-concurrency 40 \ --num-prompts 40 \ --random-input-len 16384 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 \ --request-rate inf \ --temperature 0.6 \ --top-p 0.95 ``` ### Qwen3.5-397B-A17B W4A8 8P IN16K OUT1K 50ms **Model**: Qwen3.5-397B-A17B **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 16k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=20 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=58624 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 65536 \ --prefill-max-requests 4 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 144 \ --mem-fraction-static 0.8 \ --max-total-tokens 635000 \ --cuda-graph-bs-decode 2 4 6 8 12 14 16 18 20 24 26 28 30 32 34 36 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --attn-dp-size 4 \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --warmup-requests 8 \ --max-concurrency 132 \ --num-prompts 132 \ --random-input-len 16384 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 \ --request-rate inf \ --temperature 0.6 \ --top-p 0.95 ``` ### Qwen3.5-397B-A17B W4A8 8P IN3K5 OUT1K5 22.2ms **Model**: Qwen3.5-397B-A17B **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 22.2ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export DEEPEP_NORMAL_LONG_SEQ_ROUND=6 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=58624 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 35000 \ --max-total-tokens 128000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 160 \ --mem-fraction-static 0.8 \ --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --attn-dp-size 8 \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --warmup-requests 64 \ --max-concurrency 160 \ --num-prompts 160 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 \ --request-rate inf \ --temperature 0.6 \ --top-p 0.95 ``` ### Qwen3.5-397B-A17B W4A8 8P IN3K5 OUT1K5 50ms **Model**: Qwen3.5-397B-A17B **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export DEEPEP_NORMAL_LONG_SEQ_ROUND=6 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=59648 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 17500 \ --max-total-tokens 280000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 432 \ --mem-fraction-static 0.8 \ --cuda-graph-bs-decode 2 4 6 8 12 16 20 24 28 32 36 40 44 48 50 52 54 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --attn-dp-size 8 \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --warmup-requests 16 \ --max-concurrency 432 \ --num-prompts 432 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 \ --request-rate inf \ --temperature 0.6 \ --top-p 0.95 ``` ### Qwen3.5-397B-A17B W4A8 8P IN64K OUT1K 20ms **Model**: Qwen3.5-397B-A17B **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 64k+1k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=20 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=58672 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 65536 \ --prefill-max-requests 1 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 16 \ --mem-fraction-static 0.6 \ --max-total-tokens 1065000 \ --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --attn-dp-size 2 \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --warmup-requests 6 \ --max-concurrency 6 \ --num-prompts 6 \ --random-input-len 65536 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 \ --request-rate inf \ --temperature 0.6 \ --top-p 0.95 ``` ### Qwen3.5-397B-A17B W4A8 8P IN64K OUT1K 50ms **Model**: Qwen3.5-397B-A17B **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 64k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=20 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=58672 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 65536 \ --prefill-max-requests 1 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 32 \ --mem-fraction-static 0.6 \ --max-total-tokens 1065000 \ --cuda-graph-bs-decode 2 4 6 8 12 14 16 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --attn-dp-size 2 \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --warmup-requests 8 \ --max-concurrency 24 \ --num-prompts 24 \ --random-input-len 65536 \ --random-output-len 1024 \ --random-range-ratio 1 \ --seed 1 \ --request-rate inf \ --temperature 0.6 \ --top-p 0.95 ``` ### Qwen3.5-397B-A17B W4A8 8P IN64K OUT1K PREFIX90 50ms **Model**: Qwen3.5-397B-A17B **Hardware**: Ascend A3 Series Products **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_HCCL_BUFFSIZE=2200 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=20 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 65536 \ --max-mamba-cache-size 640 \ --mamba-radix-cache-strategy extra_buffer \ --trust-remote-code \ --max-running-requests 128 \ --mem-fraction-static 0.6 \ --max-total-tokens 1310720 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --attn-dp-size 2 \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 58982` = `round(65536 * 0.9)` is the shared prefix portion. `--gsp-question-len 6554` = `round(65536 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 96 \ --gsp-system-prompt-len 58982 \ --gsp-question-len 6554 \ --gsp-output-len 1024 \ --max-concurrency 96 \ --num-prompts 96 \ --request-rate inf \ --temperature 0.6 \ --top-p 0.95 ``` # Qwen3.6-27B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b This page focuses on optimal configuration and benchmark results for Qwen3.6-27B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3.6-27B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1024x1024-30-out1024-50ms) | | Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 1080p\_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1080p-30-out256-50ms) | | Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in64k-out1k-prefix90-50ms) | | Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in3k5-out1k5-50ms) | | Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in64k-out1k-50ms) | | Qwen3.6-27B | Ascend A3 Series Products | 2 | PD Mixed | 128k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in128k-out1k-50ms) | | Qwen3.6-27B | Ascend A3 Series Products | 2 | PD Mixed | 16k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in16k-out1k-50ms) | | Qwen3.6-27B | Ascend A3 Series Products | 2 | PD Mixed | 64k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in64k-out1k-50ms) | ## Optimal Configuration ### Qwen3.6-27B 1P IN1024X1024 30 OUT1024 50ms **Model**: Qwen3.6-27B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 1024x1024 (30)+1024 *Format: resolution (input tokens) + output tokens* **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=300 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_VIT_ENABLE_CUDA_GRAPH=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 52000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 60 \ --max-mamba-cache-size 60 \ --mem-fraction-static 0.74 \ --cuda-graph-bs-decode 2 4 8 14 16 24 26 32 36 37 40 42 44 45 46 50 52 60 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --mm-enable-dp-encoder \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `IMAGE` dataset with 1024x1024 resolution. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name image \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 60 \ --num-prompts 240 \ --random-input-len 30 \ --random-output-len 1024 \ --random-range-ratio 1 \ --image-resolution 1024x1024 \ --image-count 1 \ --seed 1 ``` ### Qwen3.6-27B 1P IN1080P 30 OUT256 50ms **Model**: Qwen3.6-27B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 1080p\_30+256 **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_VIT_ENABLE_CUDA_GRAPH=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 82688 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 38 \ --max-mamba-cache-size 38 \ --mem-fraction-static 0.7 \ --cuda-graph-bs-decode 1 2 4 8 10 12 16 20 24 28 30 32 35 38 \ --enable-prefill-delayer \ --prefill-delayer-queue-min-ratio 0.45 \ --prefill-delayer-max-delay-ms 5500 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `IMAGE` dataset with 1920x1080 resolution. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name image \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 6688 \ --warmup-requests 38 \ --max-concurrency 42 \ --num-prompts 152 \ --random-input-len 30 \ --random-output-len 256 \ --random-range-ratio 1 \ --image-resolution 1920x1080 \ --image-count 1 \ --seed 1 ``` ### Qwen3.6-27B 1P IN64K OUT1K PREFIX90 50ms **Model**: Qwen3.6-27B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size 32768 \ --max-prefill-tokens 32768 \ --mamba-radix-cache-strategy extra_buffer \ --trust-remote-code \ --max-running-requests 20 \ --max-mamba-cache-size 160 \ --mem-fraction-static 0.82 \ --cuda-graph-bs-decode 1 2 5 10 15 17 19 20 \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 57600` = `round(64000 * 0.9)` is the shared prefix portion. `--gsp-question-len 6400` = `round(64000 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 20 \ --gsp-system-prompt-len 57600 \ --gsp-question-len 6400 \ --gsp-output-len 1000 \ --max-concurrency 20 \ --num-prompts 20 \ --request-rate inf ``` ### Qwen3.6-27B W8A8 1P IN3K5 OUT1K5 50ms **Model**: Qwen3.6-27B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=130 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 60000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 64 \ --max-mamba-cache-size 74 \ --mem-fraction-static 0.7 \ --cuda-graph-bs-decode 2 8 16 32 40 45 50 54 \ --enable-multimodal \ --quantization modelslim \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 54 \ --num-prompts 216 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3.6-27B W8A8 1P IN64K OUT1K 50ms **Model**: Qwen3.6-27B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 64k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 48000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 6 \ --max-mamba-cache-size 16 \ --mem-fraction-static 0.6 \ --cuda-graph-bs-decode 1 2 4 5 6 \ --quantization modelslim \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 6 \ --num-prompts 12 \ --random-input-len 64000 \ --random-output-len 1000 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3.6-27B W8A8 2P IN128K OUT1K 50ms **Model**: Qwen3.6-27B **Hardware**: Ascend A3 Series Products **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 128k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=20 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 4 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 74000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 6 \ --max-mamba-cache-size 7 \ --mem-fraction-static 0.63 \ --cuda-graph-bs-decode 1 2 4 5 6 \ --enable-multimodal \ --quantization modelslim \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 4 \ --num-prompts 16 \ --random-input-len 128000 \ --random-output-len 1000 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3.6-27B W8A8 2P IN16K OUT1K 50ms **Model**: Qwen3.6-27B **Hardware**: Ascend A3 Series Products **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 16k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=50 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 4 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 58000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 37 \ --max-mamba-cache-size 74 \ --mem-fraction-static 0.7 \ --cuda-graph-bs-decode 1 2 3 4 6 8 10 12 14 16 18 20 21 23 24 25 26 27 28 29 30 31 33 35 37 \ --quantization modelslim \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 37 \ --warmup-requests 4 \ --num-prompts 37 \ --random-input-len 16000 \ --random-output-len 1000 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3.6-27B W8A8 2P IN64K OUT1K 50ms **Model**: Qwen3.6-27B **Hardware**: Ascend A3 Series Products **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 64k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=30 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 4 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 50000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 28 \ --max-mamba-cache-size 50 \ --mem-fraction-static 0.7 \ --cuda-graph-bs-decode 2 4 6 \ --enable-multimodal \ --quantization modelslim \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 6 \ --num-prompts 24 \ --random-input-len 64000 \ --random-output-len 1000 \ --random-range-ratio 1 \ --seed 1 ``` # Qwen3.6-35B-A3B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b This page focuses on optimal configuration and benchmark results for Qwen3.6-35B-A3B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3.6-35B-A3B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 254k+1k | 16.1ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in254k-out1k) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1024x1024-30-out1024-50ms) | | Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 1080p\_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1080p-30-out256-50ms) | | Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 128k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-50ms) | | Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-prefix90-50ms) | | Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in3k5-out1k5-50ms) | | Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-50ms) | | Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-prefix90-50ms) | | Qwen3.6-35B-A3B | Ascend A3 Series Products | 2 | PD Mixed | 984k+1k | 40.91ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-2p-in984k-out1k) | ## Optimal Configuration ### Qwen3.6-35B-A3B 1P IN1024X1024 30 OUT1024 50ms **Model**: Qwen3.6-35B-A3B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 1024x1024 (30)+1024 *Format: resolution (input tokens) + output tokens* **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_VIT_ENABLE_CUDA_GRAPH=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 9999999 \ --max-total-tokens 365000 \ --prefill-max-requests 30 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 120 \ --max-mamba-cache-size 120 \ --mem-fraction-static 0.85 \ --cuda-graph-bs-decode 4 16 32 48 64 110 165 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `IMAGE` dataset with 1024x1024 resolution. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name image \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 120 \ --num-prompts 120 \ --warmup-requests 120 \ --random-input-len 30 \ --random-output-len 1024 \ --random-range-ratio 1 \ --image-resolution 1024x1024 \ --image-count 1 \ --seed 1 \ --request-rate inf ``` ### Qwen3.6-35B-A3B 1P IN1080P 30 OUT256 50ms **Model**: Qwen3.6-35B-A3B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 1080p\_30+256 **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_VIT_ENABLE_CUDA_GRAPH=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 150000 \ --max-total-tokens 200000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 42 \ --max-mamba-cache-size 42 \ --mem-fraction-static 0.75 \ --cuda-graph-bs-decode 4 8 16 24 48 64 80 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `IMAGE` dataset with 1920x1080 resolution. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name image \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 42 \ --num-prompts 42 \ --warmup-requests 42 \ --random-input-len 30 \ --random-output-len 256 \ --random-range-ratio 1 \ --image-resolution 1920x1080 \ --image-count 1 \ --seed 1 \ --request-rate inf ``` ### Qwen3.6-35B-A3B 1P IN128K OUT1K 50ms **Model**: Qwen3.6-35B-A3B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 128k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_HCCL_BUFFSIZE=1600 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=20 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-total-tokens 420000 \ --max-prefill-tokens 128000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 3 \ --max-mamba-cache-size 3 \ --mem-fraction-static 0.9 \ --cuda-graph-bs-decode 1 2 3 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 3 \ --warmup-requests 3 \ --num-prompts 3 \ --random-input-len 128000 \ --random-output-len 1000 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3.6-35B-A3B 1P IN128K OUT1K PREFIX90 50ms **Model**: Qwen3.6-35B-A3B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=30 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size 16384 \ --max-prefill-tokens 65536 \ --trust-remote-code \ --enable-prefill-delayer \ --mamba-radix-cache-strategy extra_buffer \ --max-running-requests 103 \ --max-mamba-cache-size 85 \ --mem-fraction-static 0.85 \ --cuda-graph-bs-decode 2 4 8 16 32 48 64 80 96 103 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 115200` = `round(128000 * 0.9)` is the shared prefix portion. `--gsp-question-len 12800` = `round(128000 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 412 \ --gsp-system-prompt-len 115200 \ --gsp-question-len 12800 \ --gsp-output-len 1000 \ --max-concurrency 103 \ --num-prompts 412 \ --request-rate inf ``` ### Qwen3.6-35B-A3B 1P IN254K OUT1K **Model**: Qwen3.6-35B-A3B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 254k+1k **TPOT**: 16.1ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size 131072 \ --max-prefill-tokens 254000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 1 \ --max-mamba-cache-size 6 \ --mem-fraction-static 0.65 \ --cuda-graph-bs-decode 1 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --warmup-requests 1 \ --num-prompts 1 \ --random-input-len 254000 \ --random-output-len 1000 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3.6-35B-A3B 1P IN3K5 OUT1K5 50ms **Model**: Qwen3.6-35B-A3B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=100 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-total-tokens 659840 \ --max-prefill-tokens 43400 \ --disable-radix-cache \ --trust-remote-code \ --prefill-max-requests 12 \ --max-running-requests 122 \ --max-mamba-cache-size 122 \ --mem-fraction-static 0.9 \ --cuda-graph-bs-decode 4 16 32 64 96 116 120 122 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 122 \ --warmup-requests 122 \ --num-prompts 122 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3.6-35B-A3B 1P IN64K OUT1K 50ms **Model**: Qwen3.6-35B-A3B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 64k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-total-tokens 600000 \ --max-prefill-tokens 65536 \ --disable-radix-cache \ --trust-remote-code \ --enable-prefill-delayer \ --max-running-requests 10 \ --max-mamba-cache-size 20 \ --mem-fraction-static 0.65 \ --cuda-graph-bs-decode 2 4 8 12 14 16 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 10 \ --warmup-requests 10 \ --num-prompts 40 \ --random-input-len 64000 \ --random-output-len 1000 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3.6-35B-A3B 1P IN64K OUT1K PREFIX90 50ms **Model**: Qwen3.6-35B-A3B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_HCCL_BUFFSIZE=300 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-total-tokens 470784 \ --max-prefill-tokens 65536 \ --trust-remote-code \ --mamba-radix-cache-strategy extra_buffer \ --max-running-requests 40 \ --max-mamba-cache-size 200 \ --mem-fraction-static 0.9 \ --cuda-graph-bs-decode 2 8 16 24 32 36 40 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 58982` = `round(65536 * 0.9)` is the shared prefix portion. `--gsp-question-len 6554` = `round(65536 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 40 \ --gsp-system-prompt-len 58982 \ --gsp-question-len 6554 \ --gsp-output-len 1024 \ --max-concurrency 40 \ --num-prompts 40 \ --request-rate inf ``` ### Qwen3.6-35B-A3B 2P IN984K OUT1K **Model**: Qwen3.6-35B-A3B **Hardware**: Ascend A3 Series Products **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 984k+1k **TPOT**: 40.91ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 4 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size 131072 \ --max-prefill-tokens 984000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 1 \ --max-mamba-cache-size 6 \ --mem-fraction-static 0.68 \ --cuda-graph-bs-decode 1 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --context-length 1010000 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --warmup-requests 1 \ --num-prompts 1 \ --random-input-len 984000 \ --random-output-len 1000 \ --random-range-ratio 1 \ --seed 1 ``` # Qwen3-8B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b This page focuses on optimal configuration and benchmark results for Qwen3-8B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-8B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3-8B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 5ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-5ms) | | Qwen3-8B | Ascend A3 Series Products | 1 | PD Mixed | 6k+1.5k | 11.79ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in6k-out1k5-bs16) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3-8B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 37ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-37ms) | ## Optimal Configuration ### Qwen3-8B W8A8 1P IN3K5 OUT1K5 37ms **Model**: Qwen3-8B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 37ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=50 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 70 \ --max-prefill-tokens 16384 \ --disable-radix-cache \ --chunked-prefill-size 16384 \ --tp-size 1 \ --mem-fraction-static 0.85 \ --cuda-graph-bs-decode 8 12 24 36 48 51 55 60 63 64 66 68 70 \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 64 \ --num-prompts 256 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3-8B W8A8 1P IN3K5 OUT1K5 5ms **Model**: Qwen3-8B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 5ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 1 \ --max-prefill-tokens 16384 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --tp-size 2 \ --mem-fraction-static 0.894 \ --cuda-graph-bs-decode 1 \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-attempts 5 \ --max-concurrency 1 \ --num-prompts 4 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3-8B W8A8 1P IN6K OUT1K5 BS16 **Model**: Qwen3-8B **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 6k+1.5k **TPOT**: 11.79ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 16 \ --max-prefill-tokens 16384 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --tp-size 2 \ --mem-fraction-static 0.894 \ --cuda-graph-bs-decode 1 5 15 16 \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-attempts 5 \ --max-concurrency 16 \ --num-prompts 16 \ --random-input-len 6144 \ --random-output-len 1500 \ --random-range-ratio 1 ``` # Qwen3-Next-80B-A3B-Instruct Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct This page focuses on optimal configuration and benchmark results for Qwen3-Next-80B-A3B-Instruct on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-Next-80B-A3B-Instruct Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct). On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3-Next-80B-A3B-Instruct | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-1p-in3k5-out1k5-20ms) | | Qwen3-Next-80B-A3B-Instruct | Ascend A3 Series Products | 2 | PD Mixed | 6k+1.5k | 15.62ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in6k-out1k5-bs16) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | - | - | - | - | - | - | - | - | | Qwen3-Next-80B-A3B-Instruct | Ascend A3 Series Products | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in3k5-out1k5-50ms) | ## Optimal Configuration ### Qwen3-Next-80B-A3B-Instruct W8A8 1P IN3K5 OUT1K5 20ms **Model**: Qwen3-Next-80B-A3B-Instruct **Hardware**: Ascend A3 Series Products **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_HCCL_BUFFSIZE=2000 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 export DEEPEP_NORMAL_LONG_SEQ_ROUND=10 export FORCE_DRAFT_MODEL_NON_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=400 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_NPU_USE_MULTI_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_WARMUP_TIMEOUT=3600 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 export ZBCCL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export ZBCCL_ENABLE_GRAPH=1 export ZBCCL_LOCAL_MEM_SIZE=60416 export ZBCCL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --page-size 128 \ --tp-size 2 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.85 \ --disable-radix-cache \ --max-prefill-tokens 28672 \ --context-length 26384 \ --max-total-tokens 122304 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-running-requests 2 \ --cuda-graph-bs-decode 2 \ --mamba-ssm-dtype bfloat16 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --num-prompts 1 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3-Next-80B-A3B-Instruct W8A8 2P IN3K5 OUT1K5 50ms **Model**: Qwen3-Next-80B-A3B-Instruct **Hardware**: Ascend A3 Series Products **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export FORCE_DRAFT_MODEL_NON_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=64 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=330 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_NPU_USE_MULTI_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_WARMUP_TIMEOUT=3600 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=59648 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_HCCL_OP=allreduce,_allgather_base,allgather,broadcast,scatter,reduce_scatter,_reduce_scatter_base,alltoall_base export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --page-size 128 \ --tp-size 4 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.75 \ --disable-radix-cache \ --max-prefill-tokens 14080 \ --context-length 26384 \ --chunked-prefill-size -1 \ --max-running-requests 300 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --attn-dp-size 2 \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 3 4 5 6 7 8 10 12 14 16 18 20 22 24 26 28 30 32 40 44 48 52 56 60 64 72 80 88 96 104 112 120 128 136 144 150 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 300 \ --num-prompts 300 \ --random-input-len 3500 \ --random-output-len 1500 \ --random-range-ratio 1 \ --seed 1 ``` ### Qwen3-Next-80B-A3B-Instruct W8A8 2P IN6K OUT1K5 BS16 **Model**: Qwen3-Next-80B-A3B-Instruct **Hardware**: Ascend A3 Series Products **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 6k+1.5k **TPOT**: 15.62ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_HCCL_BUFFSIZE=2000 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 export DEEPEP_NORMAL_LONG_SEQ_ROUND=10 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export FORCE_DRAFT_MODEL_NON_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=400 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_NPU_USE_MULTI_STREAM=0 export SGLANG_WARMUP_TIMEOUT=3600 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 export ZBCCL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export ZBCCL_ENABLE_GRAPH=1 export ZBCCL_LOCAL_MEM_SIZE=60416 export ZBCCL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --page-size 128 \ --tp-size 4 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.85 \ --disable-radix-cache \ --max-prefill-tokens 28672 \ --context-length 81920 \ --max-total-tokens 122304 \ --attn-dp-size 2 \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-running-requests 16 \ --cuda-graph-bs-decode 2 4 8 \ --mamba-ssm-dtype bfloat16 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-attempts 5 \ --max-concurrency 16 \ --num-prompts 16 \ --random-input-len 6144 \ --random-output-len 1500 \ --random-range-ratio 1 ``` # DeepSeek-R1 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1 ## Introduction DeepSeek-R1 is a Mixture-of-Experts (MoE) large language model developed by DeepSeek, featuring 671B total parameters with 37B active parameters. It employs Multi-head Latent Attention (MLA) and DeepSeekMoE architecture, with built-in multi-token prediction (MTP) for speculative decoding. The model excels at reasoning, math, and code tasks through reinforcement learning-based training. This document demonstrates the deployment of DeepSeek-R1 on Ascend NPUs using SGLang, including single-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (DeepSeek-R1) is fully supported in this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 4 8 20 21 22` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 2 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 3` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | | MLAPO | `export SGLANG_NPU_USE_MLAPO=1` | | Multistream MoE | `export SGLANG_NPU_USE_MULTI_STREAM=1` | | NZ Weight Format | `export SGLANG_USE_FIA_NZ=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [DeepSeek-R1-0528-W4A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8) (Quantized version, 376GB) * [DeepSeek-R1-0528-W8A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w8a8) (Quantized version, 647GB) The W4A8 variant (376GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 Series node or 8 dies on A3 Series (4 cards). The W8A8 variant (647GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 Series node (8 cards, 16 dies) or two A2 Series nodes. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [DeepSeek-R1 Best Practice — W4A8 8P PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1#single-node-pd-mixed). ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [DeepSeek-R1 Best Practice — W8A8 32P PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1#pd-disaggregation). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [DeepSeek-R1 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # DeepSeek-V3.2 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2 ## Introduction DeepSeek-V3.2 is a Mixture-of-Experts (MoE) large language model developed by DeepSeek, featuring 685B total parameters with 37B active parameters. It employs Multi-head Latent Attention (MLA) and DeepSeekMoE architecture, with built-in multi-token prediction (MTP) for speculative decoding. DeepSeek-V3.2 introduces the DeepSeek Sparse Attention (DSA) mechanism, a fine-grained sparse attention mechanism powered by a lightning indexer, enabling significant efficiency improvements in long-context scenarios. This document demonstrates the deployment of DeepSeek-V3.2 on Ascend NPUs using SGLang, including multi-node PD disaggregation mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (DeepSeek-V3.2) is fully supported in this version. To use the latest features (e.g., PD disaggregation, speculative decoding, DSA context parallel), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 32` | | Data Parallelism | `--dp-size 8` | | Expert Parallelism | `--ep-size 32 \`
`--moe-a2a-backend deepep \`
`--deepep-mode low_latency` | | Context Parallelism | `--enable-nsa-prefill-context-parallel \`
`--nsa-prefill-cp-mode in-seq-split \`
`--attn-cp-size 32` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 3 4 5 6 7 8` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [DeepSeek-V3.2-W8A8](https://www.modelscope.cn/models/sgl-npu/DeepSeek-V3.2-W8A8) (Quantized version, 694.47GB) The W8A8 variant (694.47GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 Series node (8 cards, 16 dies) or two A2 Series nodes. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [DeepSeek-V3.2 Best Practice — PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2#pd-disaggregation). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [DeepSeek-V3.2 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # DeepSeek-V4-Flash Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash ## Introduction DeepSeek-V4-Flash is a Mixture-of-Experts (MoE) large language model developed by DeepSeek, featuring 284B total parameters with 13B active parameters. It employs hybrid sparse attention (CSA + HCA) and DeepSeekMoE architecture with a bundled multi-token prediction (MTP) draft head for speculative decoding, supporting a 1M-token context length. This document demonstrates the deployment of DeepSeek-V4-Flash on Ascend NPUs using SGLang, including multi-node PD disaggregation mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (DeepSeek-V4-Flash) is fully supported in this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Attention Backend | `--attention-backend dsv4` | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 16 24 36 40 48 56` | | Speculative Decoding | `--speculative-algorithm EAGLE \`
`--speculative-num-steps 2 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 3` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | | DP Attention | `--attn-dp-size 16` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [DeepSeek-V4-Flash-w8a8-mtp](https://www.modelscope.cn/models/Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp) (W8A8 quantized version, 300GB) The W8A8 variant (300GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 Series node or 8 dies on A3 Series (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node PD mixed deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. In such scenarios, DeepSeek-V4-Flash can be launched with half of an Ascend A3 Series node (4 cards, 8 dies) or a full Ascend A2 Series node (8 cards). The configuration below was validated on an Ascend A3 Series node (4 cards, 8 dies) with `--tp-size 8`. ```bash Command theme={null} # ============================================================ # Before running, update the following variable: # MODEL_PATH: path to the model weights directory # ============================================================ MODEL_PATH=/path/to/model-weights source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export FORCE_DRAFT_MODEL_NON_QUANT=1 export HCCL_BUFFSIZE=8 export HCCL_OP_EXPANSION_MODE=AIV export INF_NAN_MODE_FORCE_DISABLE=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DSV4_FP4_EXPERTS=False export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_OPT_BF16_FP32_GEMM_ALGO=torch export SGLANG_OPT_DEEPGEMM_HC_PRENORM=False export SGLANG_OPT_FP8_WO_A_GEMM=0 export SGLANG_OPT_FUSE_WQA_WKV=0 export SGLANG_OPT_USE_FUSED_HASH_TOPK=False export SGLANG_OPT_USE_OVERLAP_STORE_CACHE=False export SGLANG_OPT_USE_TILELANG_MHC_POST=False export SGLANG_OPT_USE_TILELANG_MHC_PRE=False export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL="tcp://127.0.0.1:14699" export SGLANG_ZBAL_LOCAL_MEM_SIZE=62084 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path "${MODEL_PATH}" \ --page-size 128 \ --tp-size 8 \ --trust-remote-code \ --device npu \ --attention-backend dsv4 \ --watchdog-timeout 9000 \ --host 127.0.0.1 --port 6688 \ --mem-fraction-static 0.8 \ --prefill-max-requests 160 \ --max-prefill-tokens 80000 \ --chunked-prefill-size 131072 \ --max-running-requests 160 \ --attn-dp-size 8 \ --moe-a2a-backend deepep --deepep-mode auto \ --quantization modelslim --enable-dp-lm-head \ --kv-cache-dtype bfloat16 ``` For the complete benchmark data and additional single-node configurations, see [DeepSeek-V4-Flash Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash#single-node-pd-mixed). ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [DeepSeek-V4-Flash Best Practice — PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash#pd-disaggregation). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "deepseek-v4-flash", "messages": [{"role": "user", "content": "What is the capital of France?"}] }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [DeepSeek-V4-Flash Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # GLM-5.1 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1 ## Introduction GLM-5.1 is a Mixture-of-Experts (MoE) large language model developed by Z.ai, featuring 744B total parameters with 40B active parameters. It uses 256 routed experts (top-8) plus one shared expert, with Multi-head Latent Attention (MLA) and DeepSeek Sparse Attention (DSA), and a built-in multi-token prediction (MTP) head for speculative decoding. The model features built-in bilingual (Chinese-English) capabilities with a unified pre-training framework, excelling at reasoning, math, code, and tool calling tasks. GLM-5.1 supports both Thinking mode (step-by-step reasoning) and Instruct mode (direct response), with a native context window of approximately 200k tokens. This document demonstrates the deployment of GLM-5.1 on Ascend NPUs using SGLang, including multi-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (GLM-5.1) is fully supported in this version. To use the latest features (e.g., speculative decoding, multi-node deployment), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Context Parallelism | `--enable-nsa-prefill-context-parallel \`
`--nsa-prefill-cp-mode in-seq-split \`
`--attn-cp-size 4` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 3 4 5 6` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [GLM-5.1](https://www.modelscope.cn/models/ZhipuAI/GLM-5.1) (BF16, 1.51TB) * [GLM-5.1-w4a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.1-w4a8) (Quantized version, 420.17GB) * You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively. We recommend deploying the W4A8 variant for reduced resource usage and higher throughput. It (420.17GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 Series node or 8 dies on A3 Series (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Multi-node PD mixed deployment Multi-node deployment distributes the model across multiple Ascend A3 Series nodes using tensor parallelism while keeping prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single node can provide. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [GLM-5.1 Best Practice — Multi-node PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1#multi-node-pd-mixed). ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [GLM-5.1 Best Practice — PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1#pd-disaggregation). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [GLM-5.1 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # GLM-5.2 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2 ## Introduction GLM-5.2 is a large language model in the GLM (General Language Model) series, jointly developed by the KEG Laboratory of Tsinghua University and Zhipu AI. GLM-5.2 adopts the DeepSeek-V3/V3.2 architecture, including DeepSeek Sparse Attention (DSA) and multi-token prediction (MTP), and supports high-throughput inference with SGLang on Ascend NPUs. This document demonstrates the deployment of GLM-5.2 on Ascend NPUs using SGLang, including single-node deployment, multi-node deployment, prefill-decode disaggregation, feature configuration, and performance optimization. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 16` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [GLM-5.2](https://www.modelscope.cn/models/ZhipuAI/GLM-5.2) (BF16, 1.51TB) * [GLM-5.2-w8a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/) (Quantized version without MTP, 774.08GB) * You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively. We recommend deploying the W8A8 variant for reduced resource usage and higher throughput. It (774.08GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 Series node (8 cards, 16 dies) or two A2 Series nodes. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node deployment Quantized model `GLM-5.2-w8a8` can be deployed on one Ascend A3 Series node. Run the following script to execute online inference. ```shell theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # ============================================================ # high performance cpu echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 # bind cpu export SGLANG_SET_CPU_AFFINITY=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING # cann source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export STREAMS_PER_DEVICE=32 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 # MTP OVERLAP export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export HCCL_BUFFSIZE=1000 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME=lo export GLOO_SOCKET_IFNAME=lo # DEEPEP export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 MODEL_PATH=/path/to/model-weights python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --attention-backend ascend \ --device npu \ --tp-size 16 --nnodes 1 --node-rank 0 \ --chunked-prefill-size 16384 --max-prefill-tokens 280000 \ --trust-remote-code \ --host 127.0.0.1 \ --mem-fraction-static 0.7 \ --port 8000 \ --served-model-name glm-5 \ --cuda-graph-bs-decode 16 \ --quantization modelslim \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ --moe-a2a-backend deepep --deepep-mode auto ``` ### Multi-node deployment Quantized model `GLM-5.2-w8a8` can be deployed on two Ascend A3 Series nodes. Modify the IP addresses of the two nodes, then run the same script on both nodes. ```shell theme={null} # ============================================================ # Before running, update the following variables: # IPS: IP addresses of each node in the cluster # IP_MASTER: rank 0 node IP address with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 # bind cpu export SGLANG_SET_CPU_AFFINITY=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING # cann source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export STREAMS_PER_DEVICE=32 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 # MTP OVERLAP export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export HCCL_BUFFSIZE=1000 export HCCL_OP_EXPANSION_MODE=AIV # Run command ifconfig on two nodes, find out which inet addr has same IP with your node IP. That is your public interface, which should be added here export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= # DEEPEP export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 IPS=('' '') IP_MASTER="${IPS[0]}:5000" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` for i in "${!IPS[@]}"; do if [[ "$LOCAL_HOST1" == "${IPS[$i]}" || "$LOCAL_HOST2" == "${IPS[$i]}" ]]; then echo "${IPS[$i]}" python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --attention-backend ascend \ --device npu \ --tp-size 32 --nnodes 2 --node-rank $i --dist-init-addr $IP_MASTER \ --chunked-prefill-size 16384 --max-prefill-tokens 131072 \ --trust-remote-code \ --host 127.0.0.1 \ --mem-fraction-static 0.8 \ --port 8000 \ --served-model-name glm-5 \ --cuda-graph-max-bs-decode 32 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ --disable-radix-cache NODE_RANK=$i break fi done ``` ### Prefill-decode disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. ```shell theme={null} # ============================================================ # Before running, update the following variables: # ASCEND_MF_STORE_URL: prefill master IP address with port # P_IP: prefill node IP address # D_IP: decode node IP address # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 export SGLANG_SET_CPU_AFFINITY=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export STREAMS_PER_DEVICE=32 # pd transfer, prefill master IP export ASCEND_MF_STORE_URL="tcp://:24707" export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 P_IP=('') D_IP=('') MODEL_PATH=/path/to/model-weights export TRANSFORMERS_VERBOSITY=error LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export TASK_QUEUE_ENABLE=2 export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= # prefill node python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ --port 8000 --disaggregation-bootstrap-port 8998 --trust-remote-code --nnodes 1 --node-rank $i \ --tp-size 16 --mem-fraction-static 0.8 --attention-backend ascend --device npu --quantization modelslim \ --disaggregation-transfer-backend ascend --max-running-requests 64 \ --served-model-name glm-5 --chunked-prefill-size 524288 --max-prefill-tokens 180000 --moe-a2a-backend deepep --deepep-mode normal \ --disable-shared-experts-fusion --disable-cuda-graph --dtype bfloat16 \ --attn-dp-size 4 \ --load-balance-method round_robin \ --enable-dp-lm-head --moe-dense-tp 1 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 # cp #--enable-nsa-prefill-context-parallel \ #--nsa-prefill-cp-mode in-seq-split \ #--attn-cp-size 4 \ NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export HCCL_BUFFSIZE=650 export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export TASK_QUEUE_ENABLE=0 export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= export SGLANG_NPU_USE_MULTI_STREAM=1 python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ --port 8003 --trust-remote-code --nnodes 1 --node-rank $i --tp-size 16 --attn-dp-size 16 --ep-size 16 \ --mem-fraction-static 0.8 --max-running-requests 128 --attention-backend ascend --device npu --quantization modelslim \ --served-model-name glm-5 --moe-a2a-backend deepep --deepep-mode low_latency \ --cuda-graph-max-bs-decode 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \ --tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 NODE_RANK=$i break fi done ``` Launch the router after the prefill and decode services are ready. ```shell theme={null} # ============================================================ # Before running, update the following variables: # P_MASTER_IP: prefill master IP address # D_MASTER_IP: decode master IP address # ROUTER_HOST_IP: router node IP address # ============================================================ P_MASTER_IP="" D_MASTER_IP="" ROUTER_HOST_IP="" python3 -m sglang_router.launch_router \ --pd-disaggregation \ --policy round_robin \ --prefill http://${P_MASTER_IP}:8000 8998 \ --decode http://${D_MASTER_IP}:8003 \ --host ${ROUTER_HOST_IP} \ --port 6688 ``` ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 8000) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference GLM-5.2 has no standalone best practice page yet, as tuning parameters are not finalized. Instead, the [Online service deployment](#online-service-deployment) section above provides ready-to-use scripts for single-node, multi-node, and PD disaggregation, each embedding the recommended feature combinations and tuning parameters (e.g., DeepEP mode, speculative decoding, overlap schedule). For the full catalog of optimization features and their parameter and compatibility details, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Hy3 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3 ## Introduction Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model with 21B active parameters and 3.8B MTP layer parameters, developed by the Tencent Hy Team. It features 192 experts with top-8 activation per token, GQA attention (64 heads, 8 KV heads, head dim 128), and a 256K context length. The model supports built-in reasoning via `reasoning_effort`, tool calling, and multi-token prediction (MTP) for speculative decoding. Hy3 significantly outperforms similar-size models and rivals flagship open-source models with 2–5× the parameters, with notable improvements in agentic workflows, long-context tasks, and production reliability. This document demonstrates the deployment of Hy3 on Ascend NPUs using SGLang, including single-node (Ascend A3 Series) and multi-node (Ascend A2 Series) PD mixed mode and speculative decoding. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 16` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode`; e.g. `--cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40` | | Speculative Decoding | `--speculative-algorithm EAGLE \`
`--speculative-num-steps 2 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 3` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | Reasoning Mode | `extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}}` — deep chain-of-thought;
`"reasoning_effort": "low"` — short thinking chain;
`"reasoning_effort": "no_think"` — direct response, no thinking | | Tool Calling | `--tool-call-parser auto --reasoning-parser auto` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Hy3 has 295B total parameters. If you need to download model weights, check the model size before downloading to reserve enough space. * [Hy3](https://www.modelscope.cn/models/Tencent-Hunyuan/Hy3) (BF16, 597.60GB) The BF16 variant (597.60GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full Ascend A3 Series node (8 cards × 2 dies) or two Ascend A2 Series nodes (8 cards × 1 die each). It is recommended to download the model weights to a shared directory accessible from within the container. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=64g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=64g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. * `--shm-size=64g` is recommended for 16-NPU deployments with large batch sizes. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode). Hy3 with 295B parameters requires all 16 logical NPUs on an Ascend A3 Series server (8 physical NPUs × 2 logical cores each, 64 GB HBM per logical NPU). Set the following environment variables before launching the server: ```bash theme={null} # Performance tuning export SGLANG_SET_CPU_AFFINITY=1 echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 # CANN environment (adjust paths to match your installation) source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh # Ascend NPU settings export ASCEND_USE_FIA=1 export STREAMS_PER_DEVICE=32 export HCCL_BUFFSIZE=3000 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME=lo export GLOO_SOCKET_IFNAME=lo # SGLang settings export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 ``` Launch the server: ```bash theme={null} MODEL_PATH="/path/to/Hy3" python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --attention-backend ascend \ --reasoning-parser auto \ --tool-call-parser auto \ --device npu \ --tp-size 16 \ --host 0.0.0.0 \ --port 9999 \ --mem-fraction-static 0.84 \ --dtype bfloat16 \ --base-gpu-id 0 \ --prefill-max-requests 40 \ --max-running-requests 40 \ --cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 ``` ### Multi-node PD mixed deployment on Ascend A2 Series Each Ascend A2 Series node provides 8 NPUs (64 GB HBM per NPU). Hy3 requires all 16 logical NPUs across **2 × Ascend A2 Series** nodes in PD mixed mode (prefill and decode are colocated on the same 2-node cluster). On each node, set the following environment variables before launching the server: ```bash theme={null} # ============================================================ # Before running, update the following variables: # HCCL_SOCKET_IFNAME: network interface name for HCCL (use `ifconfig` to find) # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ # Performance tuning export SGLANG_SET_CPU_AFFINITY=1 echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 # CANN environment (adjust paths to match your installation) source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh # Ascend NPU settings export ASCEND_USE_FIA=1 export STREAMS_PER_DEVICE=32 export HCCL_BUFFSIZE=3000 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= # SGLang settings export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 ``` Launch the server on both nodes by running the same script on each node. `NODE_IPS[0]` is the master node and must be reachable from the other node: ```bash theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory (shared) # NODE_IPS: IP addresses of the two nodes, e.g. ('10.0.0.1' '10.0.0.2') # ============================================================ MODEL_PATH="/path/to/Hy3" NODE_IPS=('' '') for i in "${!NODE_IPS[@]}"; do if [[ "$(hostname -I | awk '{print $1}')" == "${NODE_IPS[$i]}" ]]; then python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --attention-backend ascend \ --reasoning-parser auto \ --tool-call-parser auto \ --device npu \ --tp-size 16 \ --nnodes 2 \ --dist-init-addr ${NODE_IPS[0]}:5000 \ --node-rank $i \ --host 0.0.0.0 \ --port 9999 \ --mem-fraction-static 0.84 \ --dtype bfloat16 \ --base-gpu-id 0 \ --prefill-max-requests 40 \ --max-running-requests 40 \ --cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 break fi done ``` * `HCCL_SOCKET_IFNAME` and `GLOO_SOCKET_IFNAME` must be set to the cluster network interface on both nodes. Do **not** use `lo` for multi-node deployments. * Model weights must be accessible from both nodes, e.g. mounted from a shared directory. * Send requests only to the master node (`NODE_IPS[0]:9999`), not to the worker node. ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 9999) # ============================================================ curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Hy3", "messages": [{"role": "user", "content": "What is the capital of France?"}], "max_tokens": 64, "temperature": 0.9, "extra_body": {"chat_template_kwargs": {"reasoning_effort": "no_think"}} }' ``` Expected result: an HTTP 200 response with `"Paris"` in the content field. To enable chain-of-thought reasoning for math or coding tasks, set `reasoning_effort` to `"high"`: ```shell theme={null} curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Hy3", "messages": [{"role": "user", "content": "Solve: if 3x + 7 = 22, what is x?"}], "max_tokens": 8192, "temperature": 0.9, "extra_body": {"chat_template_kwargs": {"reasoning_effort": "high"}} }' ``` The response will contain a `reasoning_content` field with the thinking process and a `content` field with the final answer. Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference Hy3 has no standalone best practice page yet. The [Single-node online deployment](#single-node-online-deployment) and [Multi-node PD mixed deployment on Ascend A2 Series](#multi-node-pd-mixed-deployment-on-ascend-a2-series) sections above provide the ready-to-use scripts for the supported PD mixed cases. For the full catalog of optimization features and their parameter and compatibility details, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ### Reasoning mode selection Hy3 supports three reasoning modes via `chat_template_kwargs`: | `reasoning_effort` | Behavior | Recommended for | | - | - | - | | `no_think` | No chain-of-thought, direct reply | Simple Q\&A, low-latency tasks | | `low` | Short thinking chain | Moderate reasoning tasks | | `high` | Full chain-of-thought | Math, coding, complex reasoning | Use `no_think` for interactive or latency-sensitive applications to reduce output token count significantly. ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). This section only covers model-specific issues. # Kimi-K2.6 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6 ## Introduction Kimi-K2.6 is an open-source, native multimodal agentic model developed by Moonshot AI, built through continual pretraining on approximately 15 trillion mixed visual and text tokens atop Kimi-K2-Base. It is a Mixture-of-Experts (MoE) model featuring Multi-head Latent Attention (MLA) and MoE architecture, with 1T total parameters and 32B active parameters. The model seamlessly integrates vision and language understanding with advanced agentic capabilities, supporting both instant and thinking modes as well as conversational and agentic paradigms. This document demonstrates the deployment of Kimi-K2.6 on Ascend NPUs using SGLang, including single-node PD mixed mode, multi-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Kimi-K2.6) is fully supported in this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 12 16 24 32 48 64 96 120` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 4 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 5 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | | MLAPO | `export SGLANG_NPU_USE_MLAPO=1` | | Multistream MoE | `export SGLANG_NPU_USE_MULTI_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Kimi-K2.6](https://www.modelscope.cn/models/moonshotai/Kimi-K2.6) (BF16, 595.21GB) * [Kimi-K2.6-w4a8](https://www.modelscope.cn/models/Eco-Tech/Kimi-K2.6-w4a8) (W4A8 quantized version, 535.91GB) * [kimi-k2.6-eagle3](https://www.modelscope.cn/models/lightseekorg/kimi-k2.6-eagle3) (EAGLE3 draft model for speculative decoding) * You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize `Kimi-K2.6-w4a8` from `Kimi-K2.6`. We recommend deploying the W4A8 variant for reduced resource usage and higher throughput. It (535.91GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 Series node (8 cards, 16 dies) or two A2 Series nodes. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Kimi K2.6 Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#single-node-pd-mixed). ### Multi-node online deployment Multi-node deployment distributes the model across multiple Ascend A3 Series nodes using tensor parallelism while keeping prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single node can provide. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Kimi-K2.6 Best Practice — Multi-node On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#multi-node-pd-mixed). ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Kimi-K2.6 Best Practice — PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#pd-disaggregation). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". For multimodal requests (text + image): ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Kimi-K2.6-w4a8", "messages": [ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image."} ] } ] }' ``` Expected result: an HTTP 200 response with a description of the image. Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Kimi-K2.6 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # MiMo-V2-Flash Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash ## Introduction MiMo-V2-Flash is a Mixture-of-Experts (MoE) large language model developed by Xiaomi. It employs advanced architecture with speculative decoding capabilities for accelerated inference. The model is optimized for high throughput and low latency scenarios through PD disaggregation deployment. This document demonstrates the deployment of MiMo-V2-Flash on Ascend NPUs using SGLang, including multi-node PD disaggregation mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (MiMo-V2-Flash) is fully supported in this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 8` (prefill) or `--tp-size 16` (decode) | | Data Parallelism | `--dp-size 2` | | Expert Parallelism | `--moe-a2a-backend deepep \`
`--deepep-mode low_latency` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32` | | Speculative Decoding | `--speculative-algorithm EAGLE \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--enable-multi-layer-eagle` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0` | | DP LM Head | `--enable-dp-lm-head` | | DP Attention | `--attn-dp-size 2` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [MiMo-V2-Flash-W8A8](https://www.modelscope.cn/models/iridiumine/MiMo-V2-Flash-W8A8) (Quantized version, 311.5GB) The W8A8 variant (311.5GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 Series node or 8 dies on A3 Series (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [MiMo-V2-Flash Best Practice — W8A8 24P PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash#pd-disaggregation). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 9903) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [MiMo-V2-Flash Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # MiniMax-M2.5 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5 ## Introduction MiniMax-M2.5 is a Mixture-of-Experts (MoE) large language model developed by MiniMax, featuring a sparse MoE architecture with approximately 230B total parameters and 10B active parameters. It supports native long-context processing up to 200k tokens. The model supports EAGLE3 speculative decoding with a custom eagle model for accelerated inference, and excels at general language understanding, reasoning, and long-context tasks. This document demonstrates the deployment of MiniMax-M2.5 on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (MiniMax-M2.5) is fully supported in this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend ascend_fuseep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 8 16 24 32 48 64 80` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. * [MiniMax-M2.5-w8a8-QuaRot](https://www.modelscope.cn/models/Eco-Tech/MiniMax-M2.5-w8a8-QuaRot) (W8A8 quantized version, 230.82GB) * [MiniMax-M2.5-eagle-model](https://www.modelscope.cn/models/sgl-npu/MiniMax-M2.5-eagel-model-0318) (EAGLE3 draft model for speculative decoding) The W8A8 variant (230.82GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 Series node or 8 dies on A3 Series (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [MiniMax-M2.5 Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [MiniMax-M2.5 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3-235B-A22B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b ## Introduction Qwen3-235B-A22B is a Mixture-of-Experts (MoE) large language model developed by Alibaba, featuring 235B total parameters with 22B active parameters. It employs Grouped-Query Attention (GQA) and Qwen3MoE architecture, with support for EAGLE3 speculative decoding for accelerated inference. The model excels at instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage, available in both standard and thinking/reasoning-enhanced editions. This document demonstrates the deployment of Qwen3-235B-A22B on Ascend NPUs using SGLang, including single-node PD mixed mode, multi-node PD disaggregation mode, 256k long-sequence inference, Prefill Context Parallel, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-235B-A22B) is fully supported in this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend ascend_fuseep` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 94208` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 16 20 24 26 27` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | | Context Parallelism | `--enable-prefill-context-parallel \`
`--attn-cp-size 2 \`
`--moe-dp-size 2` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Qwen3-235B-A22B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-235B-A22B-Instruct-2507) (BF16, 470.21GB) * [Qwen3-235B-A22B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-235B-A22B-W8A8) (W8A8 quantized version, 236.80GB) * [Qwen3-235B-A22B-Eagle3](https://www.modelscope.cn/models/nv-community/Qwen3-235B-A22B-Eagle3) (EAGLE3 draft model for speculative decoding) The BF16 variant (470.21GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 Series node (8 cards, 16 dies) or two A2 Series nodes. The W8A8 variant (236.80GB) can be deployed on 8 × 64GB (`--tp-size 8`), which corresponds to one full A2 Series node or 8 dies on A3 Series (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3-235B-A22B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b#single-node-pd-mixed). ### Multi-node PD disaggregation deployment #### 256K long-sequence PD disaggregation on 2 x Ascend A3 Series nodes (without CP) This configuration uses PD disaggregation for **256K long-sequence inference** on 2 x Ascend A3 Series nodes with context parallel disabled. The following command is based on the **W8A8** quantized model. 1. Set the shared environment variables on both prefill and decode nodes: ```bash Shared environment theme={null} #============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # PREFILL_HOST_IP: prefill node IP address # NETWORK_IFACE: network interface name (use ifconfig to find) #============================================================ export ASCEND_USE_FIA=1 export SGLANG_SET_CPU_AFFINITY=1 export ASCEND_MF_STORE_URL="tcp://:12345" export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= MODEL_PATH=/path/to/model-weights ``` 2. Run on the **prefill node**: ```bash Prefill node theme={null} #============================================================ # Before running, update the following variable: # PREFILL_HOST_IP: prefill node IP address #============================================================ export ASCEND_LAUNCH_BLOCKING=1 export HCCL_BUFFSIZE=1500 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_LONG_SEQ_ROUND=128 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8995 \ --attention-backend ascend \ --disable-radix-cache \ --chunked-prefill-size -1 \ --skip-server-warmup \ --device npu \ --quantization modelslim \ --tp-size 16 \ --mem-fraction-static 0.45 \ --max-running-requests 1 \ --host \ --port 8000 \ --dist-init-addr :5000 \ --nnodes 1 \ --node-rank 0 \ --moe-a2a-backend deepep \ --deepep-mode normal ``` 3. Run on the **decode node**: ```bash Decode node theme={null} #============================================================ # Before running, update the following variable: # DECODE_HOST_IP: decode node IP address #============================================================ export HCCL_BUFFSIZE=4000 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=16 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --disaggregation-transfer-backend ascend \ --attention-backend ascend \ --mem-fraction-static 0.8 \ --disable-cuda-graph \ --device npu \ --quantization modelslim \ --disable-radix-cache \ --chunked-prefill-size 8192 \ --skip-server-warmup \ --tp-size 16 \ --max-running-requests 1 \ --host \ --port 8232 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --disable-overlap-schedule ``` 4. Launch the SGLang Router (on any reachable node): ```bash Router theme={null} #============================================================ # Before running, update the following variables: # PREFILL_HOST_IP: prefill node IP address # DECODE_HOST_IP: decode node IP address # ROUTER_HOST_IP: router node IP address #============================================================ python3 -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8995 \ --decode http://:8232 \ --host \ --port 6689 \ --prometheus-port 29010 ``` #### Prefill Context Parallel (PCP) on 2 x Ascend A3 Series nodes This configuration enables **Prefill Context Parallel** (`--enable-prefill-context-parallel`) to split the context across CP ranks during prefill, reducing per-device memory pressure and improving TTFT for long sequences. PD disaggregation is required. The following command is based on the **W8A8** quantized model. **Constraints:** * Prefill side must set `--max-running-requests 1` (PCP only supports batch\_size=1) * `--attn-cp-size` must evenly divide `--tp-size`; each CP rank occupies `tp_size / cp_size` NPUs 1. Run on the **prefill node**: ```bash Prefill node theme={null} #============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # PREFILL_HOST_IP: prefill node IP address #============================================================ export SGLANG_SET_CPU_AFFINITY=1 export ASCEND_MF_STORE_URL="tcp://:23456" export ASCEND_USE_FIA=True python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --trust-remote-code \ --disaggregation-mode prefill \ --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8995 \ --quantization modelslim \ --attention-backend ascend \ --skip-server-warmup \ --mem-fraction-static 0.7 \ --chunked-prefill-size 32768 \ --device npu \ --base-gpu-id 0 \ --tp-size 16 \ --enable-prefill-context-parallel \ --attn-cp-size 2 \ --moe-dp-size 2 \ --max-running-requests 1 \ --host \ --port 8000 \ --nnodes 1 \ --node-rank 0 \ --dist-init-addr :6688 ``` Key parameters for PCP: | Parameter | Value | Description | | - | - | - | | `--enable-prefill-context-parallel` | flag | Enable PCP feature | | `--attn-cp-size` | 2 | Split context across 2 CP ranks (each rank handles half the sequence) | | `--moe-dp-size` | 2 | MoE DP size, should match `--attn-cp-size` | | `--max-running-requests` | 1 | Required by PCP (batch\_size=1 constraint) | 2. Run on the **decode node**: ```bash Decode node theme={null} #============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DECODE_HOST_IP: decode node IP address # PREFILL_HOST_IP: prefill node IP address (for ASCEND_MF_STORE_URL) #============================================================ export ASCEND_MF_STORE_URL="tcp://:23456" export ASCEND_USE_FIA=True python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --trust-remote-code \ --disaggregation-mode decode \ --disaggregation-transfer-backend ascend \ --quantization modelslim \ --attention-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ --mem-fraction-static 0.7 \ --chunked-prefill-size 32768 \ --skip-server-warmup \ --device npu \ --base-gpu-id 0 \ --tp-size 8 \ --max-running-requests 32 \ --host \ --port 8001 \ --nnodes 1 \ --node-rank 0 \ --dist-init-addr :6688 ``` `ASCEND_MF_STORE_URL` on both nodes must point to the same KV store (typically the prefill node IP). `ASCEND_USE_FIA=True` enables fast interconnect aggregation for KV transfer. PCP is a prefill-only feature; the decode side needs no CP-related flags. ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3-235B-A22B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3-30B-A3B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b ## Introduction Qwen3-30B-A3B is a Mixture-of-Experts (MoE) large language model developed by Alibaba, featuring 30B total parameters with 3B active parameters. It employs Grouped-Query Attention (GQA) and Qwen3MoE architecture, with support for EAGLE3 speculative decoding for accelerated inference. The model excels at instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage, available in both standard and thinking/reasoning-enhanced editions. This document demonstrates the deployment of Qwen3-30B-A3B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-30B-A3B) is fully supported in this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 2` | | Data Parallelism | `--dp-size 2` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Qwen3-30B-A3B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-30B-A3B-Instruct-2507) (BF16, 61.08GB, recommended) * [Qwen3-30B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3-30B-A3B-w8a8) (W8A8 quantized version, 31.29GB) * [Qwen3-a3B\_eagle3](https://www.modelscope.cn/models/vllm-ascend/Qwen3-a3B_eagle3) (EAGLE3 draft model for speculative decoding) Both variants fit within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 Series or A3 Series. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3-30B-A3B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3-30B-A3B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3-32B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b ## Introduction Qwen3-32B is a dense model in the Qwen3 series developed by Alibaba, featuring 32B parameters with Grouped-Query Attention (GQA) and up to 128k (131k with YaRN) context length. It delivers significant improvements in instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage. The model supports EAGLE3 speculative decoding for accelerated inference and supports both standard and thinking/reasoning modes. This document demonstrates the deployment of Qwen3-32B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-32B) is fully supported in this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 4` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 4 8 16` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Qwen3-32B](https://www.modelscope.cn/models/Qwen/Qwen3-32B) (BF16, 65.54GB) * [Qwen3-32B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-32B-W8A8) (W8A8 quantized version, 42.77GB) * [Eagle3-Qwen3-32B-zh](https://www.modelscope.cn/models/Zjcxy-SmartAI/Eagle3-Qwen3-32B-zh) (EAGLE3 draft model for speculative decoding) The BF16 variant (65.54GB) can be deployed on 2 × 64GB of device memory (`--tp-size 2`), which corresponds to 2 cards on A2 Series or 1 card (2 dies) on A3 Series. The W8A8 variant (42.77GB) fits within a single die. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3-32B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3-32B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3.5-397B-A17B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b ## Introduction Qwen3.5-397B-A17B is the latest flagship model in the Qwen series developed by Alibaba, featuring a Gated Delta Networks combined with sparse Mixture-of-Experts architecture (397B total parameters, 17B activated). It employs hybrid attention with Gated Delta Networks (linear, O(n) complexity) combined with full attention every 4th layer, and MoE routing with Top-10 active out of 512 routed experts plus a dedicated shared expert. The model supports multimodal inputs (text, image, video) with native context lengths of up to 262,144 tokens, and includes built-in multi-token prediction (MTP) for speculative decoding. This document demonstrates the deployment of Qwen3.5-397B-A17B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3.5-397B-A17B) is fully supported in this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 8` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp) (W4A8 quantized version with MTP, 235.88GB) The W4A8 variant (235.88GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 Series node or 8 dies on A3 Series (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3.5-397B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". For multimodal requests (text + image): ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3.5-397B-A17B-w4a8-mtp", "messages": [ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image."} ] } ] }' ``` Expected result: an HTTP 200 response with a description of the image. Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3.5-397B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3.6-27B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b ## Introduction Qwen3.6-27B is a dense model in the Qwen3.6 series developed by Alibaba, featuring 27B parameters with a hybrid architecture combining Gated Delta Networks (linear, O(n) complexity) with full attention every 4th layer. It supports multimodal inputs (text, image, video) with native context lengths of up to 262,144 tokens, and includes built-in multi-token prediction (MTP) for speculative decoding. The model delivers strong performance in instruction following, reasoning, text comprehension, and tool usage. This document demonstrates the deployment of Qwen3.6-27B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3.6-27B) is fully supported in this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 2` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 8 16 32 48` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Eco-Tech/Qwen3.6-27B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8) (W8A8 quantized version, 36.45GB) The W8A8 variant (36.45GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 Series or A3 Series. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3.6-27B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". For multimodal requests (text + image): ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3.6-27B-w8a8", "messages": [ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image."} ] } ] }' ``` Expected result: an HTTP 200 response with a description of the image. Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3.6-27B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3.6-35B-A3B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b ## Introduction Qwen3.6-35B-A3B is a Mixture-of-Experts model in the Qwen3.6 series developed by Alibaba, featuring 35B total parameters with 3B active parameters per token. It uses a hybrid architecture that combines Gated DeltaNet (linear, O(n) complexity) with full attention applied every Nth layer, enabling efficient long-context modeling. The model supports multimodal inputs (text, image, video) with long context lengths, and includes built-in multi-token prediction (NEXTN) for speculative decoding. It delivers strong performance in instruction following, reasoning, text comprehension, and tool usage. This document demonstrates the deployment of Qwen3.6-35B-A3B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3.6-35B-A3B) is fully supported in this version. To use the latest features (e.g., speculative decoding / NEXTN, multimodal), use **v0.5.16 or a later version**. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 2` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 4 8 16 24 32 48 64 80 96 112 120` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Eco-Tech/Qwen3.6-35B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-35B-A3B-w8a8) (W8A8 quantized version, 39.81GB) The W8A8 variant (39.81GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 Series or A3 Series. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3.6-35B-A3B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". For multimodal requests (text + image): ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3.6-35B-A3B-w8a8", "messages": [ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image."} ] } ] }' ``` Expected result: an HTTP 200 response with a description of the image. Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3.6-35B-A3B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3.8-Max Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8_max ## Introduction Qwen3.8-Max (published as Qwen3.8-2.4T-A95B) is a Mixture-of-Experts (MoE) model with 2.4 trillion total parameters and 95B activated per token. It uses a 92-layer hybrid architecture that combines Gated Delta Network (GDN) linear-attention with full-attention layers. The model has a native context length of 262,144 tokens (extensible to over 1 million) and built-in multi-token prediction (MTP) weights for speculative decoding. This document demonstrates the deployment of Qwen3.8-Max on Ascend NPUs using SGLang, including multi-node PD mixed mode, feature configuration, and performance optimization. Qwen3.8-Max is newly released. This document is validated and written based on the **SGLang main branch (daily build)**. It is recommended to use the latest daily build Docker image, or build SGLang from the main branch source. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 64` | | DP Attention | `--attn-dp-size 4` | | Expert Parallelism | `--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 16` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Qwen/Qwen3.8-2.4T-A95B](https://www.modelscope.cn/models/Qwen/Qwen3.8-2.4T-A95B) (BF16, approximately 4.8TB) * You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize a W4A8 variant from the BF16 checkpoint. We recommend deploying the W4A8 variant for reduced resource usage and higher throughput. The validated configuration in this tutorial deploys the W4A8 variant on 4 Ascend A3 Series nodes (`--tp-size 64`, 64 dies in total). The BF16 checkpoint alone weighs approximately 4.8TB and requires additional nodes. For the hardware specifications (memory per die, dies per card), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. The following command is based on the daily build tag, which contains the latest SGLang main branch changes. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:main-cann9.0.0-a3 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:main-cann9.0.0-a3 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Multi-node online deployment Multi-node deployment distributes the model across multiple Ascend A3 Series nodes using tensor parallelism while keeping prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single node can provide. The validated configuration deploys the W4A8 quantized checkpoint on 4 nodes with `--tp-size 64` (16 dies per node), DP attention (`--attn-dp-size 4`), and DeepEP in `auto` mode. Modify the IP addresses of the four nodes, then run the same script on all four nodes. Each node determines its own rank by matching the local IP address against `IPS`. ```shell theme={null} # ============================================================ # Before running, update the following variables: # IPS: IP addresses of each node in the cluster # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ # high performance cpu echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=10 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 # bind cpu export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ONE_VISIBLE_DEVICE_PER_PROCESS=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING # cann source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export STREAMS_PER_DEVICE=32 # deepep export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export HCCL_BUFFSIZE=2300 export DEEPEP_NORMAL_LONG_SEQ_ROUND=64 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=512 export HCCL_OP_EXPANSION_MODE=AIV # Run command ifconfig on each node, find out which inet addr has same IP with your node IP. That is your public interface, which should be added here export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= IPS=('' '' '' '') IP_MASTER="${IPS[0]}:5000" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` for i in "${!IPS[@]}"; do if [[ "$LOCAL_HOST1" == "${IPS[$i]}" || "$LOCAL_HOST2" == "${IPS[$i]}" ]]; then echo "${IPS[$i]}" # overlap schedule export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_RAGGED_VERIFY_MODE=static sglang serve \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --dist-init-addr $IP_MASTER --nnodes 4 --node-rank $i \ --model-path $MODEL_PATH \ --tokenizer-path $MODEL_PATH \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --dtype bfloat16 \ --tp-size 64 \ --attn-dp-size 4 --enable-dp-lm-head \ --mem-fraction-static 0.8 \ --chunked-prefill-size 8192 \ --cuda-graph-bs-decode 16 \ --disable-radix-cache \ --max-running-requests 64 \ --host 0.0.0.0 \ --port 30000 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --watchdog-timeout 9000 break fi done ``` ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 30000) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference Qwen3.8-Max has no standalone best practice page yet, as tuning parameters are not finalized. Instead, the [Online service deployment](#online-service-deployment) section above provides a ready-to-use script for multi-node deployment, which embeds the recommended feature combinations and tuning parameters (e.g., DP attention, DeepEP, overlap schedule). For the full catalog of optimization features and their parameter and compatibility details, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3-8B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b ## Introduction Qwen3-8B is a compact dense model in the Qwen3 series developed by Alibaba, featuring 8B parameters with Grouped-Query Attention (GQA) and up to 128k context length. It delivers significant improvements in instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage. The model supports EAGLE3 speculative decoding for accelerated inference and is available in both standard and thinking/reasoning-enhanced editions. This document demonstrates the deployment of Qwen3-8B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-8B) is fully supported in this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 2` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 6 9 10 15 16` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Qwen3-8B](https://www.modelscope.cn/models/Qwen/Qwen3-8B) (BF16, 16.40GB) * [Qwen3-8B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-8B-w8a8) (W8A8 quantized version, 11.27GB) * [Eagle3-Qwen3-8B-zh](https://www.modelscope.cn/models/Zjcxy-SmartAI/Eagle3-Qwen3-8B-zh) (EAGLE3 draft model for speculative decoding) We recommend deploying the W8A8 variant for reduced resource usage and higher throughput. It (11.27GB) fits within a single 64GB die, so `--tp-size 1` is sufficient on either A2 Series or A3 Series. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3-8B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3-8B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3-Next-80B-A3B-Instruct Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct ## Introduction Qwen3-Next-80B-A3B-Instruct is a hybrid Gated DeltaNet-Transformer Mixture-of-Experts (MoE) model developed by Alibaba, featuring 80B total parameters with 3B active parameters. It combines Gated DeltaNet SSM layers with attention layers in a sparse MoE architecture, enabling efficient long-sequence modeling with linear complexity in the SSM layers. The model supports multi-token prediction (NEXTN) for speculative decoding, delivering strong performance in instruction following, reasoning, and text generation tasks. This document demonstrates the deployment of Qwen3-Next-80B-A3B-Instruct on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-Next-80B-A3B-Instruct) is fully supported in this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | - | - | | Tensor Parallelism | `--tp-size 4` | | Data Parallelism | `--dp-size 2` | | Expert Parallelism | `--ep-size 4 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 4 8` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant \`
`--speculative-draft-model-path /path/to/draft-model-weights` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Qwen3-Next-80B-A3B-Instruct](https://www.modelscope.cn/models/Qwen/Qwen3-Next-80B-A3B-Instruct) (BF16, 162.68GB) — used as the EAGLE3 draft model * [Qwen3-Next-80B-A3B-Instruct-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-Next-80B-A3B-Instruct-W8A8) (W8A8 quantized version, 84.90GB) We recommend deploying the W8A8 variant as the main model for reduced resource usage and higher throughput. It (84.90GB) can be deployed on 2 × 64GB (`--tp-size 2`), which corresponds to 2 cards on A2 Series or 1 card (2 dies) on A3 Series. The BF16 weights serve as the EAGLE3 draft model (set `--speculative-draft-model-path` to the BF16 weight path). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3-Next-80B-A3B-Instruct Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3-Next-80B-A3B-Instruct Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Parameter Tuning Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning This guide explains the role of each parameter used in SGLang deployments on Ascend NPU. It uses the [DeepSeek-V3.2 best practice configuration](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2#pd-disaggregation) as the reference example. For a complete list of tested deployment configurations, see the [Ascend NPU Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) page. Parameters in this guide fall into two categories: * **Required configurations** (marked with `[Required]`): These must be set correctly for the target deployment scenario (e.g., multi-node communication, PD disaggregation). Incorrect values will cause deployment failures or incorrect behavior. * **Performance optimizations**: These improve throughput, latency, or memory efficiency. The optimal values depend on your specific model, hardware, and workload and may require tuning. Where the optimal value is not obvious, tuning guidance is provided. ## System-Level Optimizations The following system-level tuning steps reduce OS interference and improve CPU scheduling determinism:
Command / Variable Purpose
`echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor` Locks all CPU cores to the maximum frequency, eliminating DVFS-induced latency jitter during inference-critical paths.
`sysctl -w vm.swappiness=0` Minimizes kernel swapping of anonymous pages. Reduces the risk of page faults on NPU memory buffers pinned to host RAM.
`sysctl -w kernel.numa_balancing=0` Disables automatic NUMA page migration. Prevents the kernel from moving memory pages between NUMA nodes while inference is running, which would cause latency spikes.
`sysctl -w kernel.sched_migration_cost_ns=50000` Sets a minimum task migration cost, discouraging the scheduler from moving inference threads between CPU cores unnecessarily.
`SGLANG_SET_CPU_AFFINITY=1` Binds SGLang worker processes to specific CPU cores, avoiding cross-core migration overhead for high-frequency scheduling loops.
## Memory & Device Configuration
Variable / Argument Purpose Reference Value
`PYTORCH_NPU_ALLOC_CONF=expandable_segments:True` Enables the expandable NPU memory allocator, allowing the memory pool to grow dynamically. This avoids out-of-memory errors when workloads have variable memory requirements and is essential for large models such as MoE architectures. `expandable_segments:True`
`STREAMS_PER_DEVICE=32` Sets the maximum number of parallel streams per NPU device. More streams allow better overlap between compute and communication operations. The default of `32` is sufficient for most deployments; increase only if profiling reveals stream contention in complex pipeline parallelism setups. `32`
`--mem-fraction-static` Controls the fraction of NPU memory allocated to model weights and the KV cache pool. Lower values leave headroom for intermediate activations; higher values maximize KV cache capacity for serving more concurrent requests. The optimal value depends on your model size, sequence length, and available NPU memory. Start conservatively and increase gradually while monitoring for out-of-memory errors. Prefill: `0.73`, Decode: `0.79`
## Communication Configuration
Variable Purpose Reference Value
`HCCL_BUFFSIZE` Sets the HCCL communication buffer size in MB. Larger buffers increase throughput for bulk transfers but consume more host memory. The optimal value depends on your communication pattern — larger values benefit prefill (bulk token transfers), while smaller values are sufficient for decode (small batches). Tune based on your expected token dispatch volume. Prefill: `1200`, Decode: `400`
`HCCL_SOCKET_IFNAME` / `GLOO_SOCKET_IFNAME` \[Required for multi-node] Specifies the network interface used for HCCL and GLOO distributed communication. Must be set to the high-bandwidth inter-node network interface (e.g., RDMA-capable NIC) for multi-node deployments. Without this, the framework may default to a low-bandwidth interface, severely degrading distributed communication performance. Not needed for single-node deployments. Set per-cluster
## MoE & Expert Parallelism
Variable / Argument Purpose Reference Value
`--moe-a2a-backend` Selects the all-to-all communication backend for MoE expert dispatch and combine. On Ascend NPU, the primary options are `deepep` (DeepEP) and `ascend_fuseep` (Ascend Fused EP). DeepEP is optimized for large-scale models with flexible prefill/decode dispatch paths; `ascend_fuseep` provides a general fused MoE dispatch path. `deepep`
`--deepep-mode` Selects the DeepEP operating mode. Available options: `normal` (optimized for high throughput, long sequences, and large token counts — suitable for prefill), `low_latency` (optimized for low latency, CUDA Graph compatible, small batches — suitable for decode), and `auto` (switches automatically based on the operation type). Use `auto` if unsure; use explicit modes when managing prefill/decode independently in PD disaggregation. Prefill: `normal`, Decode: `low_latency`
`SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK` Sets the maximum number of tokens that a single rank can dispatch in one DeepEP operation (hard upper limit: `1024`). Larger values accommodate more tokens per dispatch but increase buffer allocation overhead. For prefill, use a large value or unbounded (`0`) since many tokens are processed. For decode, set to match your expected tokens per iteration. Must satisfy: `max-running-requests * (1 + draft_tokens) <= this value`. Prefill: `0` (unbounded), Decode: `8`
`DEEP_NORMAL_MODE_USE_INT8_QUANT` When set to `1`, quantizes intermediate activations to INT8 in the DeepEP dispatch operator, reducing communication volume during MoE dispatch. This is most beneficial for large-scale multi-node prefill with many tokens. The trade-off is a small accuracy impact from quantization and additional compute for the quantize/dequantize operations. Prefill: `1`
`TASK_QUEUE_ENABLE` Controls the ASCEND Runtime task queue optimization level: `0` = disabled, `1` = default optimization, `2` = aggressive optimization with greater task fusion and overlap. Higher levels improve throughput but may interfere with NPU Graph-launched tasks. Start with `1` for general use. Use `2` for throughput-critical prefill workloads; use `0` for decode where NPU Graph compatibility is needed. Prefill: `2`, Decode: `0`
`--moe-dense-tp-size` Sets the tensor parallelism size for MoE dense (shared) MLP layers. When using DP attention, setting this to `1` avoids an unnecessary all-reduce across the DP group for the dense MLP layers, since each DP shard already has the full weight. In deployments without DP attention, set this to match your TP size. `1`
## Prefill Optimizations These arguments and environment variables are critical for tuning prefill performance:
Argument / Variable Purpose Reference Value
`--chunked-prefill-size` Sets the maximum number of tokens per prefill chunk. A positive value enables chunked prefill, which interleaves prefill and decode for better concurrency in mixed workloads. Set to `-1` to disable chunking and process each request in a single forward pass, which is preferred for dedicated prefill servers with long-context sequences. `-1`
`--max-prefill-tokens` Limits the total number of tokens the prefill server can process in one batch. The effective bound is `max(this value, model_max_context_length)`. Set this based on your target sequence length and available NPU memory to bound memory usage while maximizing throughput. Tune by increasing until you encounter out-of-memory errors. `68000`
`--max-running-requests` Limits the number of concurrent requests being processed. For prefill, a low value (e.g., `1`) dedicates more compute and memory to each request, achieving higher per-request throughput — ideal for dedicated prefill nodes processing long sequences. For general-purpose serving, use a higher value to support multi-request concurrency. `1`
`--disable-radix-cache` Disables prefix caching via RadixAttention. Set this flag when processing non-overlapping long sequences where prefix caching provides no benefit and only consumes memory. Leave unset (radix cache enabled) for chat/conversation workloads with shared system prompts. true
`--disable-cuda-graph` Disables CUDA Graph capture. CUDA Graphs reduce kernel launch overhead for small, predictable batch sizes, making them ideal for decode. For prefill with large and variable batch sizes, CUDA Graphs provide minimal benefit and can cause issues with dynamic shapes. Set this flag on prefill nodes; leave unset on decode nodes. true
`--enable-dsa-prefill-context-parallel` (DeepSeek V3.2 DSA-specific) Enables context parallelism for the long-sequence prefill phase of DeepSeek V3.2 with DSA (DeepSeek Sparse Attention). Distributes the sequence across CP ranks to parallelize the computationally expensive DSA prefill for ultra-long contexts. Enabled
`--dsa-prefill-cp-mode` (DeepSeek V3.2 DSA-specific) Controls how the long sequence is split across context parallel ranks: `in-seq-split` divides each sequence uniformly across CP ranks, optimal for single-request prefill. `round-robin-split` (code default) distributes tokens by index mod CP size, supporting multi-batch prefill. Only effective when `--enable-dsa-prefill-context-parallel` is enabled. `in-seq-split`
`--attn-cp-size` Specifies the context parallelism group size for attention computation. Larger values distribute the sequence across more ranks, reducing per-rank memory and compute at the cost of increased communication. For models with DSA, this controls the CP size for sparse attention prefill. Set to the number of available devices for maximum parallelization. `32`
## Decode Optimizations These arguments and environment variables are critical for tuning decode performance:
Argument / Variable Purpose Reference Value
`--attn-dp-size` / `--attention-data-parallel-size` Sets the attention DP size for the decode server: the number of data-parallel attention groups inside the TP group. This is a key optimization for decode throughput — attention is DP-sharded to reduce KV cache duplication, while FFN/MoE layers remain TP-sharded to leverage expert parallelism. Best suited for MoE models where attention is not the compute bottleneck. Higher values create more independent decode instances, increasing throughput through parallel request processing. Choose a value that divides the TP size evenly. `8`
`--ep` / `--ep-size` / `--expert-parallel-size` Sets the expert parallelism degree. For MoE models, this distributes experts across cards, reducing per-card expert loading overhead and enabling all-to-all dispatch. The code default is `1`; set explicitly for MoE models. The optimal value depends on your model's expert count and architecture. DeepSeek V3.2 with 256 routed experts uses `ep=32`. For models with fewer experts, use a proportionally smaller value. `32`
`--enable-dp-lm-head` Enables vocabulary parallelism across the DP attention group, sharding the LM head weight across ranks. Each rank only computes logits for its vocabulary shard, avoiding a costly all-gather of logits across the DP group. This is essential when DP attention is enabled to maintain throughput. Enabled
`--cuda-graph-max-bs-decode` Caps the maximum batch size for which CUDA Graphs are captured. Larger values cover more batch sizes but increase graph capture time and memory overhead. If your `max-running-requests` is high but typical batch sizes are lower, use a smaller value to reduce capture overhead. Tune based on your observed batch size distribution during serving. `4`
`SGLANG_SCHEDULER_SKIP_ALL_GATHER=1` When DP attention is enabled, the scheduler normally performs an all-gather across DP ranks to determine the full set of ready requests. Setting this to `1` skips that operation, reducing decode scheduling latency. Only safe when load is balanced across DP ranks (e.g., via a round-robin load balancing policy). Disable if you observe uneven load distribution across DP ranks. `1`
## Speculative Decoding Speculative decoding reduces per-token latency by generating draft tokens that are then verified by the target model:
Argument / Variable Purpose Reference Value
`--speculative-algorithm` Selects the speculative decoding algorithm. `NEXTN` (aliased to `EAGLE`) uses the model's built-in MTP (Multi-Token Prediction) heads, requiring no separate draft model. `EAGLE3` uses an external draft model, which can achieve higher acceptance rates at the cost of additional memory. Other built-in options include `STANDALONE`, `NGRAM`, and `DFLASH`, plus any plugin-registered name via `SpeculativeAlgorithm.register`. Choose `NEXTN` for models with native MTP support (e.g., DeepSeek V3.2/R1); choose `EAGLE3` for models without MTP (e.g., Qwen MoE). `NEXTN`
`--speculative-num-steps` Number of speculative forward passes per iteration. More steps can increase the acceptance length and throughput but add latency. For prefill, use a small value (`1`) to minimize prefill latency impact. For decode, use a larger value (`2`–`4`) to maximize throughput. Tune based on your latency vs throughput requirements. Prefill: `1`, Decode: `3`
`--speculative-eagle-topk` Limits the number of draft tokens considered per position. Lower values reduce compute on unlikely tokens and are required for the overlap scheduler (enabled by default). Higher values may increase acceptance rates but add overhead. Start with `1` for the overlap scheduler; otherwise, `4`–`8` is typical. `1`
`--speculative-num-draft-tokens` Number of draft tokens generated per speculative step. Higher values increase potential acceptance length and throughput but add per-step computation. Balance against your latency budget — prefill typically uses fewer draft tokens (`2`) to minimize overhead; decode can use more (`4`) to maximize throughput. Prefill: `2`, Decode: `4`
`SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` Enables the overlap plan stream feature for EAGLE v2/v3 speculative decoding workers. This overlaps draft model computation with target model verification, effectively hiding draft latency. Enable when using EAGLE-based speculative decoding; not applicable for NEXTN. `1`
## Quantization
Argument Purpose Reference Value
`--quantization modelslim` Uses the Ascend ModelSlim quantization tool to load W4A4/W4A8/W8A8/W4A16 pre-quantized model weights. This reduces model weight footprint by approximately 50% (for w8) or 75% (for w4) compared to BF16, allowing larger models to fit in NPU memory with minimal accuracy degradation. The quantization method is auto-detected from the model's `quant_model_description.json` file. `modelslim`
## Throughput Configuration
Argument / Variable Purpose Reference Value
`--tokenizer-worker-num` Sets the number of parallel tokenizer worker processes. Increasing this allows concurrent tokenization of multiple input/output streams, preventing the tokenizer from becoming a bottleneck under high request concurrency. Set based on your CPU core count and expected request rate. `4`
`--load-balance-method` Selects the DP load balancing strategy. Available options: `auto` (default, automatically selects the best strategy), `round_robin` (assigns requests to DP ranks in rotation for even distribution), `total_tokens` (balances by token load), `total_requests` (balances by request count), `follow_bootstrap_room` (follows the bootstrap room assignment). Start with `round_robin` for simple even distribution; use `total_tokens` if your requests have highly variable lengths. `round_robin`
`ASCEND_MF_STORE_URL` \[Required for PD disaggregation] Sets the MemFabric config store address for PD disaggregation. This is the prefill primary node's IP with an arbitrary port, used by the decode nodes to discover and connect to the MemFabric-based KV cache transfer service. Omit this for non-disaggregated deployments. Prefill IP with port
## Additional Ascend NPU-Specific Parameters The following environment variables are used in other best practice configurations and may be applicable depending on your model and deployment:
Variable Purpose Typical Usage
`HCCL_OP_EXPANSION_MODE=AIV` Configures the HCCL communication algorithm scheduling to use AIV (Ascend Intelligent Vision) expansion mode, which can improve communication efficiency for certain collective operations. Used in Qwen MoE and R1 non-DSA configurations
`SGLANG_NPU_USE_MLAPO=1` (DeepSeek MLA-specific) Adopts the `MLAPO` fusion operator in the MLA (Multi-Head Latent Attention) preprocessing stage for DeepSeek models with MLA architecture. Used with DeepSeek R1
`SGLANG_USE_FIA_NZ=1` (DeepSeek MLA-specific) Reshapes the KV Cache into FIA NZ format for improved memory access efficiency. Must be used together with `SGLANG_NPU_USE_MLAPO=1`. Used with DeepSeek R1
`SGLANG_NPU_USE_MULTI_STREAM=1` (DeepSeek MoE-specific) Enables dual-stream computation for shared experts and routing experts in DeepSeek MoE models, allowing the two expert types to execute concurrently on separate streams. Used with DeepSeek R1
`SGLANG_USE_AG_AFTER_QLORA=1` Delays the all-gather operation until after Q-LoRA processing. This reduces communication overhead by performing Q-LoRA projection before the all-gather, requiring fewer bytes to be transferred. Used with DeepSeek V3.2/R1 prefill
## Feature Compatibility The table shows feature-by-feature compatibility of SGLang capabilities on Ascend NPU. Features that have performance impact on Ascend are included. Each cell at the intersection of two features indicates whether they can be used together. The symbols used are defined as follows: * 🟢 = Full compatibility * 🟠 = Partial compatibility * ❌ = No compatibility * ❔ = Unknown or TBD
Feature Tensor Parallelism Data Parallelism Expert Parallelism Context Parallelism PD Disaggregation Quantization Chunked Prefill NPU Graph Speculative Decoding PrefixCache Overlap Schedule DP LM Head MLAPO Multistream MoE EPLB NZ Weight Format
Tensor Parallelism 🟢 🟢 🟢 🟢 🟢 🟠 🟠 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
Data Parallelism 🟢 🟢 🟠 🟢 🟢 🟠 🟠 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
Expert Parallelism 🟢 🟢 🟠 🟢 🟢 🟠 🟠 🟢 🟢 🟢 🟠 🟢 🟢 🟢 🟢
Context Parallelism 🟢 🟠 🟠 🟢 🟢 🟠 🟠 🟢 🟢 🟠 ❔ 🟢 🟢 🟢 🟢
PD Disaggregation 🟢 🟢 🟢 🟢 🟢 🟠 🟠 🟢 🟠 🟢 🟠 🟠 🟢 🟠 🟢
Quantization 🟢 🟢 🟢 🟢 🟢 🟠 🟠 🟢 🟢 🟠 🟠 🟢 🟢 🟢 🟢
Chunked Prefill 🟠 🟠 🟠 🟠 🟠 🟠 🟠 🟠 🟢 🟠 🟠 🟠 🟠 🟠 🟢
NPU Graph 🟠 🟠 🟠 🟠 🟠 🟠 🟠 🟠 🟢 🟠 🟠 🟠 🟠 🟠 🟠
Speculative Decoding 🟢 🟢 🟢 🟢 🟢 🟢 🟠 🟠 🟢 🟢 🟠 🟢 🟢 🟢 🟢
PrefixCache 🟢 🟢 🟢 🟢 🟠 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
Overlap Schedule 🟢 🟢 🟢 🟠 🟢 🟠 🟠 🟠 🟢 🟢 🟢 🟢 🟢 🟢 🟢
DP LM Head 🟢 🟢 🟠 ❔ 🟠 🟠 🟠 🟠 🟠 🟢 🟢 🟢 🟢 🟠 🟢
MLAPO 🟢 🟢 🟢 🟢 🟠 🟢 🟠 🟠 🟢 🟢 🟢 🟢 🟢 ❔ 🟢
Multistream MoE 🟢 🟢 🟢 🟢 🟢 🟢 🟠 🟠 🟢 🟢 🟢 🟢 🟢 ❔ 🟢
EPLB 🟢 🟢 🟢 🟢 🟠 🟢 🟠 🟠 🟢 🟢 🟢 🟠 ❔ ❔ 🟠
NZ Weight Format 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟠 🟢 🟢 🟢 🟢 🟢 🟢 🟠
## Feature descriptions ### Tensor Parallelism (`--tp-size`) Splits model weights across multiple NPU devices so that large models can be loaded and run cooperatively. Each device holds a shard of every weight tensor and communicates via HCCL all-reduce. This is the primary mechanism for deploying models that exceed single-device memory. ### Data Parallelism (`--dp-size`, `--attn-dp-size`) `--dp-size` replicates the model across independent device groups to increase throughput by processing multiple requests in parallel. `--attn-dp-size` instead runs attention data parallel inside the TP group: the attention layers are replicated across attention DP ranks while the FFN/MoE layers remain tensor-parallel/expert-parallel. This reduces the communication overhead of attention layers in TP (which is significant during small-batch decode) while keeping the heavy compute and memory demands of MoE layers distributed. Model replicas (`--dp-size` > 1) cannot be combined with attention DP. ### Expert Parallelism (`--ep-size`) Each device holds a subset of experts and routes tokens via all-to-all communication based on the model's gating network. Supports load balancing methods including default round-robin expert placement and dynamic Expert Parallelism Load Balancing (EPLB). ### Context Parallelism (`--attn-cp-size`) Splits long input sequences across devices so that the KV cache and attention computation for a single request are distributed. This enables serving very long context lengths (e.g., 128k tokens) that exceed single-device memory. On Ascend, `--attn-cp-size` must equal `--tp-size`. ### PD Disaggregation (`--disaggregation-mode`) Separates the prefill (P) and decode (D) phases onto different device groups, allowing each phase to be independently optimized for its compute and memory characteristics. Prefill nodes handle long input processing at high throughput, while decode nodes focus on low-latency token generation. Uses the Ascend MemFabric transfer backend (`--disaggregation-transfer-backend ascend`). ### Quantization (`--quantization`) Reduces model weight and activation precision (e.g., W8A8, W4A8, W4A16) to decrease memory usage and increase throughput. On Ascend, the natively supported and highly optimized quantization method is ModelSlim (supporting W4A4, W8A8, W4A8 dynamic/static). Support for other community formats (such as AWQ, GPTQ, Auto-round, and Compressed-tensors) depends on the availability of specific Ascend custom kernels in your environment; see [Quantization on Ascend](/docs/hardware-platforms/ascend-npus/optimization/quantization) for details and compatibility matrices. ### Chunked Prefill (`--chunked-prefill-size`) Breaks large prefill computations into smaller, fixed-size chunks for better scheduling interleaving with decode batches. This prevents long prefill requests from blocking decode latency. Use `-1` to disable chunked prefill on dedicated prefill nodes. Has partial compatibility with most features because chunk boundaries introduce scheduling complexity. ### NPU Graph (`--cuda-graph-bs-decode`) Captures the compute graph on NPU and replays it to eliminate kernel launch overhead, analogous to CUDA Graph on NVIDIA GPUs. Internally uses `torch.npu.NPUGraph`. Most effective for decode with stable, predictable batch sizes (via `--cuda-graph-bs-decode`). Has partial compatibility with most features because graph capture requires fixed control flow and tensor shapes. ### Speculative Decoding (`--speculative-algorithm`) Reduces per-token latency by predicting multiple future tokens in a single forward pass, then verifying them against the model. Ascend supports `NEXTN` (for DeepSeek models, using the model's own hidden states) and `EAGLE3` (for Qwen MoE models, using a separate draft model). Controlled by `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. On Ascend, `SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` enables the optimized overlap scheduler for speculative decoding. For non-greedy requests, Ascend target verification applies request-level `temperature`, `top_k`, and `top_p`. The default target-only mode supports linear and tree-shaped candidates. Classic rejection sampling is available with `--speculative-use-rejection-sampling` for linear chains configured with `--speculative-eagle-topk 1`. ### PrefixCache (`--disable-radix-cache`) Reuses KV cache across requests that share common prompt prefixes (Radix Cache), reducing repeated prefill computation and lowering time-to-first-token. Enabled by default; disable with `--disable-radix-cache` when prefix reuse is not expected (e.g., dedicated prefill nodes in PD disaggregation, or random-input benchmarks). ### Overlap Schedule On Ascend, this is primarily controlled via environment variable SGLANG\_ENABLE\_OVERLAP\_PLAN\_STREAM=1, which enables the optimized overlap scheduler for speculative decoding scenarios. Most effective during decode with speculative decoding and DP attention, where expert dispatch/reduce communication can be overlapped with the next batch's computation. ### DP LM Head (`--enable-dp-lm-head`) Shards the language model head (the final vocabulary projection layer) across DP ranks to reduce memory consumption and improve decode throughput. Without this, each DP rank holds a full copy of the LM head, which is wasteful for large vocabularies. Typically used together with `--attn-dp-size`. ### MLAPO (`SGLANG_NPU_USE_MLAPO=1`) Enables the MLAPO (MLA Pre-Processing Optimization) fusion operator for DeepSeek-style Multi-head Latent Attention models, replacing multiple separate kernels with a single fused kernel for Q/K/V projection and absorption. Reduces prefill latency and memory bandwidth usage. Used in most DeepSeek best-practice configurations. ### Multistream MoE (`SGLANG_NPU_USE_MULTI_STREAM=1`) Enables dual-stream parallel execution for MoE layers, where shared experts and routing experts run concurrently on separate NPU streams. This overlaps shared-expert computation with routed-expert dispatch, improving MoE layer throughput. ### EPLB (`--enable-eplb`) Enables Expert Parallelism Load Balancing to dynamically redistribute experts across devices based on workload, ensuring even expert utilization and preventing hotspots. Uses the `deepseek` balancing algorithm. Compatible with the `deepep` MoE A2A backend, but not with `ascend_fuseep`. ### NZ Weight Format (`SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT`) Casts model weight tensors to the Ascend NPU ACL FRACTAL\_NZ format (format 29) for improved memory access efficiency on Da Vinci AI cores. Enabled by default globally; disable with `SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT=1` when the format conflicts with other features (e.g., EPLB). Applied to linear weights, MoE weights, quantized weights, MLA QKV projections, and KV Cache (via `PA_NZ` / `nzcache` modes for FIA). Related env var: `SGLANG_USE_FIA_NZ=1` (requires `SGLANG_NPU_USE_MLAPO=1`). ## See Also * [Ascend NPU Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) — Complete deployment configurations and benchmark results for all supported models * [Ascend NPU Environment Variables](/docs/hardware-platforms/ascend-npus/reference/environment_variables) — Reference for all Ascend NPU-related environment variables * [DeepSeek V3.2 Guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2) — Detailed usage guide for DeepSeek V3.2 deployment * [Expert Parallelism](/docs/advanced_features/expert_parallelism) — DeepEP configuration and tuning guide # Performance Profiling Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/optimization/profiling During inference serving, it is sometimes necessary to monitor the internal execution flow of the serving framework to identify performance issues. By collecting start/end timestamps of key flows, identifying critical functions or iterations, recording key events, and gathering relevant information, you can quickly locate performance bottlenecks. This guide walks you through the complete workflow of collecting performance data in an SGLang Ascend NPU inference service — from preparation, collection, and analysis to visualization — helping you get started with performance profiling quickly. For more profiling scenarios (e.g., Nsight Systems, PD disaggregation, etc.), see [SGLang Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling). ## Ascend PyTorch Profiler SGLang has built-in PyTorch Profiler support. Through the Ascend `torch_npu` backend, you can directly collect NPU operator-level performance data. No additional packages are required — profiling start/stop is controlled via API requests. ### 1. Environment Setup Launch an SGLang online service and set the `SGLANG_TORCH_PROFILER_DIR` environment variable to control where performance files are saved. Once the service starts, profiling is ready on standby. ```shell Command theme={null} # Set the performance data output directory export SGLANG_TORCH_PROFILER_DIR=./sglang_profile # Start SGLang server (use local model path or HuggingFace model id) sglang serve \ --model-path /path/to/your/model \ --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --tp-size 1 \ --max-running-requests 128 ``` **Profiling-related environment variables:**
Variable Description Default
SGLANG\_TORCH\_PROFILER\_DIR Trace file output directory /tmp
SGLANG\_PROFILE\_WITH\_STACK Record Python call stack (True / False) True
SGLANG\_PROFILE\_RECORD\_SHAPES Record operator input shapes (True / False) True
Priority order: **API parameter > Environment variable > Default value**. ### 2. Collection Methods SGLang provides four collection methods. The core differences are **whether you need to manually send `/start_profile` and `/stop_profile`**. All four methods produce identical results — choose the most convenient one. **Method comparison:**
Method Manual start\_profile Manual stop\_profile Notes
A: API manual start/stop Yes Yes Maximum flexibility for precise control
B: API auto-stop Yes No Set num\_steps, auto-stops and generates output
C: bench\_serving --profile No No Benchmark + profiling in one command
D: sglang.profiler CLI No No Standalone profiling CLI tool
#### Method A: API Manual Start/Stop Send `/start_profile` to start → send workload requests → send `/stop_profile` to stop. After stopping, the server automatically parses the data — **no need to manually call `analyse()`**. ```bash Command theme={null} # Step 1: Start profiling (no num_steps, requires manual stop) curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "output_dir": "./sglang_profile", "start_step": 1, "activities": ["CPU", "GPU"] }' # Step 2: Send workload requests (using curl as example) curl http://127.0.0.1:30000/generate \ -H "Content-Type: application/json" \ -d '{"text": "Hello", "sampling_params": {"max_new_tokens": 10}}' # Step 3: Stop profiling curl -X POST http://127.0.0.1:30000/stop_profile ``` `/stop_profile` returns `"Stop profiling. This will take some time."` — the server needs time to flush trace data to disk and parse it. Wait for the response to complete. This method takes a significant amount of time to parse profiling data; consider using **Method B** instead to avoid lengthy waits. #### Method B: API Auto-Stop Specify `num_steps` in the `/start_profile` request. Profiling stops automatically after N steps and generates output — **no need to manually send `/stop_profile`**. ```bash Command theme={null} # num_steps=10, wait 3 warmup steps, auto-stop after 10 steps curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "output_dir": "./sglang_profile", "start_step": 3, "num_steps": 10, "activities": ["CPU", "GPU"] }' # Just send workload — no /stop_profile needed curl http://127.0.0.1:30000/generate \ -H "Content-Type: application/json" \ -d '{"text": "Hello", "sampling_params": {"max_new_tokens": 32}}' ``` #### Method C: bench\_serving --profile Use SGLang's built-in `bench_serving` with the `--profile` flag. **Automatically handles `/start_profile` and `/stop_profile`** — no manual API calls needed. ```bash Command theme={null} # With --profile-steps: auto-stops after N steps and generates output python -m sglang.bench_serving \ --backend sglang \ --base-url http://127.0.0.1:30000 \ --model /path/to/your/model \ --tokenizer /path/to/your/model \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 100 \ --num-prompts 10 \ --profile \ --profile-steps 10 \ --profile-output-dir ./sglang_profile # Without --profile-steps: /stop_profile sent automatically after benchmark python -m sglang.bench_serving \ --backend sglang \ --base-url http://127.0.0.1:30000 \ --model /path/to/your/model \ --tokenizer /path/to/your/model \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 100 \ --num-prompts 10 \ --profile \ --profile-output-dir ./sglang_profile ``` `--profile-steps N` sends `"num_steps": N` to the server's `/start_profile`, so the server auto-stops and parses data after N steps — bench\_serving skips sending `/stop_profile`. `bench_serving --profile` creates a timestamp subdirectory inside `--profile-output-dir` (e.g., `//`). The output path is shown in the server log as `Profiling done. Traces are saved to: `. On Ascend NPU, SGLang uses `torch_npu._apply_patches()` to automatically redirect PyTorch Profiler's CUDA activity to NPU, so `activities: ["CPU", "GPU"]` actually captures NPU operator events. **`bench_serving --profile` parameters:**
ParameterDescription
--profileEnable auto profiling start/stop
--profile-steps NAuto-stop after N steps (skips /stop\_profile)
--profile-output-dirTrace output directory
#### Method D: sglang.profiler CLI Use the `sglang.profiler` CLI module, which automatically sends `/start_profile` and waits for completion. **Start `sglang.profiler` first, then send inference requests** (otherwise there are no steps to capture and the profiler will wait indefinitely). ```bash Command theme={null} # Terminal 1: Start sglang.profiler first (sends /start_profile, then waits for completion) python3 -m sglang.profiler \ --url http://127.0.0.1:30000 \ --output-dir ./my_profiles \ --num-steps 3 \ --cpu --gpu & ``` ```bash Command theme={null} # Terminal 2: Wait for "Waiting for N steps" output from Terminal 1, then send requests. # The profiler starts recording once /start_profile is received by the server. # Requests sent before the server receives /start_profile will not be captured. curl http://127.0.0.1:30000/generate \ -H "Content-Type: application/json" \ -d '{"text": "Hello", "sampling_params": {"max_new_tokens": 32}}' ``` A simpler and more reliable approach is to use `bench_serving --profile`, which handles both steps automatically: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --base-url http://127.0.0.1:30000 \ --model /path/to/your/model \ --tokenizer /path/to/your/model \ --dataset-name random \ --random-input-len 128 \ --random-output-len 32 \ --num-prompts 10 \ --profile \ --profile-steps 3 \ --profile-output-dir ./my_profiles ``` `sglang.profiler` is essentially a CLI wrapper around the `/start_profile` API. Advanced options like `--profile-by-stage` are also supported. On Ascend NPU, trace flushing is asynchronous and may take a while — the CLI may occasionally block waiting for flush. If it times out, use Method B (API auto-stop) or Method C (bench\_serving --profile) instead. **`sglang.profiler` CLI parameters:**
ParameterDescription
--urlSGLang server address
--output-dir Output directory (defaults to SGLANG\_TORCH\_PROFILER\_DIR)
--num-stepsNumber of steps to profile
--profile-by-stage Profile prefill / decode stages separately
--profile-prefixTrace filename prefix
--cpu / --gpu / --mem / --rpd Activity types to collect
### 3. Full Parameter Reference All methods ultimately send a `/start_profile` request to the server. The full set of supported parameters:
Parameter Description Default
output\_dir Output directory. Falls back to SGLANG\_TORCH\_PROFILER\_DIR or /tmp /tmp
num\_steps Number of steps. If set, profiling auto-stops — no /stop\_profile needed None
start\_step Step index to start profiling (inclusive), for skipping warmup 0
activities Activity types: CPU, GPU, MEM, RPD. On Ascend NPU, only CPU and GPU are supported. MEM depends on CUDA memory APIs and will be silently ignored. RPD requires ROCm and will cause an error. \["CPU", "GPU"]
profile\_by\_stage Profile prefill and decode stages separately false
with\_stack Record Python call stack. Also controllable via SGLANG\_PROFILE\_WITH\_STACK true
record\_shapes Record operator input shapes. Also controllable via SGLANG\_PROFILE\_RECORD\_SHAPES true
profile\_prefix Prefix for trace filenames None
profile\_stages Stages to profile, e.g., \["prefill", "decode"]. Requires profile\_by\_stage None
### 4. Finding Output Files **The server log explicitly indicates where traces are saved.** You can find them via: * **When profiling starts**: server log outputs `Profiling starts. Traces will be saved to: (with profile id: )` ```text theme={null} [2026-05-19 13:23:15] Profiling starts. Traces will be saved to: /tmp/1779196995.6948605 (with profile id: 1779196995.6979997) [2026-05-19 13:23:15] [WARNING] [350443] profiler.py: Invalid parameter export_type: None, reset it to text. [2026-05-19 13:23:15] [WARNING] [350443] profiler.py: Invalid parameter export_type: None, reset it to text. [2026-05-19 13:23:15] INFO: 127.0.0.1:40714 - "POST /start_profile HTTP/1.1" 200 OK ``` * **When profiling stops**: server log outputs `Profiling done. Traces are saved to: ` ```text theme={null} [2026-05-19 13:23:17] Stop profiling... [2026-05-19 13:23:17] [WARNING] [350443] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data. [rank0]:[W519 13:23:17.084812760 compiler_depend.ts:3136] Warning: The indexFromRank 0 is not equal indexFromCurDevice 4 , which might be normal if the number of devices on your collective communication server is inconsistent.Otherwise, you need to check if the current device is correct when calling the interface.If it's incorrect, it might have introduced an error. (function operator()) [2026-05-19 13:23:17] [INFO] [352725] profiler.py: Start parsing profiling data: /tmp/1779196995.6948605/localhost.localdomain_350443_20260519132315700_ascend_pt [2026-05-19 13:23:22] [INFO] [352734] profiler.py: CANN profiling data parsed in a total time of 0:00:04.022310 [2026-05-19 13:23:32] [INFO] [352725] profiler.py: All profiling data parsed in a total time of 0:00:14.305669 [2026-05-19 13:23:32] Profiling done. Traces are saved to: /tmp/1779196995.6948605 ``` * **CLI output**: `sglang.profiler` outputs `Dump profiling traces to ` ```text theme={null} Dump profiling traces to /tmp/1779243331.3219 Waiting for 10 steps and the trace to be flushed.... (profile_by_stage=False) ``` The directory structure is `/___ascend_pt/`. When using Method C (`bench_serving --profile`), a timestamp subdirectory is added: `//___ascend_pt/`. Always check the server log for the exact path: `Profiling done. Traces are saved to: `. ### 5. Viewing Results After profiling stops (either `/stop_profile` returns or `num_steps` auto-triggers), the server **automatically parses the raw data**. The `ASCEND_PROFILER_OUTPUT` directory directly contains the following visualization files — **no need to manually call `analyse()`**:
FileDescription
trace\_view\.json Chrome Tracing format. Open in MindStudio Insight
analysis.dbDatabase-format performance data
ascend\_pytorch\_profiler\_0.db Database-format performance data
kernel\_details.csvKernel-level data
operator\_details.csvOperator-level data
step\_trace\_time.csvStep trace timing data
`trace_view.json` can also be opened using Chrome's built-in `chrome://tracing` or [Perfetto UI](https://ui.perfetto.dev/). In a multi-node deployment, each node generates its own trace files. To merge them into a single unified trace, set `"merge_profiles": true` in the `/start_profile` request. However, on Ascend NPU, the merge feature does **not** fully support the `*_ascend_pt` format — merged results may be incomplete or incorrect. It is recommended to view `trace_view.json` on each node individually instead. For more details, see [Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling#profiler-trace-merger-for-distributed-traces). ### 6. Re-parsing Raw Data (Optional) If you need to **re-parse existing data** with different parameters, or if profiling was interrupted and `ASCEND_PROFILER_OUTPUT` was not auto-generated, use `torch_npu`'s `analyse()` tool: ```python theme={null} from torch_npu.profiler.profiler import analyse analyse("./sglang_profile/_*_ascend_pt/") ``` Normally **no need** to manually run `analyse()` — the server already parses data automatically. Only use this for re-parsing or handling interrupted data. Running `analyse()` when ASCEND\_PROFILER\_OUTPUT already exists will overwrite the original directory. If the original data is still needed, back it up before running `analyse()`. ## Best Practices ### Common Notes * **Finding output**: Check the server log for `Profiling starts. Traces will be saved to: ` and `Profiling done. Traces are saved to: `, or `sglang.profiler` output for `Dump profiling traces to `. * **Control trace file size**: Reduce the number of requests and output length using `--num-prompts` and `--random-output-len` to avoid trace files too large for browsers. * **Warmup iterations**: Set `start_step` to skip the first few warmup steps and capture performance data under steady state. * **Profile step count**: Large values for `num_steps` or `--profile-steps` can lead to lengthy profiling data parsing times. Reduce these values appropriately when you only need a quick overview. * **CUDA Graph impact**: To see the full Python call stack → operator mapping in traces, add `--disable-cuda-graph` when starting the server. Note that this reduces decode performance — only use during profiling. To analyze CUDA Graph capture specifically, use `--enable-profile-cuda-graph` — traces are saved to `SGLANG_TORCH_PROFILER_DIR/graph_capture_profile/`. * **Multi-node deployment**: In multi-node environments, performance data is distributed across nodes. On Ascend NPU, the `merge_profiles` feature has limited support — check `*_ascend_pt/ASCEND_PROFILER_OUTPUT/trace_view.json` on each node individually. In PD disaggregation mode, prefill and decode workers must be profiled separately — see [Profile In PD Disaggregation Mode](/docs/developer_guide/benchmark_and_profiling#profile-in-pd-disaggregation-mode). ## See Also * [SGLang Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling) — General SGLang profiling guide * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — Ascend NPU environment setup * [Ascend NPU Optimization](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning) — Ascend NPU optimization parameters * [Ascend NPU Performance Testing](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing) — Ascend NPU performance benchmarking * [Ascend NPU Environment Variables](/docs/hardware-platforms/ascend-npus/reference/environment_variables) — Environment variable reference # Quantization on Ascend Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/optimization/quantization To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config. SGLang supports **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers. [ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
Quantization scheme Layer type A2 Series Supported A3 Series Supported 950PR/DT Series Supported Diffusion models
W4A4 dynamic Linear √ √ TBD √
W8A8 static Linear √ √ TBD √
W8A8 dynamic Linear √ √ TBD √
MXFP8 (Diffusion, LLM dense) Linear x x √ √
MXFP4 Linear x x √ √
MXFP4 W4A8 Linear x x √ x
MXFP4 W4A8 (ModelSlim) MoE x x √ x
MXFP4 W4A4 Linear x x WIP x
MXFP4 W4A4 (ModelSlim) MoE x x √ x
W4A4 dynamic MoE √ √ TBD x
W4A8 dynamic MoE √ √ TBD x
W8A8 dynamic MoE √ √ TBD x
MXFP8 (LLM MoE) MoE x x √ x
[AWQ on Ascend support](https://github.com/sgl-project/sglang/pull/10158):
Quantization scheme Layer type A2 Series Supported A3 Series Supported 950PR/DT Series Supported
W4A16 Linear √ √ TBD
W8A16 Linear √ √ TBD
W4A16 MoE √ √ TBD
GPTQ on Ascend support
Quantization scheme Layer type A2 Series Supported A3 Series Supported 950PR/DT Series Supported
W4A16 Linear √ √ TBD
W8A16 Linear √ √ TBD
W4A16 MOE MoE √ √ TBD
W8A16 MOE MoE √ √ TBD
[Auto-round on Ascend support](https://github.com/sgl-project/sglang/pull/16699)
Quantization scheme Layer type A2 Series Supported A3 Series Supported 950PR/DT Series Supported
W4A16 Linear √ √ TBD
W8A16 Linear √ √ TBD
W4A16 MoE √ √ TBD
W8A16 MoE √ √ TBD
Compressed-tensors (LLM Compressor) on Ascend support:
Quantization scheme Layer type A2 Series Supported A3 Series Supported 950PR/DT Series Supported
W8A8 dynamic Linear √ √ TBD
W4A8 dynamic with/without activation clip MoE √ √ TBD
W4A16 MOE MoE √ √ TBD
W8A8 dynamic MoE √ √ TBD
[GGUF on Ascend support](https://github.com/sgl-project/sglang/pull/17883)
Quantization type Layer type A2 Series Supported A3 Series Supported 950PR/DT Series Supported
All GGUF types (standard, K-quant) Linear √ √ TBD
All GGUF types (standard, K-quant) MoE √ √ TBD
**Usage Examples:** * Dense model (e.g., Qwen3-14B-Q4\_K\_M.gguf): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen3-14B-Q4_K_M.gguf \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.7 --tp-size 2 ``` * MoE model (e.g., Qwen3-30B-A3B-Q4\_K\_M.gguf): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen3-30B-A3B-Q4_K_M.gguf \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 2 ``` > **Implementation Notes:** > > * GGUF weights are pre-dequantized to FP16/BF16 during model loading on CPU, then transferred to NPU for inference. This trades higher memory usage for faster runtime performance (no per-forward-pass dequantization overhead). > * MoE layers use `npu_grouped_matmul` and `npu_moe_init_routing` / `npu_moe_finalize_routing` for high-performance expert computation. > * TP (tensor parallelism) sharding is supported for both dense and MoE GGUF models. **MXFP8 for LLM dense models (e.g., Qwen3 / Qwen3.5):** LLM dense W8A8 MXFP8 Linear support on Ascend was added in [PR #22352](https://github.com/sgl-project/sglang/pull/22352). Requires Ascend 950PR/DT Series or newer (`npu_dynamic_mx_quant` is not available on A2/A3 Series). * Online MXFP8 quantization (BF16/FP16 weights → MXFP8 at load time): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --quantization mxfp8 \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 1 ``` * Offline MXFP8 quantization (msmodelslim pre-quantized weights, `W8A8_MXFP8` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path /path/to/Qwen3-8B-W8A8-MXFP8 \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 1 ``` > **Implementation Notes:** > > * Online path: `Fp8Config.get_quant_method()` dispatches to `NPUMXFP8LinearMethod`. Weights are quantized once at load via `npu_dynamic_mx_quant(weight, dst_type=torch_npu.float8_e4m3fn)` and pre-transposed to `[in, out]`; activations are per-token quantized at inference and matmul runs via `npu_quant_matmul(..., group_sizes=[1, 1, 32])` (block\_size = 32). > * Offline path: `ModelSlimMXFP8Scheme` loads `float8_e4m3fn` weights + `float8_e8m0fnu` block scales pre-exported by msmodelslim. Transpose is kept as a non-contiguous view (`.data` assignment) — calling `.contiguous()` would physically reorder the pre-quantized layout and break the block-scale mapping. > * MoE MXFP8 (FusedMoE) for LLMs is documented in **MXFP8 for LLM MoE models** below. **MXFP8 for LLM MoE models (e.g. Qwen3-30B-A3B / Qwen3.5 MoE):** LLM MoE W8A8 MXFP8 (FusedMoE) support builds on the dense MXFP8 path. Requires Ascend 950PR/DT Series or newer — the fused MoE MX kernels (`npu_grouped_matmul_swiglu_quant_v2`, `npu_dynamic_mx_quant`) are only available on the 950PR/DT Series. * Online MXFP8 quantization (BF16/FP16 expert weights → MXFP8 at load time): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-30B-A3B \ --quantization mxfp8 \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 1 ``` * Offline MXFP8 quantization (msmodelslim pre-quantized weights, `W8A8_MXFP8` scheme). No `--quantization` flag is needed: the `quant_model_description.json` shipped with the checkpoint selects both the ModelSlim path and the scheme automatically. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path /path/to/Qwen3-30B-A3B-W8A8-MXFP8 \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 1 ``` > **Implementation Notes:** > > * Both paths share the per-gmm kernel `NPUMXFP8MoEMethod` (`hardware_backend/npu/quantization/moe_methods.py`), which tells online from offline by weight dtype. Expert weights and their e8m0 block scales are kept as non-contiguous transpose views — calling `.contiguous()` would tank HBM bandwidth. > * Online path: `Fp8Config.get_quant_method()` dispatches FusedMoE layers to `NPUMXFP8OnlineMoEMethod`, which subclasses `UnquantizedFusedMoEMethod` and overrides only `create_moe_runner` to swap in the MXFP8 kernels — weight creation, weight post-processing and the forward pass are the unquantized Ascend ones. BF16 expert weights `w13`/`w2` are quantized once at load via `npu_dynamic_mx_quant(dst_type=torch.float8_e4m3fn)` (a 3D `[E, N, K]` input is accepted directly). > * Offline path: `ModelSlimMXFP8MoEScheme` (one instance per weight group) loads `float8_e4m3fn` expert weights + uint8 (e8m0, exponent + 127) block scales. The scale is reshaped `[E, N, K/32] → [E, N, K/64, 2]` (contiguous pairing, matching `npu_dynamic_mx_quant`) then transposed. > * Forward: `AscendTPDispatcher` runs `npu_moe_init_routing_v2(quant_mode=3)`, which fuses the per-token MX activation quant into routing (e4m3 payload + e8m0 block scale, reshaped to the pair-split layout). `AscendRunnerCore` then runs gmm1 `npu_grouped_matmul_swiglu_quant_v2` (cumulative `group_list`; fuses gate/up + swiglu + requant, so no separate activation step) → gmm2 `npu_grouped_matmul` (count `group_list`). The UE8M0 (`float8_e8m0fnu`) scale dtypes are passed explicitly; the e4m3 `x`/`weight` dtypes are left implicit. > * **Router gate**: msmodelslim may also quantize `mlp.gate` (`W8A8_MXFP8`). The gate is a `ReplicatedLinear`, so its quantization must be **description-driven**: for the offline `modelslim` path the gate is passed the quant config and dequantized correctly; the online path keeps it in BF16. Loading a quantized gate as BF16 without its block scale scrambles routing and produces garbage output. > * Where the activation quant happens depends on the dispatcher. On `ascend_tp` it is fused into routing as described above. DeepEP has no MXFP8 dispatch dtype, so it keeps dispatching BF16 and gmm1 quantizes the hidden states itself via `npu_dynamic_mx_quant` before the fused kernel — the two paths reach the same gmm1 input. Only the `ascend_tp` path has been validated end-to-end on the Ascend 950PR/DT Series. **MXFP4 W4A8 for LLM dense models (e.g., Qwen3 / Qwen3.5):** LLM dense W4A8 (MXFP4 4-bit weights + MXFP8 8-bit activations) Linear support was added in [PR #23650](https://github.com/sgl-project/sglang/pull/23650). Requires Ascend 950PR/DT Series or newer. * Online W4A8 quantization (BF16/FP16 weights → MXFP4 at load time): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --quantization mxfp_w4a8 \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 1 ``` * Offline W4A8 quantization (msmodelslim pre-quantized weights, `W4A8_MXFP` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`). > **Implementation Notes:** > > * Weights are packed FP4 (`float4_e2m1fn_x2`, two nibbles per byte) with a UE8M0 per-block shared exponent (block\_size = 32); activations are per-token MXFP8. Matmul runs via `npu_quant_matmul(..., x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[0, 0, 32])`. > * The packed-FP4 dtype passed to the NPU ops (`dst_type` / `x2_dtype` / `input_dtype`) must be resolved from `torch_npu.float4_e2m1fn_x2` (an int enum), not the `torch.float4_e2m1fn_x2` dtype object, which recent op-plugin builds reject. > * Online and offline share the same kernel path and layout; they differ only in the weight source (RTN at load vs msmodelslim calibration). **ModelSlim W4A8 MXFP4 for LLM MoE models:** SGLang auto-detects offline ModelSlim `W4A8_MXFP` MoE checkpoints from `quant_model_description.json`; do not pass `--quantization`. This path requires Ascend 950PR/DT Series or newer. ```bash Command theme={null} MODEL_PATH=/path/to/w4a8-mxfp4-moe-model python3 -m sglang.launch_server \ --model-path "$MODEL_PATH" \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --tp-size 1 ``` > **Implementation Notes:** > > * ModelSlim supplies packed MXFP4 `w13` and `w2` expert weights with UE8M0 block scales (block size 32). > * Ascend TP and DeepEP dispatch activations as BF16; this path does not request MXFP8 dispatch. SGLang dynamically quantizes each expert input to MXFP8 immediately before grouped matmul. **MXFP4 W4A4 for LLM dense models (e.g. Qwen3 / Qwen3.5):** LLM dense W4A4 (MXFP4 4-bit weights + 4-bit activations) Linear support was added in [PR #23795](https://github.com/sgl-project/sglang/pull/23795). Requires Ascend 950PR/DT Series or newer — the dual-level online path uses the `DualLevelQuantBatchMatmul` op, which A2/A3 Series lack. On the Ascend NPU backend `--quantization mxfp4` selects this W4A4 path (on GPU the same flag selects the upstream OCP MXFP4 MoE config instead). * Online W4A4 quantization (BF16/FP16 weights → dual-level MXFP4 at load time): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --quantization mxfp4 \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 1 ``` * Offline W4A4 quantization (msmodelslim pre-quantized weights, `W4A4_MXFP4` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`). > **Implementation Notes:** > > * **Online** (`NPUDualLevelMXFP4LinearMethod`) uses **dual-level** MXFP4: both weights and activations are quantized with a fine FP8 (E4M3) L0 block scale plus a coarser L1 scale via `npu_dynamic_dual_level_mx_quant`, and the matmul runs via `npu_dual_level_quant_matmul` (weight in FRACTAL\_NZ). Dual-level captures per-block dynamic range far better than a single UE8M0 (power-of-2) scale, which is what made an earlier single-level RTN online path degenerate (greedy decoding could loop without emitting EOS). > * **Offline** (`ModelSlimMXFP4Scheme` → `NPUSingleLevelMXFP4OfflineLinearMethod`) is **single-level**: msmodelslim's `W4A4_MXFP4` checkpoint ships single-level UE8M0 block scales (block\_size = 32), so the matmul runs via `npu_quant_matmul(..., x1_dtype=x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[1, 1, 32])`. The online and offline paths therefore use different matmul kernels — they no longer share the matmul path. > * As with W4A8, the packed-FP4 dtype passed to the NPU ops (`dst_type` / `x2_dtype`) must be resolved from `torch_npu.float4_e2m1fn_x2` (an int enum), not the `torch.float4_e2m1fn_x2` dtype object, which recent op-plugin builds reject. > * Validated end-to-end on Ascend 950PR/DT Series hardware. **ModelSlim W4A4 MXFP4 for LLM MoE models:** SGLang auto-detects offline ModelSlim `W4A4_MXFP4` MoE checkpoints from `quant_model_description.json`; do not pass `--quantization`. This path requires Ascend 950PR/DT Series or newer. ```bash Command theme={null} MODEL_PATH=/path/to/w4a4-mxfp4-moe-model python3 -m sglang.launch_server \ --model-path "$MODEL_PATH" \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --tp-size 1 ``` > **Implementation Notes:** > > * ModelSlim supplies packed MXFP4 `w13` and `w2` expert weights with UE8M0 block scales (block size 32). > * Ascend TP and DeepEP dispatch activations as BF16; this path does not request MXFP4 dispatch. SGLang dynamically quantizes each expert input to MXFP4 immediately before grouped matmul. ## Diffusion Model Quantization on Ascend NPU SGLang-Diffusion supports MXFP8 online and offline quantization for diffusion models (such as Wan2.2) on Ascend NPUs. MXFP8 requires Ascend 950PR/DT Series; the ModelSlim W8A8/W4A4 schemes work on A2/A3 Series. **Requirements for MXFP8:** CANN ≥ 8.0.RC3, Ascend 950PR/DT Series
Quantization method quant\_type in JSON Scheme class Mode A2/A3 Series Supported 950PR/DT Series Supported Trigger
MXFP8 (W8A8) — MXFP8Config Online x √ --quantization mxfp8
MXFP8 (W8A8) W8A8\_MXFP8 ModelSlimMXFP8Scheme Offline x √ auto-detected from quant\_model\_description.json
W8A8 static W8A8 ModelSlimW8A8Int8 Offline √ TBD auto-detected from quant\_model\_description.json
W8A8 dynamic W8A8\_DYNAMIC ModelSlimW8A8Int8 Offline √ TBD auto-detected from quant\_model\_description.json
W4A4 dynamic W4A4\_DYNAMIC ModelSlimW4A4Int4 Offline √ TBD auto-detected from quant\_model\_description.json
### Online MXFP8 Quantization Online quantization dynamically quantizes FP16/BF16 weights to MXFP8 at load time using `npu_dynamic_mx_quant` + `npu_quant_matmul` CANN kernels. Pass `--quantization mxfp8` to override auto-detection. ```bash Command theme={null} # Start the diffusion server with online MXFP8 quantization sglang serve \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --quantization mxfp8 \ --num-gpus 4 ``` ```bash Command theme={null} # One-shot generation sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --quantization mxfp8 \ --prompt "a beautiful sunset over the mountains" \ --save-output ``` ### Offline MXFP8 Quantization (ModelSlim) For offline quantization, pre-quantize the model with msModelSlim and load the resulting checkpoint. The quantization scheme is auto-detected from `quant_model_description.json`, so no extra `--quantization` flag is needed. **Step 1: Quantize with msModelSlim** ```bash Command theme={null} msmodelslim quant \ --model_path /path/to/wan2_2_float_weights \ --save_path /path/to/wan2_2_mxfp8_weights \ --device npu \ --model_type Wan2_2 \ --quant_type mxfp8 \ --trust_remote_code True ``` > Note: SGLang does not support quantized embeddings; disable embedding quantization when using msmodelslim. **Step 2: Convert to Diffusers format** msModelSlim saves quantized Wan2.2 weights in the original Wan format. Convert to Diffusers format using the provided repack script: ```bash Command theme={null} python python/sglang/multimodal_gen/tools/wan_repack.py \ --input-path /path/to/wan2_2_mxfp8_weights \ --output-path /path/to/wan2_2_mxfp8_diffusers ``` Then copy all files from the original Diffusers checkpoint (except the `transformer`/`transformer_2` folders) into the output directory. **Step 3: Run inference** ```bash Command theme={null} sglang generate \ --model-path /path/to/wan2_2_mxfp8_diffusers \ --prompt "a beautiful sunset over the mountains" \ --save-output ``` For pre-quantized checkpoints available on ModelScope, see [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech). # Ring-SP Performance (Wan2.1-T2V-1.3B) Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/optimization/ring_sp_performance This page reports Ring-SP performance on Ascend NPU with `torch_npu==2.10.0`. * Baseline config: `ulysses=1, ring=1` (short: `u1r1`) * Ring-SP config: `ulysses=1, ring=2` (short: `u1r2`) ## Benchmark Setup * Model: `Wan2.1-T2V-1.3B-Diffusers` * Prompt: `"a cat is playing piano"` * Framework command: `sglang generate` * Runtime: `torch_npu==2.10.0` ## Generate Commands ### Baseline (`u1r1`) ```bash theme={null} sglang generate --model-path /nas/disk1/Wan2.1-T2V-1.3B-Diffusers \ --prompt "a cat is playing piano" --num-gpus 1 --ring-degree 1 \ --save-output ``` ### Ring-SP (`u1r2`) ```bash theme={null} sglang generate --model-path /nas/disk1/Wan2.1-T2V-1.3B-Diffusers \ --prompt "a cat is playing piano" --num-gpus 2 --ring-degree 2 \ --save-output ``` ## Benchmarks Benchmark Disclaimer These numbers are from one fixed setup and one prompt case. Actual performance may vary by model settings, environment, and workload. ### Stage Time Breakdown
Stage / Metric u1r2 (s) u1r1 baseline (s) Speedup
InputValidation 0.0003 0.0002 0.67x
TextEncoding 3.5936 3.5820 1.00x
LatentPreparation 0.0007 0.0055 7.86x
TimestepPreparation 0.0008 0.0007 0.88x
Denoising 121.2788 239.2580 1.97x
Decoding 13.8685 16.4969 1.19x
Total (Pixel data generated) 141.86 266.50 1.88x
## Summary * With `torch_npu==2.10.0`, Ring-SP (`u1r2`) runs successfully on NPU for this case. * End-to-end generation time improves from `266.50s` to `141.86s` (`1.88x`). * The main gain comes from `DenoisingStage` (`1.97x`), while decoding also improves (`1.19x`). # Environment Variables Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/reference/environment_variables SGLang supports various environment variables related to Ascend NPU that can be used to configure its runtime behavior. This document provides a list of commonly used environment variables and aims to stay updated over time. ## Directly Used in SGLang
Environment Variable Description Default Value
SGLANG\_NPU\_USE\_MLAPO Adopts the MLAPO fusion operator in the attention
preprocessing stage of the MLA model.		 false
SGLANG\_USE\_FIA\_NZ Reshapes KV Cache for FIA NZ format.
SGLANG\_USE\_FIA\_NZ must be enabled with SGLANG\_NPU\_USE\_MLAPO		 false
SGLANG\_NPU\_USE\_MULTI\_STREAM Enable dual-stream computation of shared experts
and routing experts in DeepSeek models.
Enable dual-stream computation in DeepSeek DSA Indexer.		 false
SGLANG\_NPU\_DISABLE\_ACL\_FORMAT\_WEIGHT Disable casting model weight tensor to a specific NPU
ACL format.		 false
SGLANG\_DEEPEP\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK The maximum number of dispatched tokens on each rank. 128
## Used in DeepEP Ascend
Environment Variable Description Default Value
DEEPEP\_NORMAL\_LONG\_SEQ\_PER\_ROUND\_TOKENS Enable long-sequence token pipelining in dispatch stage. Indicates
the number of tokens transmitted per round on each rank.		 8192
DEEPEP\_NORMAL\_LONG\_SEQ\_ROUND Enable long-sequence token pipelining in dispatch stage. Indicates
the number of rounds transmitted on each rank.		 1
DEEPEP\_NORMAL\_COMBINE\_ENABLE\_LONG\_SEQ Enable long-sequence token pipelining in combine stage.
The value 0 means disabled.		 0
MOE\_ENABLE\_TOPK\_NEG\_ONE Needs to be enabled when the expert ID to be processed by
DEEPEP contains -1.		 0
DEEP\_NORMAL\_MODE\_USE\_INT8\_QUANT Deprecated — will be removed in a future release.
When set to 1, quantizes intermediate activations to INT8 in
the DeepEP dispatch operator during normal mode, reducing communication
volume for W8A8-quantized MoE models. This variable will become a no-op;
the quantization behavior will be inferred automatically. 		0
DEEPEP\_HCCL\_BUFFSIZE Configures the HCCL buffer size (in MB) for the process groups used by
DeepEP MoE All-to-All communication (the default group and moe-named
groups), allowing the MoE communication buffer to be tuned independently from
non-MoE groups. If unset, falls back to HCCL\_BUFFSIZE, then to 200.
A larger buffer reduces All-to-All latency at the cost of more HBM usage.		 200
## Others
Environment Variable Description Default Value
TASK\_QUEUE\_ENABLE Used to control the optimization level of the dispatch queue
for the task\_queue operator. Detail		 1
INF\_NAN\_MODE\_ENABLE Controls whether the chip uses saturation mode or INF\_NAN mode. Detail 1
STREAMS\_PER\_DEVICE Configures the maximum number of streams for the stream pool. Detail 32
PYTORCH\_NPU\_ALLOC\_CONF Controls the behavior of the cache allocator.
This variable changes memory usage and may cause performance fluctuations. Detail
ASCEND\_MF\_STORE\_URL The address of config store in MemFabric during PD separation,
which is generally set to the IP address of the P primary node
with an arbitrary port number.
ASCEND\_LAUNCH\_BLOCKING Controls whether synchronous mode is enabled during operator execution. Detail 0
HCCL\_OP\_EXPANSION\_MODE Configures the expansion position for communication algorithm scheduling. Detail
HCCL\_BUFFSIZE Controls the size of the buffer area for shared data between two NPUs.
The unit is MB, and the value must be greater than or equal to 1. Detail		 200
HCCL\_SOCKET\_IFNAME Configures the name of the network card used by the Host
during HCCL initialization. Detail
GLOO\_SOCKET\_IFNAME Configures the network interface name for GLOO communication.
# Glossary Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/reference/glossary This page covers the hardware concepts, communication libraries, deployment terminology, and common abbreviations you will encounter throughout the Ascend NPU documentation. Refer back here when you run into unfamiliar terms. ## Hardware ### Supported devices SGLang supports the following Ascend inference hardware: | Hardware | Devices | Dies per card | Memory configuration | | - | - | - | - | | Ascend A2 Series Products | 8 | 1 | 8(cards) × 1(die/card) × 64(GB/die) | | Ascend A3 Series Products | 16 | 2 | 8(cards) × 2(dies/card) × 64(GB/die) | Throughout these docs, **A2 Series** and **A3 Series** are used as shorthand for the hardware above. Docker image tags use `910b` for the A2 Series and `a3` for the A3 Series. For example, `cann9.0.0-910b-v0.5.16` and `cann9.0.0-a3-v0.5.16`. On the A3 Series, each card has 2 dies, giving 16 devices vs 8 on the A2 Series. Benchmark pages use "Cards" to refer to physical cards, so `Cards: 4` on the A3 Series means `--tp-size 8`. From a deployment perspective, the two key differences between the A2 Series and the A3 Series are: 1. dies per card — which drives both total memory and `--tp-size` configuration 2. PD disaggregation — the A2 Series requires setting `export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"`, while the A3 Series uses the default protocol. ### NPU **NPU** stands for Neural Processing Unit. Each NPU device is a single `davinci` core. The terms "NPU" and "davinci" are used interchangeably in commands and error logs. On the A2 Series, devices are numbered `/dev/davinci0` through `/dev/davinci7` (8 devices). On the A3 Series, devices are numbered `/dev/davinci0` through `/dev/davinci15` (16 devices). On either an A2 Series or an A3 Series server, run `npu-smi info` to view NPU information such as device health, memory usage, and chip status. If the command is not found or reports no devices, the driver is likely not installed. Follow the [Ascend driver installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/900/softwareinst/instg/instg_0005.html?OS=openEuler\&InstallType=local) to install it. ## Communication libraries | Library | Description | | - | - | | **HCCL** (Huawei Collective Communication Library) | The primary communication backend for multi-card data transfer on Ascend NPUs. Equivalent to NVIDIA NCCL. Used in `--nnodes`, `--tp-size`, and all distributed scenarios. | | **GLOO** | Meta's collective communications library. Used alongside HCCL for distributed initialization and coordination. | | **DeepEP** (Deep Expert Parallelism) | A communication library optimized for Mixture-of-Experts (MoE) all-to-all dispatch and combine operations. Used with `--moe-a2a-backend deepep`. | | **RDMA** (Remote Direct Memory Access) | Enables direct memory access between nodes over InfiniBand or RoCE networks. Required for multi-node PD disaggregation. | ## Quantization and precision | Notation | Meaning | | - | - | | **W8A8** | 8-bit weights, 8-bit activations | | **W4A8** | 4-bit weights, 8-bit activations | | **W4A16** | 4-bit weights, 16-bit activations | | **BF16** | Brain Floating Point 16 — 16-bit format optimized for ML workloads | | **FP8** | 8-bit Floating Point — not supported on the A2/A3 Series | | **INT8** | 8-bit Integer quantization | To apply quantization, use `--quantization modelslim` for W8A8 INT8, or load a pre-quantized checkpoint directly from a model hub. ## Deployment terminology ### Prefill-Decode (PD) disaggregation PD disaggregation separates inference into two stages running on different nodes: * **Prefill** (P): Processes the entire input prompt at once. Compute-bound. * **Decode** (D): Generates tokens one at a time. Memory-bandwidth-bound. **PD Mixed**: Both stages run on the same set of nodes. | Shorthand | Meaning | | - | - | | **1P1D** | 1 prefill node + 1 decode node | | **2P1D** | 2 prefill nodes + 1 decode node | | **1P2D** | 1 prefill node + 2 decode nodes | You will see these in [Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) section headings, e.g., `W8A8 2P1D 32P` means "W8A8 quantization, 2 prefill nodes + 1 decode node, 32 cards total." ### Parallelism strategies | Strategy | Flag | Description | | - | - | - | | **Tensor Parallelism (TP)** | `--tp-size` | Splits model weights across NPUs within a node | | **Data Parallelism (DP)** | `--dp-size` | Replicates the model across nodes for higher throughput | | **Expert Parallelism (EP)** | `--ep-size` | Distributes MoE experts across devices; requires `--moe-a2a-backend` | | **Context Parallelism (CP)** | `--attn-cp-size` | Splits long context windows across devices for extended sequence length | | **Pipeline Parallelism (PP)** | `--pp-size` | Splits model layers across devices sequentially | ### Speculative decoding | Algorithm | Description | | - | - | | **EAGLE3** | Uses an external draft model (specified via `--speculative-draft-model-path`) to propose candidate tokens. Supports top-k sampling. | | **NEXTN** | Uses the model's built-in Multi-Token Prediction (MTP) heads — no separate draft model needed. Available for models with native MTP support. | | **MTP** (Multi-Token Prediction) | A model architecture feature where the model predicts multiple future tokens per step. The foundation for NEXTN speculative decoding. | ### Performance metrics | Metric | Description | | - | - | | **TPOT** (Time Per Output Token) | Average time to generate each output token. Lower is better. | | **TTFT** (Time To First Token) | Latency from request arrival to first generated token. Critical for interactive use. | ## Model architecture terms | Term | Description | | - | - | | **MoE** (Mixture of Experts) | Model architecture where only a subset of parameters (experts) is activated per token, reducing compute. Common in DeepSeek, Qwen3-30B-A3B, and MiMo models. | | **MLA** (Multi-head Latent Attention) | Attention variant that compresses key-value representations into a lower-dimensional latent space. Used by DeepSeek models. | | **GQA** (Grouped-Query Attention) | Attention variant where multiple query heads share a single key-value head. Used by Qwen3 dense models. | | **DSA** (DeepSeek Sparse Attention) | DeepSeek's sparse attention mechanism; reduces KV cache overhead for long contexts. | | **FFN** (Feed-Forward Network) | The non-attention component of each transformer layer. In MoE models, the FFN is replaced by multiple expert FFN layers selected by a router. | ## Other common abbreviations | Abbreviation | Expansion | | - | - | | **ACL** (Ascend Computing Language) | Low-level Ascend compute API; seen in error logs | | **DVFS** (Dynamic Voltage and Frequency Scaling) | Hardware frequency scaling to maintain performance stability | | **NUMA** (Non-Uniform Memory Access) | Memory architecture affecting multi-socket performance | | **KV Cache** (Key-Value Cache) | Cached attention key-value tensors to avoid recomputation during decode | | **LoRA** (Low-Rank Adaptation) | Parameter-efficient fine-tuning method | | **HF** (Hugging Face) | Model hub; `HF_TOKEN` / `HF_ENDPOINT` control model download access | | **UB** (Unified Buffer) | On-chip NPU memory; referenced in operator optimization | | **DMA** (Direct Memory Access) | Data transfer mechanism between host and device memory | ## Where to go next * [Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — launch your first server * [Installation Guide](/docs/hardware-platforms/ascend-npus/getting-started/installation) — full installation with component version mapping * [Supported Features](/docs/hardware-platforms/ascend-npus/reference/support_features) — per-parameter Ascend support status * [Supported Models](/docs/hardware-platforms/ascend-npus/reference/support_models) — models verified on Ascend NPU * [Model Tutorials](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1) — step-by-step deployment guides * [Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) — benchmark configurations and results # Supported Features Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/reference/support_features This section describes the basic functions and features supported by the Ascend NPU. If you encounter issues or have any questions, please [open an issue](https://github.com/sgl-project/sglang/issues). If you want to know the meaning and usage of each parameter, click [Server Arguments](../../../advanced_features/server_arguments). ## Model and tokenizer
Argument Defaults Options Server supported
`--model-path`
`--model`		 `None` Type: str A2/A3 Series
`--tokenizer-path` `None` Type: str A2/A3 Series
`--tokenizer-mode` `auto` `auto`, `slow` A2/A3 Series
`--tokenizer-backend` `huggingface` `huggingface`, `fastokens` A2/A3 Series
`--tokenizer-worker-num` `1` Type: int A2/A3 Series
`--detokenizer-worker-num` `1` Type: int A2/A3 Series
`--skip-tokenizer-init` `False` bool flag (set to enable) A2/A3 Series
`--load-format` `auto` `auto`, `safetensors`, `gguf` A2/A3 Series
`--model-loader-extra-config` `{}` Type: str A2/A3 Series
`--trust-remote-code` `False` bool flag (set to enable) A2/A3 Series
`--context-length` `None` Type: int A2/A3 Series
`--is-embedding` `False` bool flag (set to enable) A2/A3 Series
`--enable-multimodal` `None` bool flag (set to enable) A2/A3 Series
`--revision` `None` Type: str A2/A3 Series
`--model-impl` `auto` `auto`, `sglang`,
`transformers`		 A2/A3 Series
`--model-config-parser` `auto` Type: str A2/A3 Series
## HTTP server
Argument Defaults Options Server supported
`--host` `127.0.0.1` Type: str A2/A3 Series
`--port` `30000` Type: int A2/A3 Series
`--skip-server-warmup` `False` bool flag (set to enable) A2/A3 Series
`--warmups` `None` Type: str A2/A3 Series
`--nccl-port` `None` Type: int A2/A3 Series
`--fastapi-root-path` `None` Type: str A2/A3 Series
`--grpc-mode` `False` bool flag (set to enable) Planned
## SSL/TLS
Argument Defaults Options Server supported
`--ssl-keyfile` `None` Type: str A2/A3 Series
`--ssl-certfile` `None` Type: str A2/A3 Series
`--ssl-keyfile-password` `None` Type: str A2/A3 Series
`--enable-ssl-refresh` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-http2` `False` bool flag
(set to enable)		 A2/A3 Series
## Quantization and data type
Argument Defaults Options Server supported
`--dtype` `auto` `auto`,
`float16`,
`bfloat16`		 A2/A3 Series
`--quantization` `None` `modelslim` A2/A3 Series
`--quantization-param-path` `None` Type: str Special for GPU
`--kv-cache-dtype` `auto` `auto` A2/A3 Series
`--enable-fp32-lm-head` `False` bool flag
(set to enable)		 A2/A3 Series
`--modelopt-quant` `None` Type: str Special for GPU
`--modelopt-checkpoint-restore-path` `None` Type: str Special for GPU
`--modelopt-checkpoint-save-path` `None` Type: str Special for GPU
`--modelopt-export-path` `None` Type: str Special for GPU
`--quantize-and-serve` `False` bool flag
(set to enable)		 Special for GPU
`--rl-quant-profile` `None` Type: str Special for GPU
## Memory and scheduling
Argument Defaults Options Server supported
`--mem-fraction-static` `None` Type: float A2/A3 Series
`--max-running-requests` `None` Type: int A2/A3 Series
`--prefill-max-requests` `None` Type: int A2/A3 Series
`--max-queued-requests` `None` Type: int A2/A3 Series
`--max-total-tokens` `None` Type: int A2/A3 Series
`--chunked-prefill-size` `None` Type: int A2/A3 Series
`--max-prefill-tokens` `16384` Type: int A2/A3 Series
`--schedule-policy` `fcfs` `lpm`,
`fcfs`,
`random`		 A2/A3 Series
`--enable-priority-scheduling` `False` bool flag
(set to enable)		 A2/A3 Series
`--disable-priority-preemption` `False` bool flag
(set to enable)		 A2/A3 Series
`--default-priority-value` `None` Type: int A2/A3 Series
`--schedule-low-priority-values-first` `False` bool flag
(set to enable)		 A2/A3 Series
`--priority-scheduling-preemption-threshold` `10` Type: int A2/A3 Series
`--retraction-policy` `length` `length`,
`priority`		 A2/A3 Series
`--schedule-conservativeness` `1.0` Type: float A2/A3 Series
`--page-size` `128` Type: int A2/A3 Series
`--swa-full-tokens-ratio` `0.8` Type: float A2/A3 Series
`--radix-eviction-policy` `lru` `lru`,
`lfu`		 A2/A3 Series
`--enable-prefill-delayer` `False` bool flag
(set to enable)		 A2/A3 Series
`--prefill-delayer-max-delay-passes` `30` Type: int A2/A3 Series
`--prefill-delayer-token-usage-low-watermark` `None` Type: float A2/A3 Series
`--prefill-delayer-forward-passes-buckets` `None` List\[float] A2/A3 Series
`--prefill-delayer-wait-seconds-buckets` `None` List\[float] A2/A3 Series
`--prefill-delayer-queue-min-ratio` `None` Type: float A2/A3 Series
`--prefill-delayer-max-delay-ms` `None` Type: float A2/A3 Series
`--abort-on-priority-when-disabled` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-dynamic-chunking` `False` bool flag
(set to enable)		 A2/A3 Series
## Runtime options
Argument Defaults Options Server supported
`--device` `None` Type: str A2/A3 Series
`--tensor-parallel-size`
`--tp-size`		 `1` Type: int A2/A3 Series
`--pipeline-parallel-size`
`--pp-size`		 `1` Type: int; Currently `2` not supported; Cannot be used together with TP A2/A3 Series
`--attention-context-parallel-size`
`--attn-cp-size`		 `1` Type: int; must be equal to --tp-size A2/A3 Series
`--moe-data-parallel-size`
`--moe-dp-size`		 `1` Type: int A2/A3 Series
`--pp-max-micro-batch-size` `None` Type: int A2/A3 Series
`--pp-async-batch-depth` `0` Type: int A2/A3 Series
`--stream-interval` `1` Type: int A2/A3 Series
`--batch-notify-size` `16` Type: int A2/A3 Series
`--incremental-streaming-output` `False` bool flag (set to enable) A2/A3 Series
`--stream-response-default-include-usage` `False` bool flag (set to enable) A2/A3 Series
`--enable-streaming-session` `False` bool flag (set to enable) A2/A3 Series
`--enable-session-radix-cache` `False` bool flag
(set to enable)		 A2/A3 Series
`--random-seed` `None` Type: int A2/A3 Series
`--constrained-json-whitespace-pattern` `None` Type: str A2/A3 Series
`--constrained-json-disable-any-whitespace` `False` bool flag (set to enable) A2/A3 Series
`--watchdog-timeout` `300` Type: float A2/A3 Series
`--soft-watchdog-timeout` `300` Type: float A2/A3 Series
`--dist-timeout` `None` Type: int A2/A3 Series
`--download-dir` `None` Type: str A2/A3 Series
`--model-checksum` `None` Type: str A2/A3 Series
`--base-gpu-id` `0` Type: int A2/A3 Series
`--gpu-id-step` `1` Type: int A2/A3 Series
`--sleep-on-idle` `False` bool flag (set to enable) A2/A3 Series
`--load-snapshot-publish-interval` `15` Type: int A2/A3 Series
`--use-ray` `False` bool flag (set to enable) Special for GPU
`--custom-sigquit-handler` `None` Only for engine A2/A3 Series
## Logging
Argument Defaults Options Server supported
`--log-level` `info` Type: str A2/A3 Series
`--log-level-http` `None` Type: str A2/A3 Series
`--log-requests` `False` bool flag
(set to enable)		 A2/A3 Series
`--log-requests-level` `2` `0`, `1`, `2`, `3` A2/A3 Series
`--log-requests-format` text `text`, `json` A2/A3 Series
`--crash-dump-folder` `None` Type: str A2/A3 Series
`--enable-metrics` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-mfu-metrics` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-metrics-for-all-schedulers` `False` bool flag
(set to enable)		 A2/A3 Series
`--tokenizer-metrics-custom-labels-header` `x-custom-labels` Type: str A2/A3 Series
`--tokenizer-metrics-allowed-custom-labels` `None` List\[str] A2/A3 Series
`--extra-metric-labels` `None` Type: JSON/Dict A2/A3 Series
`--bucket-time-to-first-token` `None` List\[float] A2/A3 Series
`--bucket-inter-token-latency` `None` List\[float] A2/A3 Series
`--bucket-e2e-request-latency` `None` List\[float] A2/A3 Series
`--prompt-tokens-buckets` `None` List\[str] A2/A3 Series
`--generation-tokens-buckets` `None` List\[str] A2/A3 Series
`--gc-warning-threshold-secs` `0.0` Type: float A2/A3 Series
`--decode-log-interval` `40` Type: int A2/A3 Series
`--enable-request-time-stats-logging` `False` bool flag
(set to enable)		 A2/A3 Series
`--kv-events-config` `None` Type: str Special for GPU
`--enable-forward-pass-metrics` `False` bool flag (set to enable) A2/A3 Series
`--forward-pass-metrics-worker-id` \`\` Type: str A2/A3 Series
`--forward-pass-metrics-ipc-name` `None` Type: str A2/A3 Series
`--enable-trace` `False` bool flag
(set to enable)		 A2/A3 Series
`--trace-modules` `request` Type: str A2/A3 Series
`--otlp-traces-endpoint` `localhost:4317` Type: str A2/A3 Series
`--log-requests-target` `None` Type: str A2/A3 Series
`--uvicorn-access-log-exclude-prefixes` `[]` List\[str] A2/A3 Series
## RequestMetricsExporter configuration
Argument Defaults Options Server supported
`--export-metrics-to-file` `False` bool flag
(set to enable)		 A2/A3 Series
`--export-metrics-to-file-dir` `None` Type: str A2/A3 Series
## API related
Argument Defaults Options Server supported
`--api-key` `None` Type: str A2/A3 Series
`--admin-api-key` `None` Type: str A2/A3 Series
`--served-model-name` `None` Type: str A2/A3 Series
`--weight-version` `default` Type: str A2/A3 Series
`--chat-template` `None` Type: str A2/A3 Series
`--hf-chat-template-name` `None` Type: str A2/A3 Series
`--completion-template` `None` Type: str A2/A3 Series
`--file-storage-path` `sglang_storage` Type: str Unused reserved parameter
`--enable-cache-report` `False` bool flag
(set to enable)		 A2/A3 Series
`--reasoning-parser` `None` `deepseek-r1`
`deepseek-v3`
`glm45`
`gpt-oss`
`kimi`
`qwen3`
`qwen3-thinking`
`step3`		 A2/A3 Series
`--default-chat-template-kwargs` `None` Type: JSON / Dict A2/A3 Series
`--strip-thinking-cache` `False` bool flag (set to enable) A2/A3 Series
`--enable-strict-thinking` `False` bool flag (set to enable) A2/A3 Series
`--tool-call-parser` `None` `llama3`
`pythonic`
`qwen`
`qwen3_coder`		 A2/A3 Series
`--sampling-defaults` `model` `openai`, `model` A2/A3 Series
`--asr-max-buffer-seconds` `60` Type: int A2/A3 Series
`--asr-max-concurrent-sessions` `32` Type: int A2/A3 Series
## Data parallelism
Argument Defaults Options Server supported
`--data-parallel-size`
`--dp-size`		 `1` Type: int A2/A3 Series
`--attention-data-parallel-size`
`--attn-dp-size`		 `1` Type: int A2/A3 Series
`--load-balance-method` `auto` `auto`,
`round_robin`,
`follow_bootstrap_room`,
`total_requests`,
`total_tokens`		 A2/A3 Series
## Multi-node distributed serving
Argument Defaults Options Server supported
`--dist-init-addr`
`--nccl-init-addr`		 `None` Type: str A2/A3 Series
`--nnodes` `1` Type: int A2/A3 Series
`--node-rank` `0` Type: int A2/A3 Series
## Model override args
Argument Defaults Options Server supported
`--json-model-override-args` `{}` Type: str A2/A3 Series
`--preferred-sampling-params` `None` Type: str A2/A3 Series
## LoRA (restricted to Qwen series models)
Argument Defaults Options Server supported
`--enable-lora` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-lora-overlap-loading` `False` bool flag
(set to enable)		 A2/A3 Series
`--max-lora-rank` `None` Type: int A2/A3 Series
`--lora-target-modules` `None` `all` A2/A3 Series
`--lora-paths` `None` Type: List\[str] /
JSON objects		 A2/A3 Series
`--max-loras-per-batch` `8` Type: int A2/A3 Series
`--max-loaded-loras` `None` Type: int A2/A3 Series
`--lora-eviction-policy` `lru` `lru`,
`fifo`		 A2/A3 Series
`--lora-backend` `csgmv` `triton`,
`csgmv`,
`ascend`,
`torch_native`		 A2/A3 Series
`--experts-shared-outer-loras` `None` Type: bool A2/A3 Series
`--lora-use-virtual-experts` `False` bool flag
(set to enable)		 Special for GPU
`--lora-strict-loading` `False` Type: bool Special for GPU
`--lora-drain-wait-threshold` `0.0` Type: float A2/A3 Series
`--max-lora-chunk-size` `16` `16`, `32`,
`64`, `128`		 Special for GPU
## Kernel backends (attention, sampling, grammar, GEMM)
Argument Defaults Options Server supported
`--attention-backend` `None` `ascend` A2/A3 Series
`--prefill-attention-backend` `None` `ascend` A2/A3 Series
`--decode-attention-backend` `None` `ascend` A2/A3 Series
`--sampling-backend` `None` `pytorch`,
`ascend`		 A2/A3 Series
`--grammar-backend` `None` `xgrammar`,
`outlines`,
`llguidance`		 A2/A3 Series
`--radix-cache-backend` `None` Type: str A2/A3 Series
`--mm-attention-backend` `None` `ascend_attn` A2/A3 Series
`--dsa-prefill-backend` `flashmla_sparse` `flashmla_sparse`,
`flashmla_decode`,
`fa3`,
`tilelang`,
`aiter`		 Special for GPU
`--dsa-decode-backend` `fa3` `flashmla_prefill`,
`flashmla_kv`,
`fa3`,
`tilelang`,
`aiter`		 Special for GPU
`--fp8-gemm-backend` `auto` `auto`,
`deep_gemm`,
`flashinfer_trtllm`,
`flashinfer_cutlass`,
`flashinfer_deepgemm`,
`cutlass`,
`triton`,
`aiter`		 Special for GPU
`--disable-flashinfer-autotune` `False` bool flag
(set to enable)		 Special for GPU
## Speculative decoding Ascend NPU supports non-greedy target verification for the supported EAGLE-based speculative algorithms. Request-level `temperature`, `top_k`, and `top_p` are applied when constructing the target probability distribution. Request-level `top_k` controls token sampling and is separate from `--speculative-eagle-topk`, which controls the draft tree branching factor. The default target-only mode supports both linear and tree-shaped draft candidates. Classic rejection sampling is available for EAGLE3 and NEXTN with `--speculative-use-rejection-sampling` and requires `--speculative-eagle-topk 1`. Rejection sampling uses the draft proposal distribution and cannot be combined with non-default speculative acceptance thresholds or deterministic inference.
Argument Defaults Options Server supported
`--speculative-algorithm` `None` `EAGLE`,
`EAGLE3`,
`NEXTN`		 A2/A3 Series
`--speculative-draft-model-path`
`--speculative-draft-model`		 `None` Type: str A2/A3 Series
`--speculative-draft-model-revision` `None` Type: str,
`branch name`,
`tag name`,
`commit id`		 A2/A3 Series
`--speculative-draft-load-format` `auto` `auto`,
`dummy`		 A2/A3 Series
`--speculative-num-steps` `None` Type: int A2/A3 Series
`--speculative-eagle-topk` `None` `1` (the only supported value on Ascend NPU) A2/A3 Series
`--speculative-num-draft-tokens` `None` Type: int A2/A3 Series
`--speculative-dflash-block-size` `None` Type: int A2/A3 Series
`--speculative-accept-threshold-single` `1.0` Type: float A2/A3 Series
`--speculative-accept-threshold-acc` `1.0` Type: float A2/A3 Series
`--speculative-use-rejection-sampling` `False` bool flag
(requires `--speculative-eagle-topk 1`)		 A2/A3 Series
`--speculative-token-map` `None` Type: str; requires `--tp-size=1` A2/A3 Series
`--speculative-attention-mode` `prefill` `prefill`,
`decode`		 A2/A3 Series
`--speculative-moe-runner-backend` `None` `auto` A2/A3 Series
`--speculative-moe-a2a-backend` `None` `ascend_fuseep` (the only supported value on Ascend NPU) A2/A3 Series
`--speculative-draft-attention-backend` `None` `ascend` A2/A3 Series
`--speculative-draft-window-size` `None` Type: int A2/A3 Series
`--speculative-draft-model-quantization` `None` `unquant` (the only supported value for speculative decoding on Ascend NPU) A2/A3 Series
## Ngram speculative decoding
Argument Defaults Options Server supported
`--speculative-ngram-min-match-window-size` `1` Type: int Experimental
`--speculative-ngram-max-match-window-size` `12` Type: int Experimental
`--speculative-ngram-min-bfs-breadth` `1` Type: int Experimental
`--speculative-ngram-max-bfs-breadth` `10` Type: int Experimental
`--speculative-ngram-match-type` `BFS` `BFS`,
`PROB`		 Experimental. `BFS` uses recency-based expansion; `PROB` uses frequency-based expansion.
`--speculative-ngram-max-trie-depth` `18` Type: int Experimental
`--speculative-ngram-capacity` `10000000` Type: int Experimental
`--speculative-ngram-external-corpus-path` `None` Type: str Experimental
`--speculative-ngram-external-sam-budget` `0` Type: int Experimental
`--speculative-ngram-external-corpus-max-tokens` `10000000` Type: int Experimental
`--speculative-adaptive` `False` bool flag (set to enable) A2/A3 Series
`--speculative-adaptive-config` `None` Type: str A2/A3 Series
`--speculative-skip-dp-mlp-sync` `False` bool flag (set to enable) A2/A3 Series
## Expert parallelism
Argument Defaults Options Server supported
`--expert-parallel-size`
`--ep-size`
`--ep`		 `1` Type: int A2/A3 Series
`--moe-a2a-backend` `none` `none`,
`deepep`,
`ascend_fuseep`(It is incompatible with eplb)		 A2/A3 Series
`--moe-runner-backend` `auto` `auto`, `triton` A2/A3 Series
`--flashinfer-mxfp4-moe-precision` `default` `default`,
`bf16`,
`fp8`		 Special for GPU
`--enable-flashinfer-allreduce-fusion` `False` bool flag
(set to enable)		 Special for GPU
`--deepep-mode` `auto` `normal`,
`low_latency`,
`auto`		 A2/A3 Series
`--fuseep-mode` `2` `1`,
`2`		 A2/A3 Series
`--deepep-dispatcher-output-dtype` `auto` `auto`,
`bf16`,
`int8`
(When enabling DeepEP for a quantized model, set the dispatcher output dtype according to your model’s quantization. If the value is int8, you must also set the environment variable:DEEP\_NORMAL\_MODE\_USE\_INT8\_QUANT=1)		 A2/A3 Series
`--deepep-config` `None` Type: str Special for GPU
`--ep-num-redundant-experts` `0` Type: int A2/A3 Series
`--ep-dispatch-algorithm` `None` `static`,
`dynamic`,
`fake`		 A2/A3 Series
`--init-expert-location` `trivial` `trivial`,
``,
``,
``		 A2/A3 Series
`--enable-eplb` `False` bool flag
(set to enable)		 A2/A3 Series
`--eplb-algorithm` `deepseek` `auto`,
`deepseek`		 A2/A3 Series
`--eplb-rebalance-num-iterations` `1000` Type: int A2/A3 Series
`--eplb-rebalance-layers-per-chunk` `None` Type: int A2/A3 Series
`--eplb-min-rebalancing-utilization-threshold` `1.0` Type: float A2/A3 Series
`--expert-distribution-recorder-mode` `None` `stat`,
`stat_approx`,
`per_pass`,
`per_token`		 A2/A3 Series
`--expert-distribution-recorder-buffer-size` `None` Type: int A2/A3 Series
`--expert-balancedness-report-mode` off off, server\_log, prometheus, both A2/A3 Series
`--moe-dense-tp-size` `None` `1` A2/A3 Series
`--elastic-ep-backend` `None` `none`, `mooncake` Special for GPU
`--mooncake-ib-device` `None` Type: str Special for GPU
`--enable-waterfill` `False` bool flag (set to enable) A2/A3 Series
## Mamba cache
Argument Defaults Options Server supported
`--max-mamba-cache-size` `None` Type: int A2/A3 Series
`--mamba-ssm-dtype` `float32` `float32`,
`bfloat16`,
`float16`		 A2/A3 Series
`--mamba-full-memory-ratio` `0.9` Type: float A2/A3 Series
`--mamba-radix-cache-strategy` `auto` `auto`,
`no_buffer`,
`extra_buffer`		 A2/A3 Series
`--mamba-track-interval` `256` Type: int A2/A3 Series
## Hierarchical cache
Argument Defaults Options Server supported
`--enable-hierarchical-cache` `False` bool flag
(set to enable).
Currently, mamba cache is not supported.		 A2/A3 Series
`--hicache-ratio` `2.0` Type: float A2/A3 Series
`--hicache-size` `0` Type: int A2/A3 Series
`--hicache-write-policy` `write_through` `write_back`,
`write_through`,
`write_through_selective`		 A2/A3 Series
`--hicache-io-backend` `kernel` `kernel_ascend`,
`direct`		 A2/A3 Series
`--hicache-mem-layout` `layer_first` `page_first_direct`,
`page_first_kv_split`		 A2/A3 Series
`--hicache-storage-backend` `None` `file` A2/A3 Series
`--hicache-storage-prefetch-policy` `timeout` `best_effort`,
`wait_complete`,
`timeout`		 Special for GPU
`--hicache-storage-backend-extra-config` `None` Type: str Special for GPU
## LMCache
Argument Defaults Options Server supported
`--enable-lmcache` `False` bool flag
(set to enable)		 Special for GPU
`--lmcache-config-file` `None` Type: str Special for GPU
## Diffusion LLM
Argument Defaults Options Server supported
`--dllm-algorithm` `None` Type: str A2/A3 Series
`--dllm-algorithm-config` `None` Type: str A2/A3 Series
## Offloading (must be used with `--disable-cuda-graph`)
Argument Defaults Options Server supported
`--cpu-offload-gb` `0` Type: int A2/A3 Series
`--offload-group-size` `-1` Type: int (DeepSeek only) A2/A3 Series
`--offload-num-in-group` `1` Type: int (DeepSeek only) A2/A3 Series
`--offload-prefetch-step` `1` Type: int (DeepSeek only) A2/A3 Series
`--offload-mode` `cpu` `cpu` (DeepSeek only)
`meta` (DeepSeek only)
`sharded_gpu` (DeepSeek only, only support tp=1 dp>1)		 A2/A3 Series
## Optimization/debug options
Argument Defaults Options Server supported
`--disable-radix-cache` `False` bool flag
(set to enable)		 A2/A3 Series
`--cuda-graph-config` `None` Type: JSON A2/A3 Series
`--cuda-graph-backend-decode` `None` `full`,
`disabled`		 A2/A3 Series
`--cuda-graph-backend-prefill` `None` `disabled`
(the former `tc_piecewise` prefill backend has been removed)		 A2/A3 Series
`--cuda-graph-max-bs-decode` `None` Type: int A2/A3 Series
`--cuda-graph-max-bs-prefill` `None` Type: int A2/A3 Series
`--cuda-graph-bs-decode` `None` Type: List\[int] A2/A3 Series
`--cuda-graph-bs-prefill` `None` Type: List\[int] A2/A3 Series
`--disable-prefill-cuda-graph` `False` bool flag
(set to enable)		 A2/A3 Series
`--disable-decode-cuda-graph` `False` bool flag
(set to enable)		 A2/A3 Series
`--disable-cuda-graph` `False` bool flag
(set to enable)		 A2/A3 Series
`--disable-cuda-graph-padding` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-profile-cuda-graph` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-cudagraph-gc` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-nccl-nvls` `False` bool flag
(set to enable)		 Special for GPU
`--enable-symm-mem` `False` bool flag
(set to enable)		 Special for GPU
`--disable-flashinfer-cutlass-moe-fp4-allgather` `False` bool flag
(set to enable)		 Special for GPU
`--enable-tokenizer-batch-encode` `False` bool flag
(set to enable)		 A2/A3 Series
`--disable-tokenizer-batch-decode` `False` bool flag
(set to enable)		 A2/A3 Series
`--disable-custom-all-reduce` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-mscclpp` `False` bool flag
(set to enable)		 Special for GPU
`--pre-warm-nccl` `False` bool flag (set to enable) A2/A3 Series
`--enable-torch-symm-mem` `False` bool flag
(set to enable)		 Special for GPU
`--disable-overlap-schedule` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-mixed-chunk` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-dp-attention-local-control-broadcast` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-dp-lm-head` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-two-batch-overlap` `False` bool flag
(set to enable)		 Planned
`--enable-single-batch-overlap` `False` bool flag
(set to enable)		 Special for GPU
`--tbo-token-distribution-threshold` `0.48` Type: float Planned
`--enable-p2p-check` `False` bool flag
(set to enable)		 Special for GPU
`--triton-attention-reduce-in-fp32` `False` bool flag
(set to enable)		 Special for GPU
`--triton-attention-num-kv-splits` `8` Type: int Special for GPU
`--triton-attention-split-tile-size` `None` Type: int Special for GPU
`--delete-ckpt-after-loading` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-memory-saver` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-weights-cpu-backup` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-draft-weights-cpu-backup` `False` bool flag
(set to enable)		 A2/A3 Series
`--allow-auto-truncate` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-custom-logit-processor` `False` bool flag
(set to enable)		 A2/A3 Series
`--flashinfer-mla-disable-ragged` `False` bool flag
(set to enable)		 Special for GPU
`--disable-shared-experts-fusion` `True` bool flag
(set to enable)		 A2/A3 Series
`--disable-chunked-prefix-cache` `True` bool flag
(set to enable)		 A2/A3 Series
`--image-processor-backend` `auto` `auto`, `torchvision`, `pil` A2/A3 Series
`--disable-fast-image-processor` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-return-hidden-states` `False` bool flag
(set to enable)		 A2/A3 Series
`--enable-return-routed-experts` `False` bool flag
(set to enable)		 A2/A3 Series
`--scheduler-recv-interval` `1` Type: int A2/A3 Series
`--numa-node` `None` List\[int] A2/A3 Series
`--enable-deterministic-inference` `False` bool flag
(set to enable)		 A2/A3 Series
`--rl-on-policy-target` `None` `fsdp` A2/A3 Series
`--disable-attn-tp-gather` `False` bool flag (set to enable) A2/A3 Series
`--enable-layerwise-nvtx-marker` `False` bool flag
(set to enable)		 Special for GPU
`--enable-attn-tp-input-scattered` `False` bool flag
(set to enable)		 Experimental
`--enable-prefill-cp` `False` bool flag (set to enable) A2/A3 Series
`--cp-strategy` `None` `zigzag` A2/A3 Series
`--enable-fused-qk-norm-rope` `False` bool flag
(set to enable)		 Special for GPU
`--enable-precise-embedding-interpolation` `False` bool flag
(set to enable)		 A2/A3 Series
`--gc-threshold` `None` List\[int] A2/A3 Series
## Dynamic batch tokenizer
Argument Defaults Options Server supported
`--enable-dynamic-batch-tokenizer` `False` bool flag
(set to enable)		 A2/A3 Series
`--dynamic-batch-tokenizer-batch-size` `32` Type: int A2/A3 Series
`--dynamic-batch-tokenizer-batch-timeout` `0.002` Type: float A2/A3 Series
## Debug tensor dumps
Argument Defaults Options Server supported
`--debug-tensor-dump-output-folder` `None` Type: str A2/A3 Series
`--debug-tensor-dump-layers` `None` List\[int] A2/A3 Series
`--debug-tensor-dump-input-file` `None` Type: str A2/A3 Series
## PD disaggregation
Argument Defaults Options Server supported
`--disaggregation-mode` `null` `null`,
`prefill`,
`decode`		 A2/A3 Series
`--disaggregation-transfer-backend` `mooncake` `ascend` (default `mooncake` not supported on NPU, must be specified manually) A2/A3 Series
`--disaggregation-bootstrap-port` `8998` Type: int A2/A3 Series
`--disaggregation-ib-device` `None` Type: str Special for GPU
`--disaggregation-decode-enable-offload-kvcache` `False` bool flag
(set to enable)		 A2/A3 Series
`--num-reserved-decode-tokens` `512` Type: int A2/A3 Series
`--disaggregation-decode-polling-interval` `1` Type: int A2/A3 Series
`--optimistic-prefill-attempts` `0` Type: int A2/A3 Series
## Encode prefill disaggregation
Argument Defaults Options Server supported
`--enable-adaptive-dispatch-to-encoder` `False` bool flag
(set to enable adaptive dispatch)		 A2/A3 Series
`--encoder-only` `False` bool flag
(set to launch an encoder-only server)		 A2/A3 Series
`--language-only` `False` bool flag
(set to load weights for the language model only)		 A2/A3 Series
`--encoder-transfer-backend` `zmq_to_scheduler` `zmq_to_scheduler`,
`zmq_to_tokenizer`,
`mooncake`		 A2/A3 Series
`--encoder-urls` `[]` List\[str]
(List of encoder server urls)		 A2/A3 Series
`--encoder-bootstrap-port` `8997` Type: int A2/A3 Series
`--encoder-register-urls` `[]` List\[str] A2/A3 Series
## Custom weight loader
Argument Defaults Options Server supported
`--custom-weight-loader` `None` List\[str] A2/A3 Series
`--weight-loader-disable-mmap` `False` bool flag
(set to enable)		 A2/A3 Series
`--weight-loader-prefetch-checkpoints` `False` bool flag
(set to enable)		 A2/A3 Series
`--weight-loader-prefetch-num-threads` `4` Type: int A2/A3 Series
`--remote-instance-weight-loader-seed-instance-ip` `None` Type: str A2/A3 Series
`--remote-instance-weight-loader-seed-instance-service-port` `None` Type: int A2/A3 Series
`--remote-instance-weight-loader-send-weights-group-ports` `None` Type: JSON
list		 A2/A3 Series
`--remote-instance-weight-loader-backend` `nccl` `transfer_engine`,
`nccl`,
`modelexpress`		 A2/A3 Series
`--weight-loader-drop-cache-after-load` `False` bool flag (set to enable) A2/A3 Series
`--remote-instance-weight-loader-start-seed-via-transfer-engine` `False` bool flag
(set to enable)		 Special for GPU
## For PD-multiplexing
Argument Defaults Options Server supported
`--enable-pdmux` `False` bool flag
(set to enable)		 Special for GPU
`--pdmux-config-path` `None` Type: str Special for GPU
`--sm-group-num` `8` Type: int Special for GPU
## For multi-modal
Argument Defaults Options Server supported
`--enable-broadcast-mm-inputs-process` `False` bool flag
(set to enable)		 A2/A3 Series
`--mm-process-config` `None` Type: JSON / Dict A2/A3 Series
`--mm-enable-dp-encoder` `False` bool flag
(set to enable)		 A2/A3 Series
`--mm-processor-worker-num` `0` Type: int A2/A3 Series
`--mm-io-worker-num` `0` Type: int A2/A3 Series
`--limit-mm-data-per-request` `None` Type: JSON / Dict A2/A3 Series
## For checkpoint decryption
Argument Defaults Options Server supported
`--decrypted-config-file` `None` Type: str A2/A3 Series
`--decrypted-draft-config-file` `None` Type: str A2/A3 Series
`--enable-prefix-mm-cache` `False` bool flag
(set to enable)		 A2/A3 Series
## Forward hooks
Argument Defaults Options Server supported
`--forward-hooks` `None` Type: JSON list A2/A3 Series
`--enable-quant-communications` `False` bool flag (set to enable)
(When using DeepEP with a quantized model, you must set --deepep-dispatcher-output-dtype)		 A2/A3 Series
## msProbe
Argument Defaults Options Server supported
`--msprobe-dump-config` `None` Type: str A2/A3 Series
## Configuration file support
Argument Defaults Options Server supported
`--config` `None` Type: str A2/A3 Series
## Other params The following parameters are not supported because the required third-party components (e.g., Ktransformer, checkpoint-engine) are not compatible with the NPU.
Argument Defaults Options
`--checkpoint-engine-wait-weights-before-ready` `False` bool flag (set to enable)
`--kt-weight-path` `None` Type: str
`--kt-method` `AMXINT4` Type: str
`--kt-cpuinfer` `None` Type: int
`--kt-threadpool-count` 2 Type: int
`--kt-num-gpu-experts` `None` Type: int
`--kt-max-deferred-experts-per-token` `None` Type: int
The following parameters have known functional deficiencies in the upstream community codebase
Argument Defaults Options
`--tool-server` `None` Type: str
# Supported Models Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/reference/support_models This section describes the models supported on the Ascend NPU, including Large Language Models, Multimodal Language Models, Diffusion Language Models, Embedding Models, Reward Models and Rerank Models. Mainstream DeepSeek/Qwen/GLM series are included. You are welcome to enable various models based on your business requirements. ## Large Language Models
Models Model Family Ascend A2 Series Products Supported Ascend A3 Series Products Supported
Eco-Tech/DeepSeek-V4-Pro-w4a8-mtp DeepSeek ✅ ✅
Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp DeepSeek ✅ ✅
sgl-npu/DeepSeek-V3.1-w8a8 DeepSeek ✅ ✅
sgl-npu/DeepSeek-V3.2-W8A8 DeepSeek ✅ ✅
sgl-npu/DeepSeek-R1-0528-W8A8 DeepSeek ✅ ✅
sgl-npu/DeepSeek-V2-Lite-W8A8 DeepSeek ✅ ✅
Qwen/Qwen3.8-2.4T-A95B Qwen3.8 ✅ ✅
Qwen/Qwen3.6-35B-A3B Qwen3.6 ✅ ✅
Eco-Tech/Qwen3.6-27B-w8a8 Qwen3.6 ✅ ✅
Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp Qwen3.5 ✅ ✅
Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp Qwen3.5 ✅ ✅
Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp Qwen3.5 ✅ ✅
Eco-Tech/Qwen3.5-27B-w8a8-mtp Qwen3.5 ✅ ✅
Qwen/Qwen3.5-9B Qwen3.5 ✅ ✅
Qwen/Qwen3.5-4B Qwen3.5 ✅ ✅
Qwen/Qwen3.5-0.8B Qwen3.5 ✅ ✅
Qwen/Qwen3-30B-A3B-Instruct-2507 Qwen3 ✅ ✅
Qwen/Qwen3-32B Qwen3 ✅ ✅
Qwen/Qwen3-0.6B Qwen3 ✅ ✅
sgl-npu/Qwen3-235B-A22B-W8A8 Qwen3 ✅ ✅
Qwen/Qwen3-Next-80B-A3B-Instruct Qwen3 ✅ ✅
Eco-Tech/Qwen3-Coder-480B-A35B-Instruct-w8a8-QuaRot Qwen3 ✅ ✅
Qwen/Qwen2.5-7B-Instruct Qwen2.5 ✅ ✅
sgl-npu/QwQ-32B-W8A8 QWQ ✅ ✅
LLM-Research/Llama-4-Scout-17B-16E-Instruct Llama ✅ ✅
AI-ModelScope/Llama-3.1-8B-Instruct Llama ✅ ✅
LLM-Research/llama-2-7b Llama ✅ ✅
LLM-Research/Llama-3.2-1B-Instruct Llama ✅ ✅
mistralai/Mistral-7B-Instruct-v0.2 Mistral ✅ ✅
google/gemma-3-4b-it Gemma ✅ ✅
microsoft/Phi-4-multimodal-instruct Phi ✅ ✅
allenai/OLMoE-1B-7B-0924 OLMoE ✅ ✅
stabilityai/stablelm-2-1\_6b StableLM ✅ ✅
CohereForAI/c4ai-command-r-v01 Command-R ✅ ✅
ZhipuAI/chatglm2-6b ChatGLM ✅ ✅
LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct ExaONE 3 ✅ ✅
xverse/XVERSE-MoE-A36B XVERSE ✅ ✅
HuggingFaceTB/SmolLM-1.7B SmolLM ✅ ✅
Eco-Tech/GLM-5.2-w8a8 GLM-5 ✅ ✅
Eco-Tech/GLM-5.1-w4a8 GLM-5 ✅ ✅
Eco-Tech/GLM-5-w4a8 GLM-5 ✅ ✅
ZhipuAI/glm-4-9b-chat GLM-4 ✅ ✅
iridiumine/MiMo-V2-Flash-W8A8 MiMo ✅ ✅
XiaomiMiMo/MiMo-7B-RL MiMo ✅ ✅
arcee-ai/AFM-4.5B-Base Arcee AFM-4.5B ✅ ✅
Howeee/persimmon-8b-chat Persimmon ✅ ✅
inclusionAI/Ling-lite Ling ✅ ✅
ibm-granite/granite-3.1-8b-instruct Granite ✅ ✅
AI-ModelScope/dbrx-instruct DBRX (Databricks) ✅ ✅
baichuan-inc/Baichuan2-13B-Chat Baichuan 2 (7B, 13B) ✅ ✅
PaddlePaddle/ERNIE-4.5-21B-A3B-PT ERNIE-4.5 (4.5, 4.5MoE series) ✅ ✅
OpenBMB/MiniCPM3-4B MiniCPM (v3, 4B) ✅ ✅
sgl-npu/Kimi-K3-W4A8 Kimi ✅ ✅
Eco-Tech/Kimi-K2.6-w4a8 Kimi ✅ ✅
Eco-Tech/Kimi-K2.5-w4a8 Kimi ✅ ✅
moonshotai/Kimi-K2-Thinking Kimi ✅ ✅
eigen-ai-labs/gpt-oss-120b-bf16 GPTOSS ✅ ✅
allenai/OLMo-2-1124-7B-Instruct OLMo ✅ ✅
Eco-Tech/MiniMax-M2.5-w8a8-QuaRot MiniMax-M2.5 ✅ ✅
cyankiwi/MiniMax-M2-BF16 MiniMax-M2 ✅ ✅
upstage/SOLAR-10.7B-Instruct-v1.0 Solar ✅ ✅
bigcode/starcoder2-7b StarCoder2 ✅ ✅
arcee-ai/Trinity-Mini Trinity (Nano, Mini) ✅ ✅
OrionStarAI/Orion-14B-Base Orion (14B) ✅ ✅
EleutherAI/gpt-j-6b GPT-J (6B) ✅ ✅
ibm-granite/granite-4.0-micro IBM Granite 4.0 (Hybrid, Dense) ✅ ✅
poolside/Laguna-XS.2 Laguna XS.2 (poolside) ✅ ✅
Tencent-Hunyuan/Hy3 Hunyuan ✅ ✅
## Multimodal Language Models
Models Model Family (Variants) Ascend A2 Series Products Supported Ascend A3 Series Products Supported
Qwen/Qwen2.5-VL-3B-Instruct Qwen2.5-VL ✅ ✅
Qwen/Qwen2.5-VL-72B-Instruct Qwen2.5-VL ✅ ✅
Qwen/Qwen3-VL-30B-A3B-Instruct Qwen3-VL ✅ ✅
Qwen/Qwen3-VL-8B-Instruct Qwen3-VL ✅ ✅
Qwen/Qwen3-VL-4B-Instruct Qwen3-VL ✅ ✅
Qwen/Qwen3-VL-235B-A22B-Instruct Qwen3-VL ✅ ✅
deepseek-ai/deepseek-vl2 DeepSeek-VL2 ✅ ✅
deepseek-ai/Janus-Pro-1B Janus-Pro (1B, 7B) ✅ ✅
deepseek-ai/Janus-Pro-7B Janus-Pro (1B, 7B) ✅ ✅
OpenBMB/MiniCPM-V-2\_6 MiniCPM-V / MiniCPM-o ✅ ✅
OpenBMB/MiniCPM-o-2\_6 MiniCPM-V / MiniCPM-o ✅ ✅
google/gemma-3-4b-it Gemma 3 (Multimodal) ✅ ✅
mistralai/Mistral-Small-3.1-24B-Instruct-2503 Mistral-Small-3.1-24B ✅ ✅
microsoft/Phi-4-multimodal-instruct Phi-4-multimodal-instruct ✅ ✅
XiaomiMiMo/MiMo-VL-7B-RL MiMo-VL (7B) ✅ ✅
AI-ModelScope/llava-v1.6-34b LLaVA (v1.5 & v1.6) ✅ ✅
lmms-lab/llava-next-72b LLaVA-NeXT (8B, 72B) ✅ ✅
moonshotai/Kimi-VL-A3B-Instruct Kimi-VL (A3B) ✅ ✅
ZhipuAI/GLM-4.5V GLM-4.5V (106B) ✅ ✅
LLM-Research/Llama-3.2-11B-Vision-Instruct Llama 3.2 Vision (11B) ✅ ✅
rednote-hilab/dots.ocr DotsVLM-OCR ✅ ✅
Qwen/Qwen3-Omni-30B-A3B-Instruct Qwen3-Omni ✅ ✅
stepfun-ai/Step3-VL-10B Step3-VL (10B) ✅ ✅
deepseek-ai/DeepSeek-OCR-2 DeepSeek-OCR / OCR-2 ✅ ✅
Qwen/Qwen3-ASR-1.7B Qwen3-ASR (0.6B, 1.7B) ✅ ✅
## Embedding Models
Models Model Family Ascend A2 Series Products Supported Ascend A3 Series Products Supported
intfloat/e5-mistral-7b-instruct E5 (Llama/Mistral based) ✅ ✅
iic/gte-Qwen2-1.5B-instruct GTE-Qwen2 ✅ ✅
Qwen/Qwen3-Embedding-8B Qwen3-Embedding ✅ ✅
iic/gme-Qwen2-VL-2B-Instruct GME (Multimodal) ✅ ✅
BAAI/bge-large-en-v1.5 BGE ✅ ✅
## Reward Models
Models Model Family Ascend A2 Series Products Supported Ascend A3 Series Products Supported
Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 Llama3.1 Reward ✅ ✅
Shanghai\_AI\_Laboratory/internlm2-7b-reward InternLM 2 Reward ✅ ✅
Qwen/Qwen2.5-Math-RM-72B Qwen2.5 Reward - Math ✅ ✅
Howeee/Qwen2.5-1.5B-apeach Qwen2.5 Reward - Sequence ✅ ✅
## Rerank Models
Models Model Family Ascend A2 Series Products Supported Ascend A3 Series Products Supported
BAAI/bge-reranker-v2-m3 BGE-Reranker ✅ ✅
Qwen/Qwen3-Reranker-8B Qwen3-Reranker (decoder-only yes/no) ✅ ✅
Qwen/Qwen3-VL-Reranker-2B Qwen3-VL-Reranker (multimodal yes/no) ✅ ✅
# CPU Servers Source: https://docs.sglang.io/docs/hardware-platforms/cpu_server The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on CPU servers. SGLang is enabled and optimized on the CPUs equipped with Intel® AMX® Instructions, which are 4th generation or newer Intel® Xeon® Scalable Processors. A number of popular LLMs are optimized and run efficiently on CPU, including the most notable open-source models like Llama series, Qwen series, and DeepSeek series like DeepSeek-R1 and DeepSeek-V3.1-Terminus. Please check the [SGLang Cookbook pages](https://docs.sglang.io/cookbook/intro) in which the support status and example commands can be found. ## Installation ### Install Using Docker It is recommended to use Docker for setting up the SGLang environment. #### Pull from Docker Hub Pull the prebuilt docker image of SGLang package releases from `lmsysorg/sglang` repository. The [CPU image tags](https://hub.docker.com/r/lmsysorg/sglang/tags?name=xeon) end with `xeon` suffix. The image pulling command is like: ```bash Command theme={null} docker pull lmsysorg/sglang:v0.5.13-xeon ``` #### Build from Dockerfile A [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile) is provided to facilitate the installation from latest source code. Replace `` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens). ```bash Command theme={null} # Clone the SGLang repository git clone https://github.com/sgl-project/sglang.git cd sglang/docker # Build the docker image docker build -t sglang-cpu:latest -f xeon.Dockerfile . # Initiate a docker container docker run \ -it \ --privileged \ --ipc=host \ --network=host \ -v /dev/shm:/dev/shm \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -p 30000:30000 \ -e "HF_TOKEN=" \ sglang-cpu:latest /bin/bash ``` ### Install From Source If you prefer to install SGLang in a bare metal environment, the setup process is as follows: Please install the required packages and libraries beforehand if they are not already present on your system. You can refer to the Ubuntu-based installation commands in [the Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile#L7) for guidance. 1. Install `uv` package manager, then create and activate a virtual environment: ```bash Command theme={null} # Taking '/opt' as the example uv env folder, feel free to change it as needed cd /opt curl -LsSf https://astral.sh/uv/install.sh | sh source $HOME/.local/bin/env uv venv --python 3.12 source .venv/bin/activate ``` 2. Create a config file to direct the installation channel (a.k.a. index-url) of `torch` related packages: ```bash Command theme={null} vim .venv/uv.toml ``` Press 'a' to enter insert mode of `vim`, paste the following content into the created file ```file theme={null} [[index]] name = "torch" url = "https://download.pytorch.org/whl/cpu" [[index]] name = "torchvision" url = "https://download.pytorch.org/whl/cpu" [[index]] name = "torchaudio" url = "https://download.pytorch.org/whl/cpu" [[index]] name = "triton" url = "https://download.pytorch.org/whl/cpu" ``` Save the file (in `vim`, press 'esc' to exit insert mode, then ':x+Enter'), and set it as the default `uv` config. ```bash Command theme={null} export UV_CONFIG_FILE=/opt/.venv/uv.toml ``` 3. Clone the `sglang` source code and build the packages ```bash Command theme={null} # Clone the SGLang code git clone https://github.com/sgl-project/sglang.git cd sglang git checkout # Use dedicated toml file cd python cp pyproject_cpu.toml pyproject.toml # Install SGLang dependent libs, and build SGLang main package uv pip install --upgrade pip setuptools uv pip install . # Build the CPU backend kernels cd sglang/kernels/aot cp pyproject_cpu.toml pyproject.toml uv pip install . ``` 4. Set the required environment variables ```bash Command theme={null} export SGLANG_USE_CPU_ENGINE=1 # Set 'LD_LIBRARY_PATH' and 'LD_PRELOAD' to ensure the libs can be loaded by sglang processes export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu export LD_PRELOAD=${LD_PRELOAD}:/opt/.venv/lib/libiomp5.so:${LD_LIBRARY_PATH}/libtcmalloc.so.4:${LD_LIBRARY_PATH}/libtbbmalloc.so.2 ``` Notes: * Note that the environment variable `SGLANG_USE_CPU_ENGINE=1` is required to enable the SGLang service with the CPU engine. * If you encounter code compilation issues during the `sgl-kernel` building process, please check your `gcc` and `g++` versions and upgrade them if they are outdated. It is recommended to use `gcc-13` and `g++-13` as they have been verified in the official Docker container. * The system library path is typically located in one of the following directories: `~/.local/lib/`, `/usr/local/lib/`, `/usr/local/lib64/`, `/usr/lib/`, `/usr/lib64/` and `/usr/lib/x86_64-linux-gnu/`. In the above example commands, `/usr/lib/x86_64-linux-gnu` is used. Please adjust the path according to your server configuration. * It is recommended to add the following to your `~/.bashrc` file to avoid setting these variables every time you open a new terminal: ```bash Command theme={null} source .venv/bin/activate export SGLANG_USE_CPU_ENGINE=1 export LD_LIBRARY_PATH= export LD_PRELOAD= ``` ## Launch of the Serving Engine Example command to launch SGLang serving: ```bash Launch Server theme={null} sglang serve \ --model-path \ --trust-remote-code \ --disable-overlap-schedule \ --device cpu \ --host 0.0.0.0 \ --tp 6 ``` Notes: 1. For running W8A8 quantized models, please add the flag `--quantization w8a8_int8`. 2. The flag `--tp 6` specifies that tensor parallelism will be applied using 6 ranks (TP6). The number of TP specified is how many TP ranks will be used during the execution. On a CPU platform, a TP rank means a sub-NUMA cluster (SNC). Usually we can get the SNC information (How many available) from the Operating System with e.g. `lscpu` command. If the specified TP rank number differs from the total SNC count, the system will automatically utilize the first `n` SNCs. Note that `n` cannot exceed the total SNC number, doing so will result in an error. `SGLANG_CPU_OMP_THREADS_BIND` allows explicit control of CPU cores for each tensor parallel (TP) rank. **example 1**: Run SGLang service with TP=6, using the first 40 cores of each SNC on a Xeon® 6980P server, which has 43-43-42 cores on the 3 SNCs of a socket, we should set: ```bash Command theme={null} export SGLANG_CPU_OMP_THREADS_BIND="0-39|43-82|86-125|128-167|171-210|214-253" ``` This configuration is equivalent to: * rank 0: `numactl -C 0-39 -m 0` * rank 1: `numactl -C 43-82 -m 1` * rank 2: `numactl -C 86-125 -m 2` * rank 3: `numactl -C 128-167 -m 3` * rank 4: `numactl -C 171-210 -m 4` * rank 5: `numactl -C 214-253 -m 5` **example 2**: Run SGLang service with TP=2, using 96 cores cross 3 SNCs on a Xeon® 6972P server, which has 32-32-32 cores on the 3 SNCs in a socket, we should set: ```bash Command theme={null} export SGLANG_CPU_OMP_THREADS_BIND="0-95|96-191" ``` This configuration is equivalent to: * rank 0: `numactl -C 0-95 -m 0-2` * rank 1: `numactl -C 96-191 -m 3-5` Please beware that with SGLANG\_CPU\_OMP\_THREADS\_BIND set, the available memory amounts of the ranks may not be determined in prior. You may need to set proper `--max-total-tokens` to avoid the out-of-memory error. 3. For optimizing decoding with torch.compile, please add the flag `--enable-torch-compile`. To specify the maximum batch size when using `torch.compile`, set the flag `--torch-compile-max-bs`. For example, `--enable-torch-compile --torch-compile-max-bs 4` means using `torch.compile` and setting the maximum batch size to 4. 4. A warmup step is automatically triggered when the service is started. The server is ready when you see the log `The server is fired up and ready to roll!`. ## Benchmarking with Requests You can benchmark the performance via the `bench_serving` script. Run the command in another terminal. An example command would be: ```bash Run Benchmark theme={null} python -m sglang.bench_serving \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1 \ --request-rate inf \ --random-range-ratio 1.0 ``` Detailed parameter descriptions are available via the command: ```bash Benchmark Help theme={null} python -m sglang.bench_serving -h ``` Additionally, requests can be formatted using [the OpenAI Completions API](../basic_usage/openai_api_completions) and sent via the command line (e.g., using `curl`) or through your own scripts. ## Example Usage Commands Large Language Models can range from fewer than 1 billion to several hundred billion parameters. Dense models larger than 20B are expected to run on flagship 6th Gen Intel® Xeon® processors with dual sockets and a total of 6 sub-NUMA clusters. Dense models of approximately 10B parameters or fewer, or MoE (Mixture of Experts) models with fewer than 10B activated parameters, can run on more common 4th generation or newer Intel® Xeon® processors, or utilize a single socket of the flagship 6th Gen Intel® Xeon® processors. ### Example: Running DeepSeek-V3.1-Terminus An example command to launch service of W8A8\_INT8 DeepSeek-V3.1-Terminus on a Xeon® 6980P server: ```bash W8A8_INT8 theme={null} sglang serve \ --model-path IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \ --trust-remote-code \ --disable-overlap-schedule \ --device cpu \ --quantization w8a8_int8 \ --enable-torch-compile \ --torch-compile-max-bs 4 \ --host 0.0.0.0 \ --tp 6 ``` Similarly, an example command to launch service of FP8 DeepSeek-V3.1-Terminus would be: ```bash FP8 theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-V3.1-Terminus \ --trust-remote-code \ --disable-overlap-schedule \ --device cpu \ --enable-torch-compile \ --torch-compile-max-bs 4 \ --host 0.0.0.0 \ --tp 6 ``` Note: Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment. The value `4` in the examples is illustrative. ### Example: Running Llama-3.2-3B An example command to launch service of Llama-3.2-3B with BF16 precision: ```bash BF16 theme={null} sglang serve \ --model-path meta-llama/Llama-3.2-3B-Instruct \ --trust-remote-code \ --disable-overlap-schedule \ --device cpu \ --enable-torch-compile \ --torch-compile-max-bs 16 \ --host 0.0.0.0 \ --tp 3 ``` The example command to launch service of W8A8\_INT8 version of Llama-3.2-3B: ```bash W8A8_INT8 theme={null} sglang serve \ --model-path RedHatAI/Llama-3.2-3B-quantized.w8a8 \ --trust-remote-code \ --disable-overlap-schedule \ --device cpu \ --quantization w8a8_int8 \ --enable-torch-compile \ --torch-compile-max-bs 16 \ --host 0.0.0.0 \ --tp 3 ``` Note: The `--torch-compile-max-bs` and `--tp` settings are examples that should be adjusted for your setup. For instance, use `--tp 3` to utilize 1 socket with 3 sub-NUMA clusters on an Intel® Xeon® 6980P server. Once the server have been launched, you can test it using the `bench_serving` command or create your own commands or scripts following [the benchmarking example](#benchmarking-with-requests). # Moore Threads GPUs Source: https://docs.sglang.io/docs/hardware-platforms/mthreads_gpu This document describes how to run SGLang on Moore Threads GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues). SGLang uses the MUSA runtime through [`torch_musa`](https://github.com/MooreThreads/torch_musa) and [`torchada`](https://github.com/MooreThreads/torchada). The `python[all_musa]` extra installs the MUSA torch, Triton, TileLang, MATE, and runtime Python stack from the Moore Threads package index, with public PyPI as the fallback for common Python dependencies. ## Prerequisites Install the Moore Threads driver and MUSA toolkit before installing SGLang. For MTT S5000, follow the [official driver installation guide](https://docs.mthreads.com/driver-linux-server/driver-linux-server-doc-online/MTT_S5000/install_guide). ## Install SGLang You can install SGLang using one of the methods below. ### Install from Source Start from an environment with the prerequisites above installed. ```bash Command theme={null} git clone https://github.com/sgl-project/sglang.git cd sglang python -m pip install --upgrade pip "setuptools<82" wheel cp python/pyproject_other.toml python/pyproject.toml python -m pip install -e "python[all_musa]" \ --index-url https://dl.mthreads.com/repo/api/pypi/pypi/simple \ --extra-index-url https://pypi.org/simple \ --trusted-host dl.mthreads.com \ --no-build-isolation # Ensure the MUSA Triton wheel (with triton.backends.mtgpu) is installed. python -m pip install --no-cache-dir --force-reinstall --no-deps \ --index-url https://dl.mthreads.com/repo/api/pypi/pypi/simple \ --trusted-host dl.mthreads.com \ triton==3.2.0 python -c 'import triton.backends.mtgpu' cd python/sglang/kernels/aot cp pyproject_musa.toml pyproject.toml MTGPU_TARGET=mp_31 python setup_musa.py install ``` ### Install Using Docker The Dockerfile installs the MUSA runtime and Python stack declared by `python/pyproject_other.toml`. ```bash Command theme={null} git clone https://github.com/sgl-project/sglang.git cd sglang docker build -f docker/musa.Dockerfile -t sglang:main-musa . ``` Run the image with MUSA devices exposed by the host container toolkit. ```bash Command theme={null} alias drun='docker run -it --rm --network=host \ --env MTHREADS_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ --env MTHREADS_DRIVER_CAPABILITIES=all \ --shm-size=32g \ -v ~/.cache/huggingface:/root/.cache/huggingface' drun sglang:main-musa \ python3 -c 'import triton.backends.mtgpu; print("MUSA Triton backend OK")' ``` Launch the server: ```bash Command theme={null} drun sglang:main-musa \ python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --host 0.0.0.0 \ --port 30000 ``` When the server displays `The server is fired up and ready to roll!`, startup is successful. ## Verify You can send an OpenAI-compatible request to the engine: ```bash Command theme={null} curl http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 32 }' ``` Or run a benchmark: ```bash Command theme={null} drun sglang:main-musa \ python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 4000 \ --random-input 128 \ --random-output 128 ``` # NVIDIA GPUs Source: https://docs.sglang.io/docs/hardware-platforms/nvidia-gpus Please refer to the [Installation Guide](../get-started/install) to get started with SGLang on NVIDIA GPUs. # NVIDIA Jetson Orin Source: https://docs.sglang.io/docs/hardware-platforms/nvidia_jetson Guide for installing and running SGLang on NVIDIA Jetson Orin devices. ## Prerequisites Before starting, ensure the following: * [**NVIDIA Jetson AGX Orin Devkit**](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/) is set up with **JetPack 6.1** or later. * **CUDA Toolkit** and **cuDNN** are installed. * Verify that the Jetson AGX Orin is in **high-performance mode**: ```bash theme={null} sudo nvpmodel -m 0 ``` *** ## Installing and running SGLang with Jetson Containers Clone the jetson-containers github repository: ```bash theme={null} git clone https://github.com/dusty-nv/jetson-containers.git ``` Run the installation script: ```bash theme={null} bash jetson-containers/install.sh ``` Build the container image: ```bash theme={null} jetson-containers build sglang ``` Run the container: ``` jetson-containers run $(autotag sglang) ``` Or you can also manually run a container with this command: ``` docker run --runtime nvidia -it --rm --network=host IMAGE_NAME ``` *** ## Running Inference Launch the server: ```bash theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-Distill-Llama-8B \ --device cuda \ --dtype half \ --attention-backend flashinfer \ --mem-fraction-static 0.8 \ --context-length 8192 ``` The quantization and limited context length (`--dtype half --context-length 8192`) are due to the limited computational resources in [Nvidia jetson kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/). A detailed explanation can be found in [Server Arguments](../advanced_features/server_arguments). After launching the engine, refer to [Chat completions](../basic_usage/openai_api_completions#usage) to test the usability. *** ## Structured output with XGrammar Please refer to [SGLang doc structured output](../advanced_features/structured_outputs). *** Thanks to the support from [Nurgaliyev Shakhizat](https://github.com/shahizat), [Dustin Franklin](https://github.com/dusty-nv) and [Johnny Núñez Cano](https://github.com/johnnynunez). ## References * [NVIDIA Jetson AGX Orin Documentation](https://developer.nvidia.com/embedded/jetson-agx-orin) # Hardware Platforms Source: https://docs.sglang.io/docs/hardware-platforms/overview Platform-specific guides for running SGLang on GPUs, TPUs, NPUs, CPUs, and more. * [NVIDIA GPUs](./nvidia-gpus) * [AMD GPUs](./amd_gpu) * [Ascend NPUs](./ascend-npus/getting-started/installation) * [CPU Server](./cpu_server) * [NVIDIA Jetson Orin](./nvidia_jetson) * [TPU](./tpu) * [XPU](./xpu) * [Moore Threads GPUs](./mthreads_gpu) # SGLang Plugin System Source: https://docs.sglang.io/docs/hardware-platforms/plugin ## Overview Allows hardware vendors and developers to extend SGLang **without modifying the main repository code**. The framework provides two plugin types, both discovered via Python's standard `setuptools` entry\_points:
Plugin Type Entry Point Groups Purpose
Hardware Platform Plugin sglang.srt.platforms
sglang.multimodal\_gen.platforms		 Register a custom hardware platform (device operations, KV cache pools, attention backends, graph capture, compilation backends, etc.)
General Plugin sglang.srt.plugins
sglang.multimodal\_gen.plugins		 Inject hooks (before/after/around/replace) into any function/method, or replace entire classes
### Principles * **Non-intrusive**: Built-in platforms remain the fallback when no OOT platform activates. * **Install-time discovery**: Plugins are discovered from Python entry points after installation. * **Environment variable control**: `SGLANG_PLATFORM` selects an SRT platform, `SGLANG_DIFFUSION_PLATFORM_OVERRIDE` selects a diffusion platform, and `SGLANG_PLUGINS` filters general plugins in either hook group. ### Current scope The platform plugin system targets **out-of-tree (OOT) hardware platforms**. Diffusion support is experimental and covers the seams documented below, not every device-specific branch. ## Architecture ### Runtime-specific platform interfaces SRT and diffusion have separate platform base classes and platform identity types in `sglang.srt.platforms` and `sglang.multimodal_gen.runtime.platforms`. A package supporting both runtimes should define separate platform classes and use each runtime's own `PlatformEnum.OOT` value. Each platform entry point resolves to a zero-argument activation callback: ```python theme={null} def activate() -> str | None: """Return the platform class qualname when this hardware is available.""" ``` Return a fully qualified class name when the provider can run, or `None` otherwise. Keep activation import-safe: do not access `current_platform` or initialize runtime or device state. Put required backend setup in the platform's `init_backend()` method and reserve general plugins for hooks that the platform interface cannot express. Explicit selection enumerates entry-point metadata and imports only the selected callback. Automatic selection invokes installed platform callbacks to determine which provider is active. Diffusion reserves `cpu`, `cuda`, `rocm`, `xpu`, `mps`, `npu`, and `musa`. Automatic discovery also rejects duplicate entry-point names; explicit selection validates only the selected name and does not import unrelated providers. #### SRT selection `current_platform` is a **lazy singleton** in `sglang.srt.platforms`. On first access it resolves the active platform through the following priority chain: ```text theme={null} entry_points("sglang.srt.platforms") → Enumerate ALL plugins by name (metadata only) │ ├─ SGLANG_PLATFORM set (front-loading filter): │ ├─ Name not found in discovered → RuntimeError │ ├─ activate() returns non-None → load that platform │ └─ activate() returns None → RuntimeError (hardware unavailable) │ └─ SGLANG_PLATFORM unset (auto-discover, activate all): ├─ 0 activated + SGLANG_USE_CPU_ENGINE=1 → fallback CpuSRTPlatform ├─ 0 activated + CUDA available → fallback CudaSRTPlatform ├─ 0 activated + ROCm available → fallback RocmSRTPlatform ├─ 0 activated + XPU available → fallback XpuSRTPlatform ├─ 0 activated + none of the above → fallback base SRTPlatform ├─ 1 activated → use it └─ N activated → RuntimeError (must set SGLANG_PLATFORM) ``` #### Diffusion selection SGLang Diffusion resolves its platform in this order: 1. When `SGLANG_DIFFUSION_PLATFORM_OVERRIDE` names `cpu`, `cuda`, `rocm`, `mps`, `npu`, or `musa`, select that built-in platform without hardware probing. XPU remains automatic-only, preserving the existing selector behavior. 2. When it contains another name, load only the matching `sglang.multimodal_gen.platforms` entry point. An unknown name or a callback that returns `None` is an error. 3. When it is unset, activate installed diffusion platform plugins. No active plugin continues to built-in detection, one selects that plugin, and multiple active plugins raise an error that asks you to set the selector. An activation callback that raises aborts startup rather than falling back, so a broken vendor runtime cannot silently run the job on a built-in platform. 4. Try built-in platforms in order: MPS, XPU, ROCm, CUDA, NPU, MUSA, then CPU. The existing override variable is therefore the single explicit selector for supported built-in aliases and OOT entry-point names. Selection resolves lazily, the first time anything in a process touches `current_platform`, so it needs no call site and happens in every process automatically. SGLang Diffusion also records which distribution supplied the selected platform, and skips the hooks of every other installed platform package. #### Required platform initialization vs. optional hooks The two mechanisms have different failure semantics, and it matters which one you use: | | Platform contract | General plugin | | - | - | - | | Entry-point group | `sglang.multimodal_gen.platforms` | `sglang.multimodal_gen.plugins` | | Delivery | methods on your `Platform` subclass | hooks that monkey-patch a target | | Activation | lazy `current_platform`, plus guarded `init_backend()` in each worker | registration with `load_plugins()`, then explicit `apply_plugin_hooks()` per process | | A failure | aborts startup | aborts startup if the plugin ships in the selected platform's distribution; otherwise logged | Anything your hardware needs in order to be correct belongs on the `Platform` subclass, so a broken platform cannot silently serve. Reach for a general plugin only when the `Platform` interface has no seam for what you need — and please report that gap. A plugin shipped in the selected platform's own distribution is treated as part of that platform's contract: a failure to load it, to run its callback, or to apply any hook it registered aborts startup rather than leaving the platform half-initialized. Plugins from any other installed package stay best-effort, so a broken third party cannot take the server down. An explicit `SGLANG_PLUGINS` allowlist can disable any general plugin, including one from the selected platform package; required hardware setup therefore belongs in `init_backend()`. ### Plugin Loading Flow Each runtime has a process-local hook registry. SRT retains its single `load_plugins()` activation step. Diffusion separates registration (`load_plugins()`) from target resolution (`apply_plugin_hooks()`), because resolving a dotted hook target can import that target's entire module graph. Both runtimes honor `SGLANG_PLUGINS`. The loader is called at these SRT entry points:
Call Site Process Timing
cli/serve.py serve() Main Before prepare\_server\_args()
launch\_server.py **main** Main Before prepare\_server\_args()
engine.py \_launch\_subprocesses() Main Before server\_args.check\_server\_args()
scheduler.py run\_scheduler\_process() Subprocess Before Scheduler() construction
> **Note**: Diffusion plugin registration and hook application each run once per process. Spawned subprocesses start from a blank interpreter and establish their own hooks — nothing the parent patched survives the spawn boundary. SRT's `load_plugins()` performs both phases in one call. ```text theme={null} load_plugins() ├── _get_excluded_dists() → compute dists to skip (via SGLANG_PLATFORM) ├── load_plugins_by_group("sglang.srt.plugins", → discover entry_points, filter by SGLANG_PLUGINS │ excluded_dists=...) skip plugins from unselected platform packages ├── for each plugin: → set _current_plugin_source context var │ func() side effects (register hooks with source tracking) └── HookRegistry.apply_hooks() → monkey-patch targets ``` Diffusion launchers call `apply_plugin_hooks()`, which first performs registration if necessary and then resolves and patches targets. Scheduler children use a stricter lifecycle in `runtime/managers/worker_bootstrap.py`: ```text theme={null} spawn unpickles SchedulerProcessSpec (stdlib types + opaque ServerArgs bytes) -> initialize_current_platform() -> load_plugins() # discover callbacks and register hooks only -> apply_plugin_hooks() # target imports are allowed from here onward -> materialize ServerArgs -> resolve and invoke the patched run_scheduler_process ``` HTTP children likewise finish plugin registration and hook application before importing `runtime.launch_server` or materializing `ServerArgs`. This matters for class replacements and other hooks whose effects cannot be retroactively applied to classes or registrations created while importing the server module graph. The bootstrap module is the `mp.Process` target and imports no diffusion runtime module at module scope. `ServerArgs` is serialized inside `ServerArgsPayload`, so the multiprocessing unpickler cannot import pipeline configuration modules before the target starts. Workers always use a local `spawn` context, independent of an embedding application's global multiprocessing setting. This ordering makes backend initialization precede plugin callback imports, hook-target resolution, `ServerArgs` materialization, and worker imports. A platform activation module necessarily loads before its own `init_backend()`; activation modules must therefore remain import-safe. Prefer platform methods and registries for required behavior. Each activation phase runs once per process behind a lock: a re-entrant call from a plugin callback returns and lets the outer call finish, another thread waits for it, and a failure is terminal. A callback must not hand activation to a second thread and join it — that deadlocks. #### Offline scripts and the spawn boundary `spawn` re-executes the launching script's module scope in every child *before* it unpickles the target's arguments, so an offline script's own imports run ahead of that child's platform initialization and `ServerArgsPayload` cannot help. The supported script layout is: ```python theme={null} from sglang.multimodal_gen import DiffGenerator # a proxy: imports nothing yet if __name__ == "__main__": generator = DiffGenerator.from_pretrained(model_path="...") ``` `DiffGenerator` on the `sglang.multimodal_gen` facade is a lazy proxy, so binding it at module scope costs no diffusion import and the child still reaches `initialize_current_platform()` with a clean module table. Module scope also stays open to `envs` and `runtime.platforms`, including `runtime.platforms.plugins`. These modules remain import-safe so a plugin can subclass `Platform` and register hooks before backend initialization. Everything else, including `SamplingParams` and `PipelineConfig` from the same facade, belongs inside the `if __name__ == "__main__":` guard or inside the function that uses it. A child that finds runtime modules already imported names them in a warning: hook application can still patch those modules, but classes and registrations created while importing them are already past reach. *** ## Plugin Type 1: Hardware Platform Plugin ### Description A hardware platform plugin registers an SRT `SRTPlatform` subclass, a diffusion `Platform` subclass, or both. The selected class tells that runtime how to interact with a specific hardware backend. ### SRT quick start **1. Create a minimal package:** ``` my_platform_plugin/ ├── pyproject.toml └── my_platform_plugin/ ├── __init__.py # activate() function ├── device.py # MyDeviceMixin └── platform.py # MySRTPlatform ``` **2. `pyproject.toml`:** ```toml theme={null} [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" [project] name = "my-platform-plugin" version = "0.1.0" [project.entry-points."sglang.srt.platforms"] my_device = "my_platform_plugin:activate" ``` **3. `__init__.py`** — activation function: ```python theme={null} def activate(): """Return fully-qualified class name to activate, or None to skip.""" if _my_device_is_available(): return "my_platform_plugin.platform.MySRTPlatform" return None ``` **4. `device.py`** — device mixin: ```python theme={null} from sglang.srt.platforms.device_mixin import DeviceMixin, PlatformEnum class MyDeviceMixin(DeviceMixin): _enum = PlatformEnum.OOT device_name = "my_device" device_type = "my_device" # torch device type def set_device(self, device) -> None: ... def get_device_name(self, device_id=0) -> str: ... def get_device_total_memory(self, device_id=0) -> int: ... def get_current_memory_usage(self, device=None) -> float: ... def get_device_capability(self, device_id=0): ... def get_torch_distributed_backend_str(self) -> str: ... ``` **5. `platform.py`** — SRT platform: ```python theme={null} from sglang.srt.platforms.interface import PlatformCapabilities, SRTPlatform from my_platform_plugin.device import MyDeviceMixin class MySRTPlatform(SRTPlatform, MyDeviceMixin): capabilities = PlatformCapabilities(supports_triton=False, graph_capture=True) def get_default_attention_backend(self) -> str: ... # ... override other methods as needed; vendor-private helpers start with `_` ``` Every public name on the subclass (and on any mixin it composes) must exist on `SRTPlatform` with a compatible signature. The check runs when the class is defined, so a hook that core renamed or never had raises `TypeError` naming it instead of silently never being called. **6. Install and verify:** ```bash theme={null} pip install -e my_platform_plugin/ python -c "from sglang.srt.platforms import current_platform; print(current_platform)" ``` ### SRT platform interface reference #### Identity Queries (from DeviceMixin)
Method Default Description
is\_cuda() Based on \_enum Whether this is an NVIDIA CUDA platform
is\_rocm() Based on \_enum Whether this is an AMD ROCm platform
is\_npu() Based on \_enum Whether this is a Huawei NPU platform
is\_cpu() Based on \_enum Whether this is a CPU-only platform
is\_xpu() Based on \_enum Whether this is an Intel XPU platform
is\_musa() Based on \_enum Whether this is a Moore Threads MUSA platform
is\_cuda\_alike() CUDA+ROCM+MUSA True if the hardware supports CUDA-like APIs
is\_out\_of\_tree() True for OOT Automatically detected based on \_enum = PlatformEnum.OOT
#### Device Operations (from DeviceMixin) > Methods annotated **\[Active]** are called by SGLang core through `current_platform` — OOT implementations take effect immediately. > Methods annotated **\[Planned]** are reserved interfaces — SGLang core still uses hardcoded calls (e.g. `torch.cuda.empty_cache()`). OOT implementations will NOT take effect until the core is migrated in a future PR.
Method Default Status Description
get\_device(local\_rank) raise NotImplementedError Planned Return torch.device for a given local rank
set\_device(device) raise NotImplementedError Planned Set the current device
get\_device\_name(device\_id) raise NotImplementedError Planned Get human-readable device name
get\_device\_uuid(device\_id) raise NotImplementedError Planned Get unique device identifier
get\_device\_capability(device\_id) raise NotImplementedError Planned Get DeviceCapability(major, minor). None if N/A
empty\_cache() pass Planned Release cached device memory
synchronize() pass Planned Synchronize device operations
get\_device\_total\_memory(device\_id) raise NotImplementedError Active Get total device memory in bytes
get\_available\_memory(device\_id) raise NotImplementedError Planned Return (free\_bytes, total\_bytes)
get\_current\_memory\_usage(device) raise NotImplementedError Active Get current peak memory usage in bytes
is\_pin\_memory\_available(device=None) False Active Whether pinned host memory is available for a target device
get\_torch\_distributed\_backend\_str() raise NotImplementedError Planned Distributed backend string (e.g. "nccl", "hccl")
get\_communicator\_class() None Planned Platform-specific communicator class
inference\_mode() torch.inference\_mode(True) Planned Return inference mode context manager
seed\_everything(seed) Set random/np/torch seeds Planned Set random seeds for reproducibility
verify\_quantization(quant) pass Planned Validate quantization method support
get\_cpu\_architecture() Auto-detect x86/arm Planned Detect CPU architecture (CpuArchEnum)
#### Types (from DeviceMixin)
Type Description
PlatformEnum Enumeration of platform types: CUDA, ROCM, CPU, XPU, MUSA, NPU, TPU, MPS, OOT, UNSPECIFIED
CpuArchEnum CPU architecture: X86, ARM, UNSPECIFIED
DeviceCapability NamedTuple(major, minor) with comparison support. Methods: as\_version\_str(), to\_int()
#### Capabilities (from SRTPlatform) `capabilities` is a frozen `PlatformCapabilities` class attribute. Core reads it before constructing anything and picks a code path the platform can serve; every field defaults to the conservative answer.
Field Default What core does with it
supports\_triton False When False, the paged allocator and the req-to-token writers use their torch-native paths instead of Triton kernels.
graph\_capture False Runs the decode graph runner (get\_graph\_runner\_cls()).
hicache\_device\_kernels False The sgl\_kernel HiCache transfer / write-back kernels are available; otherwise host pools skip their staging buffers.
#### Subsystem Factory Methods (from SRTPlatform)
Method Default Description
get\_default\_attention\_backend() raise NotImplementedError Default attention backend name
get\_graph\_runner\_cls() return None Graph Runner class
get\_kv\_pool\_cls(kind=...) return None Return this platform's KV pool class for kind in "mha" / "mla" / "dsa", or None for the in-tree class. The class must subclass MHATokenToKVPool / MLATokenToKVPool / DSATokenToKVPool; core constructs it with exactly the keyword arguments the in-tree class receives, at every site the in-tree class is built (standalone, inside the SWA composite, as the hybrid-linear full-attention leaf). Override \_create\_buffers to change how the backing storage is carved -- addressing, index translation and PD registration stay core's. If you override \_create\_buffers entirely you also own \_kv\_buffer\_descs (PD transfer and post-capture backing read it); override \_create\_buffers\_normal or \_kv\_buffer\_shapes instead to keep the base descriptors, which require the leading buffer axis to be tokens or pages. The in-tree pool allocates on device and is correct on any torch device, so most platforms return None.
get\_paged\_allocator\_cls() return None Paged allocator class, honored at every paged-allocator construction site. With capabilities.supports\_triton=False the in-tree allocator already uses torch-native kernels, so most platforms return None. At page\_size == 1 core uses the flat allocator on every platform.
get\_quantization\_config(quantization) return None Return hardware-specific quantization config for the specific quantization scheme, raise an error if not supported or return None to use the default config.
get\_compile\_backend(mode) "inductor" Compilation backend string
get\_dispatch\_key\_name() "native" BaseFusedOp (fused-op) dispatch key name
#### Lifecycle Hooks (from SRTPlatform)
Method Invocation Timing Purpose
apply\_server\_args\_defaults(server\_args) After ServerArgs parsing, in **post\_init** Set platform-specific defaults
init\_backend() In each worker, before model construction One-time backend initialization
post\_load\_model(model) In each worker, once the model's weights are loaded Relocate parameters, swap quant methods or install forward hooks on an in-tree model class; mutate model in place
### Platform and plugin environment variables
Variable Description
SGLANG\_PLATFORM Select the platform plugin by entry\_point name (e.g. kunlun, demo\_cuda). When set, only the named plugin's activate() is called (front-loading filter) — other plugins are not touched. Additionally, general plugins (sglang.srt.plugins) from unselected platform packages are automatically skipped to avoid importing their dependencies. Required when multiple plugins would activate. Errors if the name is not found or if the plugin's hardware is unavailable.
SGLANG\_PLUGINS Comma-separated whitelist of general plugin names to load from either hook group. It also filters automatic SRT platform discovery when SGLANG\_PLATFORM is unset; explicit platform selection ignores it.
### Add diffusion support A package may support SRT, SGLang Diffusion, or both. Diffusion uses a zero-argument `Platform` subclass and separate platform and hook entry points: ```toml theme={null} [project.entry-points."sglang.multimodal_gen.platforms"] my_device = "my_platform_plugin:activate_diffusion" [project.entry-points."sglang.multimodal_gen.plugins"] my_device = "my_platform_plugin:register_hooks" ``` Keep activation import-safe and return the fully qualified class name only when the backend is available: ```python theme={null} def activate_diffusion() -> str | None: try: import my_device_runtime except ImportError: return None if not my_device_runtime.is_available(): return None return "my_platform_plugin.diffusion_platform.MyDiffusionPlatform" ``` The referenced class must be zero-argument constructible: ```python theme={null} from sglang.multimodal_gen.runtime.platforms import Platform, PlatformEnum class MyDiffusionPlatform(Platform): _enum = PlatformEnum.OOT device_name = "my_device" device_type = "my_device" dispatch_key = "PrivateUse1" def get_dispatch_key_name(self) -> str: return "my_device" ``` Set `dispatch_key` to the PyTorch dispatcher key used by direct `torch.library` registrations. Keep it separate from `get_dispatch_key_name()`, which selects `CustomOp` implementations. Bring up the remaining contract in dependency order: 1. Implement `get_device_name()`, `get_device_total_memory()`, and `get_available_gpu_memory()` before constructing `ServerArgs`. 2. Implement `get_device()` and `get_local_torch_device()` before worker binding. 3. Configure distributed initialization, attention selection, and custom-op implementations for supported workloads. 4. Run an end-to-end workload and audit model-specific kernels, compilation, and remaining device-family branches. Register custom-op implementations from `init_backend()`, which runs once in each worker before any pipeline module is constructed: ```python theme={null} class MyDiffusionPlatform(Platform): ... def init_backend(self) -> None: from my_platform_plugin.ops import rms_norm_forward from sglang.multimodal_gen.runtime.layers.custom_op import CustomOp from sglang.multimodal_gen.runtime.layers.layernorm import RMSNorm CustomOp.register_oot_forward( RMSNorm, fn=rms_norm_forward, platform_key="my_device" ) ``` `init_backend()` runs at most once per worker process, before the worker implementation is imported. Raising aborts startup and the failed initialization is not retried in that process, because registrations and other backend side effects may be only partially reversible. Custom-op dispatch is resolved when each operation is constructed, after backend initialization has completed. The selected callable therefore remains stable when the operation is compiled instead of changing on the first compiled call. The function receives the operation instance before its normal arguments. Registration matches the exact operation class and the value returned by `get_dispatch_key_name()`. Without a registration, SGLang looks for a matching `forward_()` implementation on the operation, then falls back to `forward_oot()`, whose base implementation calls `forward_native()`. Returning `"cuda"`, for example, lets a CUDA-compatible OOT platform reuse operation-specific `forward_cuda()` implementations without reporting CUDA platform identity. The `sglang.multimodal_gen.plugins` entry point remains for seams the `Platform` interface does not cover. Its callbacks run in launchers and workers before scheduler or model construction, so use a `BEFORE` or `AROUND` hook on `GPUWorker.init_device_and_model()` for rank-aware initialization. Hooks from platform distributions other than the selected one are skipped. Because these ship in your platform's distribution, a failure at any stage — load, callback, or hook application — aborts startup, the same as `init_backend()`. #### Diffusion platform contract **Active** interfaces are called through `current_platform`. **Compatibility** interfaces are retained adapters; unlisted `Platform` methods are not a stable OOT contract. | Status | Area | Interface | | - | - | - | | Active | Initialization | `init_backend()` | | Active | Configuration | `apply_server_args_defaults()` | | Active | Device and memory | `get_device()`, `set_device()`, `get_local_torch_device()`, `get_device_name()`, `get_device_uuid()`, `get_device_total_memory()`, `get_available_gpu_memory()`, `get_device_capability()` | | Active | Distributed | `get_torch_distributed_backend_str()`, `supports_distributed_device_id()`, `get_all_to_all_communicator_cls()`, `get_cpu_architecture()` | | Active | Dispatch and model | `dispatch_key` (read by `get_torch_library_dispatch_key()`), `get_dispatch_key_name()`, `get_attn_backend_cls_str()`, `verify_model_arch()`, `optimize_vae()` | | Active | Execution | `get_compile_backend()`, `get_compile_options()`, `inference_mode()`, `seed_everything()`, `enable_dit_layerwise_offload_by_default()` | | Compatibility | Communicator | `get_device_communicator_cls()` remains the fallback used by `get_all_to_all_communicator_cls()` for existing platform subclasses. New OOT platforms should override the latter. | Install the package, restart Python to refresh entry-point metadata, and verify explicit selection: ```bash theme={null} pip install -e my_platform_plugin/ SGLANG_DIFFUSION_PLATFORM_OVERRIDE=my_device python -c \ "from sglang.multimodal_gen.runtime.platforms import current_platform; print(current_platform)" ``` ### Diffusion platform limitations * SRT and diffusion require separate platform classes. * Only one external platform can be active per process; set the runtime selector if multiple callbacks activate. * `get_all_to_all_communicator_cls()` controls only `all_to_all_4D()`, not every collective, graph-capture, or synchronization path; `get_device_communicator_cls()` remains for compatibility. * Compile settings affect only `build_torch_compile_kwargs()` callers; audit static `@torch.compile` decorators and other direct compile paths. * Explicit attention selectors accept only `AttentionBackendEnum` names. Returning a custom backend works when none is selected; selecting it by name requires a hook or downstream patch. * Device-family branches remain outside the interface. Each supported workload needs a native fallback or an actionable unsupported-feature error. * The diffusion `PlatformEnum.OOT` identifies an external provider. Override built-in identity predicates only after auditing every enabled branch. *** ## Plugin Type 2: General Plugin ### Description General function plugins inject behavior into SRT or diffusion **without requiring a custom platform**. The two runtimes have separate entry-point groups and hook registries: ```python theme={null} # SRT plugin from sglang.srt.plugins.hook_registry import HookType, plugin_hook # Diffusion plugin from sglang.multimodal_gen.runtime.platforms.plugins import HookType, plugin_hook ``` Register an SRT plugin under `sglang.srt.plugins` and a diffusion plugin under `sglang.multimodal_gen.plugins`, and import the hook API from the matching module above. Each runtime applies only its own registry, so a hook registered through the other runtime's API never runs. The examples below use SRT. The diffusion hook API lives alongside platform discovery in `sglang.multimodal_gen.runtime.platforms.plugins`; `sglang.multimodal_gen.plugins` is its entry-point group, not a Python module path. Use cases include: * **Observability**: Add logging, metrics, and tracing to any function * **Behavior modification**: Modify function arguments or return values * **Performance profiling**: Add timing to critical functions * **A/B testing**: Replace implementations at runtime ### Quick Start **1. Create a minimal package:** ``` my_general_plugin/ ├── pyproject.toml └── my_general_plugin/ └── __init__.py # register() function ``` **2. `pyproject.toml`:** ```toml theme={null} [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" [project] name = "my-general-plugin" version = "0.1.0" [project.entry-points."sglang.srt.plugins"] my_plugin = "my_general_plugin:register" ``` **3. `__init__.py`** — register hooks: ```python theme={null} from sglang.srt.plugins.hook_registry import HookRegistry, HookType def register(): """Entry point called by load_plugins().""" HookRegistry.register( "sglang.srt.managers.scheduler.Scheduler.__init__", my_hook, HookType.AROUND, ) def my_hook(original_fn, self, *args, **kwargs): result = original_fn(self, *args, **kwargs) print(f"Scheduler initialized! gpu_id={self.gpu_id}") return result ``` **4. Install and run:** ```bash theme={null} pip install -e my_general_plugin/ sglang serve --model-path [options] # Look for "Scheduler initialized!" in logs ``` ### Hook Types `HookRegistry` supports four hook types:
Hook Type Signature Description
BEFORE fn(\*args, \*\*kwargs) -> (args, kwargs) | None Runs before the original. Return None to keep args unchanged, or (args, kwargs) to modify.
AFTER fn(result, \*args, \*\*kwargs) -> new\_result | None Runs after the original. Return None to keep result, or a new value to replace.
AROUND fn(original\_fn, \*args, \*\*kwargs) -> result Wraps the original. You must call original\_fn yourself. Full control over execution.
REPLACE fn(\*args, \*\*kwargs) -> result or class Replace the original function or class entirely. For class targets, pass a replacement class directly — it is substituted via setattr preserving isinstance()/issubclass() semantics.
> **Note**: Only `REPLACE` accepts a class as the hook. Passing a class to `BEFORE`/`AFTER`/`AROUND` raises `TypeError` at registration time. ### Registration API Hooks can be registered using the **imperative API** or the **decorator API**: ```python theme={null} # --- Imperative API --- from sglang.srt.plugins.hook_registry import HookRegistry, HookType def my_timer(original_fn, *args, **kwargs): start = time.perf_counter() result = original_fn(*args, **kwargs) print(f"Elapsed: {time.perf_counter() - start:.3f}s") return result HookRegistry.register( "sglang.srt.managers.scheduler.Scheduler.get_next_batch_to_run", my_timer, HookType.AROUND, ) # --- Decorator API --- from sglang.srt.plugins.hook_registry import plugin_hook, HookType @plugin_hook( "sglang.srt.managers.scheduler.Scheduler.get_next_batch_to_run", type=HookType.AROUND, ) def my_timer(original_fn, *args, **kwargs): start = time.perf_counter() result = original_fn(*args, **kwargs) print(f"Elapsed: {time.perf_counter() - start:.3f}s") return result # --- Class replacement (REPLACE) --- from sglang.srt.plugins.hook_registry import plugin_hook, HookType from sglang.srt.managers.scheduler import Scheduler @plugin_hook( "sglang.srt.managers.scheduler.Scheduler", type=HookType.REPLACE, ) class MyScheduler(Scheduler): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) print("Enhanced scheduler initialized!") ``` ### Hook Target Resolution Target paths use fully-qualified dotted notation. Both formats are supported: * **Dotted**: `sglang.srt.managers.scheduler.Scheduler.__init__` * **Entry-points style**: `sglang.srt.managers.scheduler:Scheduler.__init__` (colon treated as dot) ### Common SRT Hook Targets
Target Description
sglang.srt.server\_args.ServerArgs.add\_cli\_args Add custom CLI arguments
sglang.srt.server\_args.ServerArgs.**post\_init** Modify ServerArgs after parsing
sglang.srt.server\_args.ServerArgs.check\_server\_args Add/relax validation
sglang.srt.managers.scheduler.Scheduler.**init** Custom scheduler state
sglang.srt.managers.scheduler.Scheduler.get\_next\_batch\_to\_run Custom scheduling policy
sglang.srt.managers.scheduler.Scheduler.run\_batch Profiling / inspection
sglang.srt.managers.scheduler.Scheduler.process\_batch\_result Custom metrics
sglang.srt.managers.tp\_worker.TpModelWorker.**init** Custom worker state
sglang.srt.managers.tp\_worker.TpModelWorker.forward\_batch\_generation Forward pass wrapping
*** ## File Reference
File Description
sglang/srt/platforms/device\_mixin.py DeviceMixin base class and SRT platform identity types
sglang/srt/platforms/interface.py SRTPlatform base class (extends DeviceMixin)
sglang/srt/platforms/**init**.py current\_platform lazy singleton + discovery logic
sglang/multimodal\_gen/runtime/platforms/interface.py Diffusion Platform base class
sglang/multimodal\_gen/runtime/platforms/**init**.py Diffusion current\_platform lazy singleton and built-in fallback order
sglang/multimodal\_gen/runtime/platforms/plugins.py Diffusion plugin registration, explicit hook-application phase, and hook registry
sglang/multimodal\_gen/runtime/managers/worker\_bootstrap.py Import-neutral process specifications and spawn targets that initialize the backend before resolving runtime hooks
sglang/srt/plugins/**init**.py load\_plugins() + load\_plugins\_by\_group()
sglang/srt/plugins/hook\_registry.py HookRegistry, HookType, plugin\_hook decorator
# TPU Source: https://docs.sglang.io/docs/hardware-platforms/tpu SGLang supports high-performance TPU inference through the SGLang-JAX backend, which is specifically optimized for Google Cloud TPUs. The JAX-based implementation delivers exceptional throughput and low latency for Large Language Model (LLM) serving workloads on TPU hardware. SGLang supports high-performance TPU inference through the SGLang-JAX backend, which is specifically optimized for Google Cloud TPUs. The JAX-based implementation delivers exceptional throughput and low latency for Large Language Model (LLM) serving workloads on TPU hardware. For TPU-specific issues or feature requests, please visit the [sglang-jax GitHub issues page](https://github.com/sgl-project/sglang-jax/issues). **NOTE:** SGLang TPU support is implemented via the SGLang-JAX backend, a dedicated JAX-based inference engine maintained as a separate repository at [https://github.com/sgl-project/sglang-jax](https://github.com/sgl-project/sglang-jax). ## System Requirements ### Supported TPU Hardware
TPU Type HBM Memory Availability
TPU v6e 32 GB Google Cloud
TPU v7 96 GB per core Google Cloud
### Software Requirements * **Python:** 3.12 or higher * **JAX:** Latest version with TPU support * **Environment:** Google Cloud TPU VM or compatible TPU runtime * **Optional:** SkyPilot for simplified cloud deployment ## Feature Support Matrix SGLang-JAX provides comprehensive TPU-optimized features for production LLM serving:
Feature Support Status Description
High-Throughput Continuous Batching ✅ Dynamic request batching for maximum TPU utilization
Radix Tree KV Cache ✅ Memory-efficient prefix sharing between requests
FlashAttention Backend ✅ TPU-optimized attention kernel for long sequences
Tensor Parallelism ✅ Distribute models across multiple TPU cores
Paged Attention ✅ Flexible KV cache management with paging
Speculative Decoding (EAGLE/EAGLE3) ✅ 20-40% throughput improvement for compatible models
Chunked Prefill ✅ Mixed prefill-decode batching
OpenAI-Compatible API ✅ Drop-in replacement for OpenAI API
Data Parallel Attention 🚧 In development - Attention computation with data parallelism
Quantization 🚧 In development - Model quantization for reduced memory usage
Multi-LoRA 🚧 In development - Serve multiple LoRA adapters simultaneously
### Attention Backend Comparison
**Backend** **Paged Attention** **Spec Decoding** **MLA** **Sliding Window**
FlashAttention (fa) ✅ ✅ ❌ ✅
Native ❌ ❌ ❌ ❌
**NOTE:** FlashAttention backend is recommended for production workloads due to superior memory efficiency and performance. ## Optimized Model List SGLang-JAX supports the following model families. Use the linked TPU cookbooks for deployment instructions.
Model Family Cookbook
Qwen 3 Qwen3
Qwen 3 MoE Qwen3 MoE
Qwen2.5-VL Qwen2.5-VL
Llama/LLaMA Llama 3.1 / Llama 3.3 70B
DeepSeek V3 / R1 DeepSeek V3 / DeepSeek R1
Kimi-Linear Kimi-Linear
Grok-2 Grok-2
Gemma 2 Gemma 2
MiMo-V2-Flash MiMo-V2-Flash
MiMo-V2.5-Pro MiMo-V2.5-Pro
Ling-2.6 Ling-2.6
## Installation ### Method 1: Using PyPI (Recommended) ```bash Command theme={null} pip install sglang-jax ``` ### Method 2: From Source ```bash Command theme={null} git clone https://github.com/sgl-project/sglang-jax cd sglang-jax uv venv --python 3.12 && source .venv/bin/activate uv pip install -e "python[all]" ``` ### Method 3: Using Docker **NOTE:** Docker support for TPU is currently under development. Please use PyPI or source installation methods. ### Method 4: Cloud TPU with SkyPilot [SkyPilot](https://github.com/skypilot-org/skypilot) provides simplified deployment on Google Cloud TPU: 1. Install SkyPilot and configure GCP access (see [SkyPilot documentation](https://skypilot.readthedocs.io/)) 2. Create a SkyPilot configuration file: SkyPilot YAML: sglang-jax.sky.yaml}> ```yaml Config theme={null} # sglang-jax.sky.yaml resources: accelerators: tpu-v6e-4 accelerator_args: tpu_vm: True runtime_version: v2-alpha-tpuv6e run: | git clone https://github.com/sgl-project/sglang-jax.git cd sglang-jax uv venv --python 3.12 source .venv/bin/activate uv pip install -e "python[all]" ``` 3. Launch your TPU cluster: ```bash Command theme={null} # Standard deployment sky launch -c sglang-jax sglang-jax.sky.yaml --infra=gcp # With spot instances for cost savings sky launch -c sglang-jax sglang-jax.sky.yaml --infra=gcp --use-spot ``` ## Launch of the Serving Engine ### Basic Example: Qwen-7B ```bash Command theme={null} JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache python3 -u -m sgl_jax.launch_server \ --model-path Qwen/Qwen-7B-Chat \ --trust-remote-code \ --dist-init-addr=0.0.0.0:10011 \ --nnodes=1 \ --tp-size=4 \ --device=tpu \ --random-seed=3 \ --node-rank=0 \ --mem-fraction-static=0.8 \ --max-prefill-tokens=8192 \ --download-dir=/tmp \ --dtype=bfloat16 \ --skip-server-warmup \ --host 0.0.0.0 \ --port 30000 ``` **Key Parameters Explained:** 1. `JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache` - Enables JIT compilation caching to accelerate server startup on subsequent runs 2. `--tp-size=4` - Tensor parallelism size; match this to your TPU core count (typically 1, 4, or 8) 3. `--device=tpu` - Specifies TPU device (this is the default for sglang-jax) 4. `--dtype=bfloat16` - Uses bfloat16 precision, which TPUs are optimized for 5. `--mem-fraction-static=0.8` - Allocates 80% of TPU HBM for static memory (adjustable from 0.2 to 0.9) 6. `--max-prefill-tokens=8192` - Maximum number of tokens processed in the prefill phase ### High-Performance Configuration: Qwen3-8B For production workloads with optimal throughput: ```bash Command theme={null} python3 -u -m sgl_jax.launch_server \ --model-path Qwen/Qwen3-8B \ --trust-remote-code \ --tp-size=4 \ --device=tpu \ --mem-fraction-static=0.8 \ --chunked-prefill-size=2048 \ --dtype=bfloat16 \ --max-running-requests=256 \ --page-size=128 \ --attention-backend=fa ``` ### Advanced: Speculative Decoding (EAGLE3) Speculative decoding can improve throughput by 20-40% for compatible models: ```bash Command theme={null} python3 -u -m sgl_jax.launch_server \ --model-path Qwen/Qwen3-32B \ --trust-remote-code \ --device=tpu \ --tp-size=4 \ --mem-fraction-static=0.8 \ --max-prefill-tokens=4096 \ --attention-backend=fa \ --dtype=bfloat16 \ --port=30000 \ --host=0.0.0.0 \ --disable-overlap-schedule \ --speculative-algorithm=EAGLE3 \ --speculative-draft-model-path=AngelSlim/Qwen3-32B_eagle3 \ --page-size=64 \ --speculative-eagle-topk=1 \ --speculative-num-steps=3 \ --speculative-num-draft-tokens=4 ``` **NOTE:** Speculative decoding is currently supported for Qwen3 and LLaMA model families. See the [Speculative Decoding documentation](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/speculative_decoding.md) for detailed configuration guidance. ### Multi-Node Distributed Serving For large models requiring multiple TPU VMs: ```bash Command theme={null} # Node 0 (coordinator) python3 -m sgl_jax.launch_server \ --model-path MODEL_PATH \ --dist-init-addr=NODE0_IP:10011 \ --nnodes=2 \ --node-rank=0 \ --tp-size=8 \ [other parameters...] # Node 1 (worker) python3 -m sgl_jax.launch_server \ --model-path MODEL_PATH \ --dist-init-addr=NODE0_IP:10011 \ --nnodes=2 \ --node-rank=1 \ --tp-size=8 \ [other parameters...] ``` ## Benchmarking with Requests ### Throughput Testing Basic throughput benchmark: ```bash Command theme={null} python3 -m sgl_jax.bench_serving \ --backend sgl-jax \ --dataset-name random \ --num-prompts=100 \ --random-input=512 \ --random-output=128 \ --max-concurrency=8 \ --random-range-ratio=1 \ --warmup-requests=0 ``` ### Latency Testing Measure single-batch latency: ```bash Command theme={null} python3 -m sgl_jax.bench_one_batch_server \ --base-url http://127.0.0.1:30000 \ --model-path Qwen/Qwen-7B-Chat \ --batch-size=32 \ --input-len=256 \ --output-len=32 ``` ### Comprehensive Benchmark Script For systematic performance evaluation across different configurations: ```bash Command theme={null} #!/bin/bash set -e backend=${1:-sgl-jax} num_prompts_per_concurrency=3 input_seq_lens=(1024 4096 8192) output_seq_lens=(1 1024) max_concurrencies=(8 16 32 64 128 256) for input_seq_len in "${input_seq_lens[@]}"; do for output_seq_len in "${output_seq_lens[@]}"; do echo "=======================================" echo "Testing ISL/OSL: $input_seq_len/$output_seq_len" echo "=======================================" for max_concurrency in "${max_concurrencies[@]}"; do num_prompts=$((num_prompts_per_concurrency * max_concurrency)) python3 -m sgl_jax.bench_serving \ --backend ${backend} \ --dataset-name random \ --num-prompts ${num_prompts} \ --random-input ${input_seq_len} \ --random-output ${output_seq_len} \ --max-concurrency ${max_concurrency} \ --random-range-ratio 1 \ --disable-ignore-eos \ --warmup-requests 0 done done done ``` For detailed help on all benchmark parameters: ```bash Command theme={null} python3 -m sgl_jax.bench_serving --help ``` See the [Benchmark and Profiling Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/benchmark_and_profiling.md) for advanced benchmarking techniques and profiling with JAX Profiler. ## Performance Optimization ### Memory Optimization **Reduce memory usage:** * Lower `--mem-fraction-static` (from 0.8 → 0.5 → 0.3) * Decrease `--max-prefill-tokens` (from 16384 → 8192 → 4096) * Reduce `--max-running-requests` **Handle OOM errors:** * Start with conservative memory settings (`--mem-fraction-static=0.5`) * Gradually increase until you find the optimal balance * Increase `--page-size` for better memory locality (1 → 16 → 64 → 128) ### Throughput Optimization To maximize tokens per second: * Use FlashAttention backend: `--attention-backend=fa` * Enable speculative decoding (EAGLE3) for Qwen3 models (20-40% improvement) * Increase `--max-running-requests` to 256+ * Set `--mem-fraction-static` to 0.8+ (if memory allows) * Use larger page sizes (64-128) * Enable chunked prefill: `--chunked-prefill-size=2048` ### Latency Optimization To minimize time-to-first-token (TTFT) and inter-token latency: * Reduce `--page-size` to 1-4 * Lower `--max-running-requests` (16-32) for smaller batches * Reduce `--chunked-prefill-size` * Use conservative memory settings to avoid GC pauses ### TPU-Specific Optimizations 1. **JIT Compilation Cache:** ```bash Command theme={null} export JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache ``` Always set this environment variable to cache compiled kernels and accelerate server startup. 2. **Data Type Optimization:** Use `--dtype=bfloat16` for TPU native optimization. TPUs are specifically designed for bfloat16 computations. 3. **Tensor Parallelism:** Match `--tp-size` to your TPU core configuration (1, 4, or 8) for optimal model distribution. 4. **Attention Backend:** Always use `--attention-backend=fa` (FlashAttention) for production workloads. ## Troubleshooting ### OOM (Out of Memory) Errors If you encounter out-of-memory errors: 1. Reduce `--mem-fraction-static` from 0.8 to 0.5 or lower 2. Decrease `--max-prefill-tokens` from 8192 to 4096 or 2048 3. Lower `--max-running-requests` to reduce concurrent batch size 4. Increase `--page-size` for better memory layout efficiency ### Compilation Long-Time If the server takes too long to start: 1. Ensure `JAX_COMPILATION_CACHE_DIR` is properly set 2. Understand that the first run requires JIT compilation (this is normal) 3. Subsequent runs will be significantly faster with cached compilations 4. Consider using `--skip-server-warmup` to defer compilation until first request ### Low Throughput If you're not achieving expected throughput: 1. Verify `--tp-size` matches your TPU core configuration 2. Check that `--attention-backend=fa` is enabled 3. Increase `--max-running-requests` to enable larger batch formation 4. Consider enabling speculative decoding for compatible models 5. Ensure memory settings allow for sufficient batch sizes ### Connection Issues If clients cannot connect to the server: 1. Ensure `--host=0.0.0.0` for external access (not just `127.0.0.1`) 2. Verify firewall rules allow traffic on the specified port (default: 30000) 3. Check that the server process is running: `curl http://localhost:30000/health` ## Advanced Features ### Speculative Decoding SGLang-JAX supports EAGLE and EAGLE3 speculative decoding algorithms for Qwen3 and LLaMA model families. Speculative decoding can improve throughput by 20-40% without affecting output quality. See the [Speculative Decoding documentation](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/speculative_decoding.md) for detailed configuration and supported model combinations. ### Chunked Prefill Enable mixed prefill-decode batching for better TPU utilization: ```bash Command theme={null} --chunked-prefill-size=2048 --enable-mixed-chunk ``` This allows the scheduler to mix prefill operations with decode operations in the same batch, improving overall throughput. ### Custom Attention Backends SGLang-JAX supports a plugin-based attention backend system. You can implement custom attention kernels optimized for specific use cases. See the [Attention Backend documentation](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/attention_backend.md) for implementation details. ### Environment Verification Verify your TPU setup before deploying: ```bash Command theme={null} python -c "from sgl_jax import check_env; check_env.check_env()" ``` This command checks: * Installed package versions * TPU device availability and specifications * System resources and configuration * Compatibility of settings ## Contributing We welcome contributions to improve TPU support in SGLang-JAX! ### Areas for Contribution **Check the [Development Roadmap](https://github.com/sgl-project/sglang-jax/issues/190)** to see planned features and find opportunities to contribute new functionality. Current contribution areas include: * Performance optimizations for specific TPU generations * Support for additional model architectures * Documentation improvements and examples * Bug reports and fixes * Benchmark results and performance analysis ### How to Contribute 1. Visit the [sglang-jax repository](https://github.com/sgl-project/sglang-jax) 2. Read the [Contribution Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/contribution_guide.md) 3. Join the [SGL-JAX Slack community](https://sgl-fru7574.slack.com/archives/C09EBE5HT5X) for discussions 4. Report issues at [sglang-jax/issues](https://github.com/sgl-project/sglang-jax/issues) ### Testing on TPU For contributors who need TPU access for testing: * Refer to the [TPU Resources Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/tpu_resources_guide.md) for information on accessing TPU hardware * Use SkyPilot with spot instances for cost-effective testing * Follow the [Benchmark and Profiling Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/benchmark_and_profiling.md) for performance validation ## References ### Documentation * [SGLang-JAX Repository](https://github.com/sgl-project/sglang-jax) * [SGLang-JAX Installation Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/get_started/install.md) * [Qwen Models Quick Start](https://github.com/sgl-project/sglang-jax/blob/main/docs/basic_usage/qwen.md) * [Benchmark and Profiling Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/benchmark_and_profiling.md) * [Speculative Decoding](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/speculative_decoding.md) ### External Resources * [JAX Documentation](https://jax.readthedocs.io/) * [Google Cloud TPU Documentation](https://cloud.google.com/tpu/docs) * [SkyPilot Documentation](https://skypilot.readthedocs.io/) # XPU Source: https://docs.sglang.io/docs/hardware-platforms/xpu The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on Intel GPU, [see more context about Intel GPU support within PyTorch ecosystem](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html). Specifically, SGLang is optimized for [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics.html) and [ Intel® Arc™ B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics.html). A number of popular LLMs are optimized and run efficiently on XPU, including the most notable open-source models like Llama series, Qwen series, and Diffusion model series like FLUX and Z-Image. Please check the [SGLang Cookbook pages](https://docs.sglang.io/cookbook/intro) in which the support status and example commands can be found. Quantized MoE models are covered separately in [MXFP4 MoE Quantization](#mxfp4-moe-quantization) below. ## Installation ### Install From Source Currently SGLang XPU only supports installation from source. Please refer to ["Getting Started on Intel GPU"](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) to install XPU dependency. ```bash Command theme={null} # Below commands shows Ubuntu/Debian OS as example, other OS requires similar command change # add sudo if required to install below packages # Install curl and protobuf compiler apt-get update && apt-get install -y curl protobuf-compiler # Install Rust (non-interactive) curl --proto '=https' --tlsv1.2 --retry 3 --retry-delay 2 -sSf https://sh.rustup.rs | sh -s -- -y # Make cargo available in the current shell source "$HOME/.cargo/env" # Create and activate a conda environment conda create -n sgl-xpu python=3.12 -y conda activate sgl-xpu # Set PyTorch XPU as primary pip install channel to avoid installing the larger CUDA-enabled version and prevent potential runtime issues. pip3 install torch==2.14.0+xpu torchvision==0.29.0+xpu torchaudio==2.11.0+xpu --index-url https://download.pytorch.org/whl/xpu pip3 install xgrammar==0.1.33 --no-deps # xgrammar will introduce CUDA-enabled triton which might conflict with XPU # Clone the SGLang code git clone https://github.com/sgl-project/sglang.git cd sglang git checkout # Use dedicated toml file cd python cp pyproject_xpu.toml pyproject.toml # Install SGLang dependent libs, and build SGLang main package pip install --upgrade pip setuptools pip install -v ".[dev,diffusion]" --extra-index-url https://download.pytorch.org/whl/xpu ``` ### Install Using Docker It is recommended to use Docker for setting up the SGLang environment. #### Pull from Docker Hub Pull the prebuilt docker image of SGLang package releases from `lmsysorg/sglang` repository. The [XPU image tags](https://hub.docker.com/r/lmsysorg/sglang/tags?name=xpu) end with `xpu` suffix. The image pulling command is like: ```bash Command theme={null} docker pull lmsysorg/sglang:v0.5.20-xpu ``` #### Build from Dockerfile [The SGLang XPU Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xpu.Dockerfile) is provided to facilitate the installation. Replace `` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens). ```bash Command theme={null} # Clone the SGLang repository git clone https://github.com/sgl-project/sglang.git cd sglang/docker # Build the docker image docker build -t sglang-xpu:latest -f xpu.Dockerfile . # Initiate a docker container docker run \ -it \ --privileged \ --ipc=host \ --network=host \ --user root \ --group-add $(getent group video | cut -d: -f3) \ --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path \ -v /dev/shm:/dev/shm \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -p 30000:30000 \ -e "HF_TOKEN=" \ sglang-xpu:latest /bin/bash ``` ## Launch of the Serving Engine Example command to launch SGLang serving: ```bash theme={null} sglang serve \ --model-path \ --trust-remote-code \ --device xpu \ --host 0.0.0.0 \ --tp 2 \ # using multi GPUs --attention-backend intel_xpu \ # using intel optimized XPU attention backend --page-size \ # intel_xpu attention backend supports [32, 64, 128] ``` ## MXFP4 MoE Quantization Native MXFP4 MoE checkpoints (OCP microscaling FP4: packed `e2m1` weights plus per-32-element `ue8m0` block scales) run on Intel GPUs through the `sgl-kernel-xpu` W4A16 grouped GEMM. The expert weights stay in the checkpoint's packed layout end to end — there is no dequantization to BF16 — so GPT-OSS-20B loads in roughly 13 GB rather than the \~42 GB a BF16 upcast would need, which is what lets it fit on a single 24 GB card. The `mxfp4` method is registered automatically on `--device xpu`, which already requires `sgl-kernel-xpu` — no extra flag is needed: ```bash theme={null} sglang serve \ --model-path openai/gpt-oss-20b \ --device xpu \ --attention-backend intel_xpu ``` **Tested models:** | Model | Notes | | :- | :- | | [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) | 32 experts, top-k 4, hidden 2880, intermediate 2880; clamped-swiglu activation (`gemm1_alpha` 1.702, limit 7.0) with expert biases | **Requirements and limitations:** | Item | Status | | - | - | | GPU architecture | Xe2 / BMG only (Intel® Arc™ B-Series and Arc™ Pro B-Series) | | Quantized layers | Fused MoE experts only; attention, router, embeddings and LM head stay BF16 (per the checkpoint's `modules_to_not_convert`) | | Activations | BF16 (W4A16). Clamped swiglu (GPT-OSS) and plain SiLU are supported | | Expert parallelism | Not yet validated on this path | ## Benchmarking with Requests You can benchmark the performance via the `bench_serving` script. Run the command in another terminal. ```bash theme={null} python -m sglang.bench_serving \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1 \ --request-rate inf \ --random-range-ratio 1.0 ``` The detail explanations of the parameters can be looked up by the command: ```bash theme={null} python -m sglang.bench_serving -h ``` Additionally, the requests can be formed with [OpenAI Completions API](../basic_usage/openai_api_completions) and sent via the command line (e.g. using `curl`) or via your own script. ## XPU Graph \[Experimental] SGLang enables XPU graph capture to reduce per-step kernel-launch overhead. | Phase | Backend | Mechanism | Default | | - | - | - | - | | Decode | `full` | One `torch.xpu.XPUGraph` per batch size, captured on startup | **Off** (opt-in) | | Prefill | `breakable` | Segmented `torch.xpu.XPUGraph` capture/replay (no `torch.compile`); eager break points at attention / MoE boundaries | **Off** (opt-in) | ### Enable Decode Graph Decode graph capture is **opt-in** on XPU. Enable it explicitly: ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-decode full ``` ### Enable Prefill Graph Prefill graph capture is **opt-in** on XPU and must be enabled explicitly. Use the `breakable` backend. The former `tc_piecewise` backend has been removed. #### breakable Captures the transformer stack as segmented `XPUGraph`s with eager break points at attention / MoE boundaries, without `torch.compile`: ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-prefill breakable ``` You can also configure both phases together with a single `--cuda-graph-config` JSON argument: ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-config '{"decode":{"backend":"full"},"prefill":{"backend":"breakable"}}' ``` ### Enable torch.compile for Decode `--enable-torch-compile` adds a `torch.compile` pass on top of the decode XPU graph: the model forward is compiled first, and the compiled forward is then captured as an `XPUGraph`. This can reduce per-kernel overhead further but increases startup time. ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --enable-torch-compile ``` ### Disable XPU Graph Both phases are disabled by default. To explicitly disable them anyway: ```bash theme={null} # Disable decode graph (already off by default; explicit form) python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-decode=disabled # Disable prefill graph (already off by default; explicit form) python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-prefill=disabled # Disable both phases python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-decode=disabled \ --cuda-graph-backend-prefill=disabled ``` ### Customize Capture Buckets By default, prefill capture sizes are derived from `--chunked-prefill-size`. To specify explicit token-length buckets: ```bash theme={null} python -m sglang.launch_server \ --model-path --device xpu \ --cuda-graph-backend-prefill breakable \ --cuda-graph-bs-prefill 64 128 256 512 ``` To specify explicit decode graph batch sizes: ```bash theme={null} python -m sglang.launch_server \ --model-path --device xpu \ --cuda-graph-bs-decode 1 2 4 8 ``` ### Server Args | Argument | XPU allowed values | Default | Description | | - | - | - | - | | `--cuda-graph-backend-decode` | `full`, `disabled` | `disabled` | Backend for the decode phase. Only `full` is supported on XPU. Set to `full` to enable. | | `--cuda-graph-backend-prefill` | `breakable`, `disabled` | `disabled`\* | Backend for the prefill phase. Set to `breakable` explicitly to enable. | | `--cuda-graph-bs-prefill` | list of ints | auto | Explicit token-length buckets to capture for prefill. | | `--cuda-graph-bs-decode` | list of ints | auto | Explicit batch sizes to capture for decode. | | `--cuda-graph-config` | JSON string | — | One-shot JSON config for both phases, e.g. `'{"decode":{"backend":"full"},"prefill":{"backend":"breakable"}}'`. Overrides all per-phase flags. | | `--disable-decode-cuda-graph` | — | `False` | Shorthand for `--cuda-graph-backend-decode=disabled`. | | `--disable-prefill-cuda-graph` | — | `False` | Shorthand for `--cuda-graph-backend-prefill=disabled`. | | `--enable-torch-compile` | — | `False` | Apply `torch.compile` on top of the decode XPU graph for further kernel optimization. | | `--torch-compile-max-bs` | int | `32` | Maximum batch size compiled by `torch.compile` when `--enable-torch-compile` is set. | \* Prefill graph is auto-disabled on XPU unless you lock the backend explicitly via `--cuda-graph-backend-prefill` or `--cuda-graph-config`. ### Limitations | Feature | Status | | - | - | | Two-batch overlap (`--enable-two-batch-overlap`) | Not yet supported | | Speculative decoding | Not yet implemented | ## Decode Context Parallel (DCP) on Intel XPU \[Experimental] DCP shards the KV cache across the ranks of a TP group by token position. It grows the servable context only when `tp_size` exceeds the model's KV heads, which TP would otherwise replicate, and decode gets slower, so use it only when the context you need does not fit. ```bash theme={null} python -m sglang.launch_server --model-path Qwen/Qwen3.5-4B --device xpu \ --attention-backend intel_xpu --tp 2 --dcp-size 2 \ --disable-radix-cache --mem-fraction-static 0.60 ``` It needs the `intel_xpu` attention backend and ranks that share their DCP group's KV heads: Qwen3.5 models, or models with at most `tp_size / dcp_size` KV heads. Unsupported features are refused at startup. If startup fails with `UR_RESULT_ERROR_OUT_OF_RESOURCES`, lower `--mem-fraction-static`. ## Memory Saver (release/resume memory occupation) on Intel XPU \[Experimental] SGLang can temporarily release most of the GPU memory it holds — model weights and/or KV cache — and reclaim it later without restarting the process. This is the same `release_memory_occupation` / `resume_memory_occupation` feature available on CUDA, used for RL rollout/training hand-off and for freeing the device between inference bursts. This is backed by the [`torch_memory_saver`](https://github.com/fzyzcjy/torch_memory_saver) package — the same package used on CUDA — which gained an Intel XPU backend built natively on Level Zero (keeping virtual addresses fixed while releasing/re-committing physical pages via `zeVirtualMemUnmap` / `zeVirtualMemMap`). **Install `torch_memory_saver`.** Unlike CUDA (prebuilt wheel), the XPU backend is built from source against your local oneAPI + `torch+xpu` runtime (the `.so` links `libsycl.so.`, which must match the installed `intel-sycl-rt`). `TMS_PLATFORM=xpu` forces the XPU backend, and `--no-build-isolation` lets the build import your installed `torch` so it can match the `libsycl` major to it: The published wheels are CUDA-only, so install from git and let it build. The ref below is the `v0.0.10b2` release, pinned so the build is reproducible. ```bash theme={null} source /opt/intel/oneapi/setvars.sh TMS_PLATFORM=xpu pip install --no-build-isolation \ git+https://github.com/fzyzcjy/torch_memory_saver.git@a5c99f11b18ebb8e9fda71a68812e476ae49e417 ``` **Use it** by launching with `--enable-memory-saver` (the XPU backend is selected automatically); optionally add `--enable-weights-cpu-backup` to keep weights in host RAM across a release: ```bash theme={null} python -m sglang.launch_server --model-path Qwen/Qwen3-0.6B \ --trust-remote-code --device xpu --enable-memory-saver ``` ```bash theme={null} # Release GPU memory while idle, then reclaim it (server must be idle). curl -X POST http://127.0.0.1:30000/release_memory_occupation curl -X POST http://127.0.0.1:30000/resume_memory_occupation ``` The Python engine API (`engine.release_memory_occupation(tags=...)` / `engine.resume_memory_occupation(tags=...)`) and the `weights` / `kv_cache` tags behave the same as on CUDA. Pauseable CUDA-graph capture is not used on XPU, so the `cuda_graph` tag is a no-op there. > **Verifying memory was freed:** neither `torch.xpu.memory_allocated()` nor > `torch.xpu.mem_get_info()` drops when physical pages are released — the first is > allocator accounting, and the second stays flat because torch keeps the freed block > cached. Query actual device memory via sysman (`ZES_ENABLE_SYSMAN=1`) instead. ## Prefill-Decode (P/D) Disaggregation on Intel XPU \[Experimental] SGLang supports prefill-decode disaggregation on Intel XPU using the [NIXL](https://github.com/ai-dynamo/nixl) KV-transfer backend. **Tested models:** | Model | Notes | | :-: | :-: | | [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) | Used in integration tests; verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode) | | [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) | Verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode) | **Prerequisites:** `pip install nixl sglang-router` **Start the prefill server (GPU 0):** ```bash theme={null} ZE_AFFINITY_MASK=0 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \ --model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \ --disaggregation-mode prefill --disaggregation-transfer-backend nixl \ --disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30000 ``` **Start the decode server (GPU 1):** ```bash theme={null} ZE_AFFINITY_MASK=1 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \ --model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \ --disaggregation-mode decode --disaggregation-transfer-backend nixl \ --disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30001 ``` **Start the router:** ```bash theme={null} python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://127.0.0.1:30000 \ --decode http://127.0.0.1:30001 \ --host 0.0.0.0 --port 8000 ``` **Send a request:** ```bash theme={null} curl http://127.0.0.1:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen3-0.6B", "prompt": "The capital of France is", "max_tokens": 32}' ``` > **Note:** `UCX_POSIX_USE_PROC_LINK=n` is required on Intel XPU to avoid UCX shared-memory transport issues. # Custom Chat Template Source: https://docs.sglang.io/docs/references/custom_chat_template **NOTE**: There are two chat template systems in SGLang project. This document is about setting a custom chat template for the OpenAI-compatible API server (defined at [conversation.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/parser/conversation.py)). It is NOT related to the chat template used in the SGLang language frontend (defined at [chat\_template.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/lang/chat_template.py)). By default, the server uses the chat template specified in the model tokenizer from Hugging Face. It should just work for most official models such as Llama-2/Llama-3. If needed, you can also override the chat template when launching the server: ```bash Command theme={null} python -m sglang.launch_server \ --model-path meta-llama/Llama-2-7b-chat-hf \ --port 30000 \ --chat-template llama-2 ``` If the chat template you are looking for is missing, you are welcome to contribute it or load it from a file. ## JSON Format You can load the JSON format, which is defined by `conversation.py`. ```json Config theme={null} { "name": "my_model", "system": "<|im_start|>system", "user": "<|im_start|>user", "assistant": "<|im_start|>assistant", "sep_style": "CHATML", "sep": "<|im_end|>", "stop_str": ["<|im_end|>", "<|im_start|>"] } ``` ```bash Command theme={null} python -m sglang.launch_server \ --model-path meta-llama/Llama-2-7b-chat-hf \ --port 30000 \ --chat-template ./my_model_template.json ``` ## Jinja Format You can also use the [Jinja template format](https://huggingface.co/docs/transformers/main/en/chat_templating) as defined by Hugging Face Transformers. ```bash Command theme={null} python -m sglang.launch_server \ --model-path meta-llama/Llama-2-7b-chat-hf \ --port 30000 \ --chat-template ./my_model_template.jinja ``` # Environment Variables Source: https://docs.sglang.io/docs/references/environment_variables SGLang supports various environment variables that can be used to configure its runtime behavior. This document provides a comprehensive list and aims to stay updated over time. *Note: The canonical prefix for all SGLang environment variables is `SGLANG_`. The legacy `SGL_` prefix is deprecated: any `SGL_*` variable set in the environment is automatically rewritten to its `SGLANG_*` equivalent at import time with a deprecation warning, and the alias will be removed in a future release. A few variables keep an upstream/vendor prefix (e.g. `MOONCAKE_*`, `ASCEND_*`) because that is their canonical name.* ## General Configuration
Environment Variable Description Default Value
`SGLANG_USE_MODELSCOPE` Enable using models from ModelScope `false`
`SGLANG_HOST_IP` Host IP address for the server `0.0.0.0`
`SGLANG_PORT` Port for the server auto-detected
`SGLANG_LOGGING_CONFIG_PATH` Custom logging configuration path Not set
SGLANG\_LOG\_REQUEST\_HEADERS Comma-separated list of additional HTTP headers to log when --log-requests is enabled. Appends to the default x-smg-routing-key. Not set
SGLANG\_HEALTH\_CHECK\_TIMEOUT Timeout for health check in seconds 20
SGLANG\_EPLB\_HEATMAP\_COLLECTION\_INTERVAL The interval of passes to collect the metric of selected count of physical experts on each layer and GPU rank. 0 means disabled. 0
SGLANG\_EPLB\_P2P\_BATCH\_CHUNK\_SIZE Number of expert IDs per batch when submitting P2P ops during EPLB rebalance (CUDA and ROCm). Smaller values prevent NCCL/RCCL GPU-side accumulation hangs but increase overhead; set >= num\_physical\_experts to submit a single batch. Deprecated alias: SGLANG\_EPLB\_ROCM\_P2P\_BATCH\_CHUNK\_SIZE. 32
SGLANG\_FORWARD\_UNKNOWN\_TOOLS Forward unknown tool calls to clients instead of dropping them false (drop unknown tools)
SGLANG\_REQ\_WAITING\_TIMEOUT Timeout (in seconds) for requests waiting in the queue before being scheduled `-1`
SGLANG\_REQ\_RUNNING\_TIMEOUT Timeout (in seconds) for requests running in the decode batch `-1`
SGLANG\_CACHE\_DIR Cache directory for model weights and other data. Also the default root for compiled-kernel caches: Triton, Inductor, FlashInfer, the CUDA driver and DeepGEMM are pointed under it unless their own env vars (`TRITON_CACHE_DIR`, `TORCHINDUCTOR_CACHE_DIR`, `FLASHINFER_WORKSPACE_BASE`, `CUDA_CACHE_PATH`, `SGLANG_DG_CACHE_DIR`) are set explicitly \~/.cache/sglang
SGLANG\_QWEN4\_PLE\_FILE\_DIR Default directory for the file-backed Qwen4-Exp PLE table (`--ple-offload-backend file`); one subdirectory per model path is created under it /ple
SGLANG\_QWEN4\_PLE\_FILE\_PREFETCH Hint the page cache (`posix_fadvise(WILLNEED)`) with the rows a prefill-sized PLE gather is about to read from the file-backed table. Set to `0` to disable 1
SGLANG\_QWEN4\_PLE\_FILE\_SKIP\_DEVICE\_CHECK Skip the load-time check that the device reads pageable host memory through the host page tables (`cudaDevAttrPageableMemoryAccessUsesHostPageTables`) before using the file-backed PLE table; only for devices that cannot be queried 0
SGLANG\_QWEN4\_PLE\_FILE\_RSS\_BUDGET\_GB Cap (GiB) on the resident set of the file-backed PLE mapping. A row fault maps a whole page-cache folio, so the mapping otherwise creeps towards the full table and consumes the free memory that sizes the KV pool; over the cap the pages are dropped with `MADV_DONTNEED` and stay in the page cache. Set to `0` to disable 8.0
SGLANG\_QWEN4\_PLE\_FILE\_RSS\_INTERVAL\_S How often the file-backed PLE mapping's resident set is compared against `SGLANG_QWEN4_PLE_FILE_RSS_BUDGET_GB` 30.0
SGLANG\_CUTE\_AOT\_CACHE\_DIR Trusted directory for persistent CuTe DSL AOT objects shared across process restarts. Artifacts are namespaced by source, runtime ABI, host platform, and target GPU architecture. Set to an empty string to keep compilation process-local. Only use a directory writable by trusted users because SGLang loads cached object files into the process. /cute\_aot
SGLANG\_PREFETCH\_BLOCK\_SIZE\_MB Block size (in MB) for sequential checkpoint prefetch reads that warm the OS page cache before workers load weights via mmap 16
## Performance Tuning
Environment Variable Description Default Value
`SGLANG_ENABLE_TORCH_INFERENCE_MODE` Control whether to use torch.inference\_mode `false`
`SGLANG_ENABLE_TORCH_COMPILE` Enable torch.compile false
`SGLANG_SET_CPU_AFFINITY` Enable CPU affinity setting (often set to `1` in Docker builds) false
`SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN` Allows the scheduler to overwrite longer context length requests (often set to `1` in Docker builds) false
`SGLANG_IS_FLASHINFER_AVAILABLE` Control FlashInfer availability check `true`
`SGLANG_FLASHINFER_AUTOTUNE_CACHE` Reuse persisted FlashInfer autotune results from `SGLANG_CACHE_DIR` across runs. Set to `0` to force re-autotuning on every startup; the fresh result is written to a `runs/..json` sibling file (the canonical cache is left untouched). `true`
`SGLANG_SKIP_P2P_CHECK` Skip P2P (peer-to-peer) access check `false`
`SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD` Sets the threshold for enabling chunked prefix caching `8192`
`SGLANG_MAX_KV_CHUNK_CAPACITY` Maximum number of tokens in each KV chunk for DeepSeek MHA chunked prefix cache `131072`
`SGLANG_FUSED_MLA_ENABLE_ROPE_FUSION` Enable RoPE fusion in Fused Multi-Layer Attention `1`
`SGLANG_DISABLE_CONSECUTIVE_PREFILL_OVERLAP` Disable overlap schedule for consecutive prefill batches `false`
`SGLANG_SCHEDULER_MAX_RECV_PER_POLL` Set the maximum number of requests per poll, with a negative value indicating no limit `-1`
SGLANG\_MAX\_NEW\_TOKENS\_LIMIT Hard server-side limit for each generation request's `max_new_tokens`; requests asking for more are clipped. Disabled when unset or non-positive. Not set
`SGLANG_DATA_PARALLEL_BUDGET_INTERVAL` Interval for DPBudget updates `1`
`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_DEFAULT` Default weight value for scheduler recv skipper counter (used when forward mode doesn't match specific modes). Only active when --scheduler-recv-interval > 1. The counter accumulates weights and triggers request polling when reaching the interval threshold. `1000`
`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_DECODE` Weight increment for decode forward mode in scheduler recv skipper. Works with `--scheduler-recv-interval` to control polling frequency during decode phase. `1`
SGLANG\_SCHEDULER\_RECV\_SKIPPER\_WEIGHT\_TARGET\_VERIFY Weight increment for target verify forward mode in scheduler recv skipper. Works with `--scheduler-recv-interval` to control polling frequency during verification phase. `1`
`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_NONE` Weight increment when forward mode is None in scheduler recv skipper. Works with `--scheduler-recv-interval` to control polling frequency when no specific forward mode is active. `1`
`SGLANG_MM_BUFFER_SIZE_MB` Size of preallocated GPU buffer (in MB) for multi-modal feature hashing optimization. When set to a positive value, temporarily moves features to GPU for faster hash computation, then moves them back to CPU to save GPU memory. Larger features benefit more from GPU hashing. Set to `0` to disable. `0`
`SGLANG_MM_PRECOMPUTE_HASH` Enable precomputing of hash values for MultimodalDataItem `false`
`SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH` Enable NCCL for gathering when preparing mlp sync batch under overlap scheduler (without this flag gloo is used for gathering) `false`
`SGLANG_SYMM_MEM_PREALLOC_GB_SIZE` Size of preallocated GPU buffer (in GB) for NCCL symmetric memory pool to limit memory fragmentation. Only have an effect when server arg `--enable-symm-mem` is set. -1
SGLANG\_SKIP\_SOFTMAX\_PREFILL\_THRESHOLD\_SCALE\_FACTOR Skip-softmax threshold scale factor for TRT-LLM prefill attention in flashinfer. None means standard attention. See [https://arxiv.org/abs/2512.12087](https://arxiv.org/abs/2512.12087) None
SGLANG\_SKIP\_SOFTMAX\_DECODE\_THRESHOLD\_SCALE\_FACTOR Skip-softmax threshold scale factor for TRT-LLM decode attention in flashinfer. None means standard attention. See [https://arxiv.org/abs/2512.12087](https://arxiv.org/abs/2512.12087) None
SGLANG\_USE\_SGL\_FA3\_KERNEL Use sgl-kernel implementation for FlashAttention v3 true
## DeepGEMM Configuration (Advanced Optimization)
Environment Variable Description Default Value
`SGLANG_ENABLE_JIT_DEEPGEMM` Enable Just-In-Time compilation of DeepGEMM kernels (enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) GPUs when the DeepGEMM package is installed; set to `"0"` to disable) `"true"`
`SGLANG_JIT_DEEPGEMM_PRECOMPILE` Enable precompilation of DeepGEMM kernels `"true"`
`SGLANG_JIT_DEEPGEMM_COMPILE_WORKERS` Number of workers for parallel DeepGEMM kernel compilation `4`
`SGLANG_IN_DEEPGEMM_PRECOMPILE_STAGE` Indicator flag used during the DeepGEMM precompile script `"false"`
`SGLANG_DG_CACHE_DIR` Directory for caching compiled DeepGEMM kernels `{SGLANG_CACHE_DIR}/deep_gemm`
SGLANG\_DG\_USE\_NVRTC Use NVRTC (instead of Triton) for JIT compilation (Experimental) "false"
SGLANG\_USE\_DEEPGEMM\_BMM Use DeepGEMM for Batched Matrix Multiplication (BMM) operations `"false"`
SGLANG\_JIT\_DEEPGEMM\_FAST\_WARMUP Precompile less kernels during warmup, which reduces the warmup time from 30min to less than 3min. Might cause performance degradation during runtime. `"false"`
## DeepEP Configuration
Environment Variable Description Default Value
`SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK` The maximum number of dispatched tokens on each GPU `"128"`
`SGLANG_FLASHINFER_NUM_MAX_DISPATCH_TOKENS_PER_RANK` The maximum number of dispatched tokens on each GPU for --moe-a2a-backend=flashinfer `"1024"`
SGLANG\_FLASHINFER\_MEGAMOE\_COMBINE\_DTYPE FlashInfer NVFP4 MegaMOE cross-rank combine wire format. Supported values are bf16, mxfp8, and nvfp4. Quantized formats are incompatible with SGLANG\_FLASHINFER\_MEGAMOE\_IN\_KERNEL\_FC2\_REDUCE=1. "bf16"
SGLANG\_FLASHINFER\_MEGAMOE\_MAX\_TOKENS\_PER\_RANK Per-rank FlashInfer MegaMOE symmetric-workspace token capacity. A value of 0 derives the capacity from the runtime token limits. 0
SGLANG\_FLASHINFER\_MEGAMOE\_IN\_KERNEL\_FC2\_REDUCE Use the CuTe DSL MegaMOE in-kernel FC2 reduction. This can reduce workspace size and improve large-batch performance, but BF16 atomic accumulation is nondeterministic. It is incompatible with quantized combine dtypes. "false"
`SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS` Number of SMs used for DeepEP combine when single batch overlap is enabled `"32"`
`SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO` Run shared experts on an alternate stream when single batch overlap is enabled on GB200. When not setting this flag, shared experts and down gemm will be overlapped with DeepEP combine together. `"false"`
`SGLANG_ENABLE_QWEN_DEEPEP_SHARED_OVERLAP` Run the Qwen shared expert on an alternate CUDA stream while DeepEP executes the routed experts. `"true"`
SGLANG\_DISABLED\_MODEL\_ARCHS Comma-separated list of model architectures to disable from auto-registration. Not set
SGLANG\_SORT\_WEIGHT\_FILES Controls weight-file ordering for load-time I/O optimization. -1 disables sorting/staggering (original order); 0 sorts files only; a value k greater than 0 sorts and staggers per-rank order with factor k for better multi-rank I/O concurrency. 0
SGLANG\_RETURN\_ORIGINAL\_LOGPROB Return the original (pre-temperature) logprobs instead of the post-sampling values. false
SGLANG\_ENABLE\_COLOCATED\_BATCH\_GEN Enable colocated batch generation. false
SGLANG\_ENABLE\_MOE\_DEFERRED\_FINALIZE Defer the MoE finalize step to overlap it with other work. true
SGLANG\_PATCH\_TOKENIZER Patch the tokenizer to cache all\_special\_tokens/all\_special\_ids (notably for Kimi tiktoken, where ITL can otherwise regress under high batch size). true
SGLANG\_ENABLE\_LOGPROB\_CHUNK Process logits in chunks to reduce peak memory. true
SGLANG\_LOGPROB\_CHUNK\_SIZE Chunk size (in tokens) used when logits-processor chunking is enabled. 2048
SGLANG\_FLASHINFER\_USE\_PAGED Use the paged FlashInfer attention path. false
SGLANG\_FLASHINFER\_WORKSPACE\_SIZE FlashInfer workspace size in bytes (default ≈ 384 MiB). 402653184
SGLANG\_EAGER\_INPUT\_NO\_COPY In eager forward, wrap the ForwardBatch's own tensors instead of copying them into the CUDA graph buffer registry (skips a per-iter device-to-device copy). false
SGLANG\_DEEPGEMM\_SANITY\_CHECK Run extra sanity checks on DeepGEMM kernels. false
SGLANG\_DEEPGEMM\_PDL Enable Programmatic Dependent Launch (PDL) for DeepGEMM kernels. true
SGLANG\_PP\_PARALLEL\_DEEPGEMM\_WARMUP Run DeepGEMM warmup in parallel across pipeline-parallel ranks. false
SGLANG\_DISABLE\_STATIC\_WATERFILL Force dynamic Waterfill with runtime EP all-reduce instead of the default static local-batch path. false
SGLANG\_NIXL\_EP\_BF16\_DISPATCH Use BF16 for NIXL-EP dispatch. false
SGLANG\_NIXL\_EP\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK Maximum number of dispatched tokens per GPU for NIXL-EP. 128
## MORI Configuration
Environment Variable Description Default Value
SGLANG\_MORI\_DISPATCH\_DTYPE Override MoRI-EP dispatch quantization type. auto uses auto-detection from weight dtype; bf16/fp8/fp4 forces the specified type for all layers "auto"
SGLANG\_MORI\_COMBINE\_DTYPE Combine dtype: auto, bf16, fp8, fp4, or fp8\_direct\_cast "auto"
MORI\_DISABLE\_AUTO\_XGMI Set to 0 to allow Mori to automatically use XGMI for same-node PD disaggregation when no active RDMA device is available. unset
SGLANG\_MORI\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK Maximum number of dispatch tokens per rank for MORI-EP buffer allocation 4096
SGLANG\_MORI\_DISPATCH\_INTER\_KERNEL\_SWITCH\_THRESHOLD Threshold for switching between InterNodeV1 and InterNodeV1LL kernel types. InterNodeV1LL is used if SGLANG\_MORI\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK is less than or equal to this threshold; otherwise, InterNodeV1 is used. 256
SGLANG\_MORI\_PREALLOC\_MAX\_RECV\_TOKENS This argument devives SGLANG\_MORI\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK which indicates customized amount of tokens preallocated for a rank, valid range from 1 to world\_size\*SGLANG\_MORI\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK, by default 0 means maximum. Setting a smaller value will reduce memory footprint but too small value could cause buffer overflow. 0
SGLANG\_MORI\_MOE\_MAX\_INPUT\_TOKENS Manual MoE input row limit for MORI EPv1 and EPv2. A positive value overrides SGLANG\_MORI\_RECV\_BOUND and is used unrounded, capped at the receive capacity; it only narrows the MoE input view, not the communication buffer. It must be >= the actual received tokens, or results are incorrect. 0 leaves it to SGLANG\_MORI\_RECV\_BOUND. 0
SGLANG\_MORI\_EP\_V2 Select MORI EPv2 for --moe-a2a-backend mori. When unset, EPv2 is used only for DeepSeek-V4 models on AITER with ep\_size \<= 8 (on gfx1250, any ep\_size that fits one UALink vPOD), a bfloat16 model dtype and MORI\_ENABLE\_SDMA off, and MORI EPv1 otherwise. Set to true to use EPv2 even when these checks fail (a warning names the failed check), or false to always use EPv1. true
SGLANG\_MORI\_RECV\_BOUND Hand the MoE only the receive rows the per-rank DP token counts can fill (rounded up) instead of the full MORI receive buffer. Unverified layouts, missing metadata and two-batch overlap keep the full view. false
SGLANG\_PPLX\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK Maximum number of dispatched tokens per rank for the PPLX (--moe-a2a-backend pplx) NVSHMEM buffer allocation. Must be ≥ the worst-case per-rank forward tokens (the per-rank prefill chunk chunked\_prefill\_size / dp\_size, or the decode CUDA-graph batch size); the server refuses to start otherwise. 128
SGLANG\_MORI\_QP\_PER\_TRANSFER Number of RDMA Queue Pairs (QPs) used per transfer operation 1
SGLANG\_MORI\_POST\_BATCH\_SIZE Number of RDMA work requests posted in a single batch to each QP -1
SGLANG\_MORI\_NUM\_WORKERS Number of worker threads in the RDMA executor thread pool 1
## DSA Backend Configuration (For DeepSeek V3.2)
Environment Variable Description Default Value
SGLANG\_DSA\_FUSE\_TOPK Fuse the operation of picking topk logits and picking topk indices from page table. true
SGLANG\_DSA\_TOPK\_FLASHINFER\_DETERMINISTIC Use deterministic FlashInfer topk kernels when either --dsa-topk-backend=flashinfer or --speculative-dsa-topk-backend=flashinfer. false
SGLANG\_DSA\_TOPK\_FLASHINFER\_TIE\_BREAK Tie-break mode for FlashInfer DSA topk when either --dsa-topk-backend=flashinfer or --speculative-dsa-topk-backend=flashinfer: unset disables explicit tie-breaking, small prefers the smaller candidate index for equal scores, and large prefers the larger candidate index for equal scores. Setting this variable makes FlashInfer use deterministic topk. unset
SGLANG\_DSA\_PREFILL\_DENSE\_ATTN\_KV\_LEN\_THRESHOLD When the maximum kv len in current prefill batch exceeds this value, the sparse mla kernel will be applied, else it falls back to dense MHA implementation. Default to the index topk of model (2048 for DeepSeek V3.2). 2048
SGLANG\_DSA\_TOPK\_BROADCAST Experimental. When enabled, broadcast the finalized NSA/DSA indexer top-k result from attention TP rank 0 to the other attention TP ranks. This can mitigate top-k mismatches in TP attention runs at the cost of some speed. false
SGLANG\_MORI\_SEND\_AUX\_RDMA Send CPU-resident AUX data via RDMA instead of ZMQ TCP. false
SGLANG\_MORI\_TRANSFER\_SHARDS Number of sharded synchronous worker threads draining KV transfers; also bounds outstanding transfers (primary RDMA send-queue throttle). 8
SGLANG\_MORI\_WAIT\_POLL\_MS Poll cadence (ms) at which a transfer worker wakes to check the SLA while waiting for completion. 1000
SGLANG\_MORI\_TRANSFER\_TIMEOUT\_MS Per-transfer SLA (ms) before a KV transfer is failed; 0 disables the SLA. 0
SGLANG\_DSA\_HIP\_DISABLE\_PRESHUFFLE Disable weight pre-shuffle on the HIP DSA path. false
SGLANG\_DSA\_MQA\_LOGITS\_FREE\_MEM\_FRACTION Fraction of free memory the MQA-logits step may use on the DSA path. 0.2
## Memory Management
Environment Variable Description Default Value
SGLANG\_DEBUG\_MEMORY\_POOL Enable memory pool debugging `false`
SGLANG\_CLIP\_MAX\_NEW\_TOKENS\_ESTIMATION Clip max new tokens estimation for memory planning 4096
SGLANG\_DETOKENIZER\_MAX\_STATES Maximum states for detokenizer Default value based on system
SGLANG\_ENABLE\_TP\_MEMORY\_INBALANCE\_CHECK Enable checks for memory imbalance across Tensor Parallel ranks true
SGLANG\_MOONCAKE\_CUSTOM\_MEM\_POOL Configure the custom memory pool type for Mooncake. Supports NVLINK, BAREX, INTRA\_NODE\_NVLINK. If set to true, it defaults to NVLINK. None
## Model-Specific Options
Environment Variable Description Default Value
SGLANG\_USE\_AITER Use AITER optimize implementation `false`
SGLANG\_ROCM\_USE\_MULTI\_STREAM Allocate alt CUDA/HIP stream on ROCm/AITER to overlap shared and routed experts in DeepseekV2 MoE. Requires the HIP env GPU\_MAX\_HW\_QUEUES>=5 (default 4, the cap on HSA/ROCr HW queues HIP creates) so the alt stream gets its own queue instead of serializing with the main stream. Best paired with --deepep-mode low\_latency so Mori's AsyncLL kernel offloads dispatch/combine to copy engines and frees CUs. `false`
SGLANG\_MOE\_PADDING Enable MoE padding (sets padding size to 128 if value is 1, often set to 1 in Docker builds) `false`
SGLANG\_USE\_FUSED\_PARALLEL\_QKNORM Use the fused parallel QK RMSNorm kernel for MiniMax-M2.x on CUDA when attention TP size > 1 `false`
SGLANG\_ENABLE\_STRICT\_MEM\_CHECK\_DURING\_BUSY Enable strict memory checks while the scheduler is busy. 0
SGLANG\_ENABLE\_STRICT\_MEM\_CHECK\_DURING\_IDLE Enable strict memory checks while the scheduler is idle. true
SGLANG\_NATIVE\_MOVE\_KV\_CACHE Use the native implementation to move KV cache entries. false
SGLANG\_USE\_BREAKABLE\_CUDA\_GRAPH Use a breakable CUDA graph so it can be interrupted/rebuilt at runtime. false
SGLANG\_MEMORY\_SAVER\_CUDA\_GRAPH Allow CUDA graphs under the release/resume memory saver. false
SGLANG\_GEMMA\_OUT\_OF\_PLACE\_POSITION\_MUTATION Use out-of-place position mutation for Gemma models. false
SGLANG\_MAMBA\_CONV\_DTYPE dtype for the Mamba convolution state. bfloat16
SGLANG\_MAMBA\_SSM\_DTYPE dtype for the Mamba SSM state (defaults to the model dtype when unset). Not set
SGLANG\_EMBEDDINGS\_SPARSE\_HEAD Name of the sparse-embeddings head to expose for embedding models. Not set
SGLANG\_DSV4\_FP4\_EXPERTS Whether DeepSeek V4 experts use FP4. Set to false when using an FP4-to-FP8 converted DeepSeek V4 checkpoint. true
SGLANG\_DSV4\_REASONING\_EFFORT Default reasoning\_effort for the DeepSeek V4 chat encoder when a request does not set it. The preview profile accepts high and max; the official profile accepts low, high, and max. The profile is detected from the bundled encoder. Override it with --json-model-override-args '\{"dsv4\_reasoning\_effort\_profile":"official"}'. ""
SGLANG\_DSV41\_REASONING\_EFFORT Default reasoning\_effort for the DeepSeek-V4.1 chat encoder when a request does not set it: one of low, high, xhigh, max, or an integer budget in \[1, 100]. Unset means the encoder default high. ""
SGLANG\_DSV4\_USE\_BF16\_KV\_QUANT\_SOURCE For DeepSeek V4, quantize the SWA FP8 KV cache from BF16-rounded values instead of FP32 registers. This matches trainer-side QAT and the DSA prefill-CP path, at the cost of an extra BF16 KV materialization and separate cache-store kernels. false
## Quantization
Environment Variable Description Default Value
SGLANG\_INT4\_WEIGHT Enable INT4 weight quantization false
SGLANG\_FORCE\_FP8\_MARLIN Force using FP8 MARLIN kernels even if other FP8 kernels are available false
SGLANG\_NVFP4\_CKPT\_FP8\_GEMM\_IN\_ATTN Quantize q\_b\_proj from BF16 to FP8 when launching DeepSeek NVFP4 checkpoint false
SGLANG\_MOE\_NVFP4\_DISPATCH Use nvfp4 for moe dispatch (on flashinfer\_cutlass or flashinfer\_cutedsl moe runner backend) "false"
SGLANG\_FLASHINFER\_NVFP4\_PER\_TOKEN\_ACTIVATION Enable FlashInfer TRT-LLM or CuTe DSL v2 (no A2A or FlashInfer A2A) per-token FP32 activation scaling for serialized modelopt\_fp4 checkpoints; checkpoint activation scales are treated as 1 false
SGLANG\_FLASHINFER\_CUTEDSL\_NVFP4\_W4A16 Use BF16 activations and outputs with FlashInfer CuTe DSL NVFP4 weights on NVIDIA SM100-family GPUs (SM100/SM103). Serialized dense linear layers use this mode with --fp4-gemm-backend flashinfer\_cutedsl. MoE layers use it with --moe-runner-backend flashinfer\_cutedsl; the MoE path supports online weight quantization and serialized ModelOpt NVFP4 weights with either no A2A or FlashInfer A2A, and honors SGLANG\_FLASHINFER\_MOE\_FUSED\_FINALIZE. false
SGLANG\_FLASHINFER\_MOE\_FUSED\_FINALIZE Use FlashInfer's fused atomic CUTLASS and CuTe DSL MoE finalize. Disabled by default for better numerical accuracy. Set to 1 to opt in for more aggressive performance. Deterministic inference overrides this to false. false
FLASHINFER\_NVFP4\_4OVER6 Enable FlashInfer NVFP4 4over6 scaling for NVFP4 per-token activation and online NVFP4 MoE weight quantization paths false
FLASHINFER\_NVFP4\_4OVER6\_E4M3\_USE\_256 Use 256 as the E4M3 scale maximum for FlashInfer NVFP4 4over6 scaling; otherwise uses 448 false
SGLANG\_NVFP4\_CKPT\_FP8\_NEXTN\_MOE Quantize moe of nextn layer from BF16 to FP8 when launching DeepSeek NVFP4 checkpoint `false`
SGLANG\_QUANT\_ALLOW\_DOWNCASTING Allow weight dtype downcasting during loading (e.g., fp32 → fp16). By default, SGLang rejects this kind of downcasting when using quantization. `false`
SGLANG\_FP8\_IGNORED\_LAYERS A comma-separated list of layer names to ignore during FP8 quantization. For example: model.layers.0,model.layers.1.,qkv\_proj. ""
SGLANG\_FP4\_IGNORED\_LAYERS A comma-separated list of layer names to keep out of online FP4 conversion for modelopt\_fp4 or nvfp4\_online. For example: model.layers.40,model.layers.41. ""
SGLANG\_HUMMING\_ONLINE\_QUANT\_CONFIG JSON object or JSON file path for Humming online weight quantization. When a layer has no checkpoint quantization config, this config tells Humming how to quantize the loaded fp16/bf16 weight. When the checkpoint already has a Humming config, add "force\_requant": true to requantize it to this schema during loading. Common keys include dtype/weight\_dtype, scale\_dtype, group\_size, scale\_type, ignored\_layers, ignore, and modules\_to\_not\_convert. None
SGLANG\_HUMMING\_INPUT\_QUANT\_CONFIG JSON object or JSON file path for Humming input activation quantization. This controls the activation dtype and scale grouping passed into Humming kernels, independently of the weight schema. For example, quantizes Humming inputs to FP8 E4M3. None
SGLANG\_HUMMING\_USE\_F16\_ACCUM Use FP16 accumulation in Humming compute/tuning config. This is only meaningful for Humming dtype combinations that support FP16 accumulation, such as fp16 or FP8 E4M3 activations with float16 output. Leave it false for the default accumulator behavior. false
SGLANG\_HUMMING\_MOE\_GEMM\_TYPE Select the Humming MoE GEMM path for standard dispatch and DeepEP normal dispatch. indexed uses top-k expert ids directly and is the fallback for unset or unknown values. grouped maps to Humming grouped-contiguous GEMM. DeepEP low-latency dispatch uses grouped-masked GEMM internally and does not use this selector. "" (indexed)
## Distributed Computing
Environment Variable Description Default Value
`SGLANG_BLOCK_NONZERO_RANK_CHILDREN` Control blocking of non-zero rank children processes `1`
`SGLANG_IS_FIRST_RANK_ON_NODE` Indicates if the current process is the first rank on its node `"true"`
`SGLANG_PP_LAYER_PARTITION` Pipeline parallel layer partition specification Not set
`SGLANG_ONE_VISIBLE_DEVICE_PER_PROCESS` Set one visible device per process for distributed computing `false`
SGLANG\_RAY\_BUNDLE\_INDICES Comma-separated bundle indices for Ray actor placement (e.g., "0,1,2,3"). Must match world\_size. Enables fine-grained GPU assignment in custom placement groups. Not set
SGLANG\_CPU\_QUANTIZATION Enable CPU-side quantization. false
SGLANG\_USE\_DYNAMIC\_MXFP4\_LINEAR Use dynamic MXFP4 quantization for linear layers. false
USE\_TRITON\_W8A8\_FP8\_KERNEL Use the Triton W8A8 FP8 kernel. false
SGLANG\_USE\_MESSAGE\_QUEUE\_BROADCASTER Use the shared-memory message-queue broadcaster for inter-process tensor broadcast. true
SGLANG\_DISTRIBUTED\_INIT\_METHOD\_OVERRIDE Override the init method used by torch.distributed.init\_process\_group. Set to env:// to use an externally-created TCPStore via MASTER\_ADDR/MASTER\_PORT. Not set
SGLANG\_TCP\_STORE\_PORT Port for the torch.distributed TCPStore. 29600
SGLANG\_SYNC\_TOKEN\_IDS\_ACROSS\_TP Synchronize sampled token ids across tensor-parallel ranks. false
## PD Disaggregation — Staging Buffer (Heterogeneous TP)
Environment Variable Description Default Value
SGLANG\_DISAGG\_STAGING\_BUFFER Enable GPU staging buffer for heterogeneous TP KV transfer. Required when prefill and decode use different TP/attention-TP sizes. Only for non-MLA models (e.g. GQA, MHA). false
SGLANG\_DISAGG\_STAGING\_POOL\_SIZE\_MB Decode-side ring buffer pool total size in MB. Shared buffer receiving RDMA data from all prefill ranks. Larger values support higher concurrency. 4096
SGLANG\_STAGING\_USE\_TORCH Force using PyTorch gather/scatter fallback instead of Triton fused kernels for staging operations. Useful for debugging. false
## Testing & Debugging (Internal/CI) *These variables are primarily used for internal testing, continuous integration, or debugging.*
Environment Variable Description Default Value
SGLANG\_IS\_IN\_CI Indicates if running in CI environment false
SGLANG\_IS\_IN\_CI\_AMD Indicates running in AMD CI environment false
SGLANG\_TEST\_RETRACT Enable retract decode testing `false`
SGLANG\_TEST\_RETRACT\_NO\_PREFILL\_BS When SGLANG\_TEST\_RETRACT is enabled, no prefill is performed if the batch size exceeds SGLANG\_TEST\_RETRACT\_NO\_PREFILL\_BS. 2 \*\* 31
SGLANG\_RECORD\_STEP\_TIME Record step time for profiling `false`
SGLANG\_TEST\_REQUEST\_TIME\_STATS Test request time statistics `false`
SGLANG\_DEBUG\_SYMM\_MEM Enable debug checks that verify tensors passed to NCCL communication ops are allocated in the symmetric memory pool. Logs warnings (rank 0 only) with stack traces for any tensor not in the pool. `false`
SGLANG\_KERNEL\_API\_LOGLEVEL Controls crash-debug kernel API logging. 0 disables logging, 1 logs API names, 3 logs tensor metadata, 5 adds tensor statistics, and 10 also writes pre-call dump snapshots. 0
SGLANG\_KERNEL\_API\_LOGDEST Destination for crash-debug kernel API logs. Use stdout, stderr, or a file path. %i is replaced with the process PID. stdout
SGLANG\_KERNEL\_API\_DUMP\_DIR Output directory for level-10 kernel API input/output dumps. %i is replaced with the process PID. sglang\_kernel\_api\_dumps
SGLANG\_KERNEL\_API\_DUMP\_INCLUDE Comma-separated wildcard patterns for kernel API names to include in level-10 dumps. Not set
SGLANG\_KERNEL\_API\_DUMP\_EXCLUDE Comma-separated wildcard patterns for kernel API names to exclude from level-10 dumps. Not set
## Profiling & Benchmarking
Environment Variable Description Default Value
`SGLANG_TORCH_PROFILER_DIR` Directory for PyTorch profiler output `/tmp`
`SGLANG_PROFILE_WITH_STACK` Set `with_stack` option (bool) for PyTorch profiler (capture stack trace) `true`
`SGLANG_PROFILE_RECORD_SHAPES` Set `record_shapes` option (bool) for PyTorch profiler (record shapes) `true`
`SGLANG_OTLP_EXPORTER_SCHEDULE_DELAY_MILLIS` Config BatchSpanProcessor.schedule\_delay\_millis if tracing is enabled `500`
`SGLANG_OTLP_EXPORTER_MAX_EXPORT_BATCH_SIZE` Config BatchSpanProcessor.max\_export\_batch\_size if tracing is enabled `64`
`SGLANG_TRACE_ASYNC` Enable async tracing: span creation is offloaded to a dedicated exporter process via ZMQ, reducing OTel overhead on the inference hot path `false`
`SGLANG_TRACE_ASYNC_FLUSH_THRESHOLD` Number of buffered trace operations before an automatic flush to the exporter process `100`
SGLANG\_PROFILE\_V2 Use the v2 profiler implementation. false
SGLANG\_DETECT\_SLOW\_RANK Detect and report ranks that fall behind during collective ops. false
SGLANG\_ENABLE\_RANK\_CONSENSUS\_CHECKER Check for PP/TP-rank divergence. Kill the server when divergence occurs. Helpful for trouble-shooting server hangs issues. False
SGLANG\_FORCE\_SHUTDOWN Force an immediate process-group shutdown on exit. false
SGLANG\_PYSPY\_DUMP\_BEFORE\_CRASH Capture a py-spy stack dump of all processes before crashing. true
SGLANG\_CUDA\_COREDUMP Enable CUDA coredump generation (auto-injects the required CUDA\_\* env vars). false
SGLANG\_CUDA\_COREDUMP\_DIR Directory for CUDA coredumps. If unset, resolves to RUNNER\_TEMP in CI, else /tmp. Not set
SGLANG\_CUDA\_COREDUMP\_BEFORE\_CRASH Trigger a CUDA coredump before crashing. true
SGLANG\_CUDA\_COREDUMP\_BEFORE\_CRASH\_WAIT\_SECS Seconds to wait for the CUDA coredump to finish before exiting. 60.0
## Storage & Caching
Environment Variable Description Default Value
SGLANG\_WAIT\_WEIGHTS\_READY\_TIMEOUT Timeout period for waiting on weights 120
SGLANG\_DISABLE\_OUTLINES\_DISK\_CACHE Disable Outlines disk cache false
SGLANG\_USE\_CUSTOM\_TRITON\_KERNEL\_CACHE Use SGLang's custom Triton kernel cache implementation for lower overheads (automatically enabled on CUDA) false
SGLANG\_HICACHE\_DECODE\_OFFLOAD\_STRIDE Decode-side incremental KV cache offload stride. Rounded down to a multiple of --page-size (min is --page-size). If unset/invalid/\<=0, it falls back to --page-size. Not set (uses --page-size)
SGLANG\_HICACHE\_NIXL\_USE\_DIRECT\_IO Enable O\_DIRECT for any file-based NIXL backend (POSIX, GDS, GDS\_MT, 3FS) when opening cache files (bypasses the OS page cache, reducing memory pressure and improving throughput on NVMe). Can also be disabled via in --hicache-storage-backend-extra-config. Falls back to buffered I/O with a warning when O\_DIRECT is unavailable on the current OS. true
SGLANG\_HUGEPAGE\_SIZE Use huge pages for host KV cache allocations (HiCache / disaggregation offload). Valid values: 2MB (2 MiB pages via MAP\_HUGE\_2MB) or 1GB (1 GiB pages via MAP\_HUGE\_1GB). Requires huge pages to be pre-allocated on the host OS (/proc/sys/vm/nr\_hugepages or /sys/kernel/mm/hugepages). If the allocation fails, the allocator logs a warning and falls back to regular page-size mmap automatically. Not set (uses OS default page size)
SGLANG\_HICACHE\_HF3FS\_CONFIG\_PATH Path to the HiCache HF3FS backend config file. Not set
SGLANG\_HICACHE\_FILE\_BACKEND\_STORAGE\_DIR Storage directory for the HiCache file backend. Not set
SGLANG\_HICACHE\_FILE\_BACKEND\_MAX\_SIZE Max size for HiCache file-backend LRU eviction (accepts SI/IEC suffixes; 0 disables eviction). Not set (eviction off)
SGLANG\_HICACHE\_FILE\_BACKEND\_EVICTION\_RATIO Target fraction to evict down to when the file-backend max size is reached. 0.9
SGLANG\_HICACHE\_FILE\_BACKEND\_MIN\_FREE\_SPACE Minimum free space to keep on the file-backend volume (accepts SI/IEC suffixes). 0
SGLANG\_HICACHE\_NIXL\_BACKEND\_STORAGE\_DIR Storage directory for the HiCache NIXL backend. Not set
## Function Calling / Tool Use
Environment Variable Description Default Value
SGLANG\_TOOL\_STRICT\_LEVEL Controls the strictness level of tool call parsing and validation. \
Level 0: Off - No strict validation \
Level 1: Function strict - Enables structural tag constraints for all tools (even if none have strict=True set) \
Level 2: Parameter strict - Enforces strict parameter validation for all tools, treating them as if they all have strict=True set		 0
SGLANG\_DEFAULT\_THINKING Enable model thinking/reasoning output by default. false
SGLANG\_MAX\_THINK\_TOKENS Cap on thinking tokens. Negative means unlimited; 0 or greater caps the count. -1
## Logging & Observability
Environment Variable Description Default Value
SGLANG\_LOG\_GC Log Python garbage-collection pauses. false
SGLANG\_LOG\_FORWARD\_ITERS Log each forward iteration. false
SGLANG\_LOG\_MS Log per-step timing in milliseconds. false
SGLANG\_LOG\_REQUEST\_EXCEEDED\_MS Log requests whose processing time exceeds this many milliseconds. -1 disables. -1
SGLANG\_LOG\_SCHEDULER\_STATUS\_TARGET Target (e.g. a file path) for periodic scheduler-status logging. ""
SGLANG\_LOG\_SCHEDULER\_STATUS\_INTERVAL Interval (seconds) between scheduler-status log lines. 60.0
## Constrained Decoding (Grammar)
Environment Variable Description Default Value
SGLANG\_GRAMMAR\_POLL\_INTERVAL Poll interval (seconds) for asynchronous grammar compilation. 0.005
SGLANG\_GRAMMAR\_MAX\_POLL\_ITERATIONS Maximum poll iterations before grammar compilation is treated as stuck. 10000
## Scheduler & Batching
Environment Variable Description Default Value
SGLANG\_INIT\_NEW\_TOKEN\_RATIO Initial new-token ratio used for memory planning. 0.7
SGLANG\_MIN\_NEW\_TOKEN\_RATIO\_FACTOR Floor factor for the new-token ratio after decay. 0.14
SGLANG\_NEW\_TOKEN\_RATIO\_DECAY\_STEPS Number of steps over which the new-token ratio decays. 600
SGLANG\_RETRACT\_DECODE\_STEPS Number of decode steps to look ahead when deciding to retract. 20
SGLANG\_EMPTY\_CACHE\_INTERVAL Interval (seconds) at which to empty the device cache; set this if memory accumulates over a long serving period. -1 disables. -1
SGLANG\_FORCE\_STREAM\_INTERVAL For non-streaming requests, flush intermediate output batches to the tokenizer manager every N decoded tokens (lower to 1 for accurate TTFT benchmarking). 50
SGLANG\_DYNAMIC\_CHUNKING\_SMOOTH\_FACTOR Smoothing factor for dynamic prefill chunking. 0.75
SGLANG\_SWA\_EVICTION\_INTERVAL\_MULTIPLIER Multiplier applied to the sliding-window-attention eviction interval. 1.0
SGLANG\_RADIX\_FORCE\_MISS Force radix-cache misses (debugging/benchmarking). false
SGLANG\_SCHEDULER\_SKIP\_ALL\_GATHER Skip the scheduler all-gather step. false
SGLANG\_ENABLE\_WAR\_BARRIER Force-enable the write-after-read barrier for the overlap scheduler even when CUDA is not detected (e.g. AMD/ROCm). On CUDA the barrier is always enabled. false
SGLANG\_PP\_SKIP\_PURE\_CHUNKED\_OUTPUT\_COMM In pipeline parallel, skip output send/recv when a batch is entirely non-final chunked-prefill requests. false
SGLANG\_KILLPG\_ON\_SCHEDULER\_EXCEPTION Kill the whole process group when the scheduler raises an exception. false
SGLANG\_REQUEST\_STATE\_WAIT\_TIMEOUT Tokenizer-manager request-state wait timeout (seconds). 4
## PD Disaggregation (Runtime)
Environment Variable Description Default Value
SGLANG\_DISAGGREGATION\_THREAD\_POOL\_SIZE Thread-pool size for KV transfers. Defaults to a value computed from the CPU count at runtime. Not set (computed at runtime)
SGLANG\_DISAGGREGATION\_QUEUE\_SIZE Disaggregation transfer queue size. 4
SGLANG\_DISAGGREGATION\_BOOTSTRAP\_TIMEOUT Timeout (seconds) for the disaggregation bootstrap handshake. 300
SGLANG\_DISAGGREGATION\_WAITING\_TIMEOUT Timeout (seconds) for a request waiting on KV transfer. 300
SGLANG\_DISAGGREGATION\_HEARTBEAT\_INTERVAL Interval (seconds) between disaggregation heartbeats. 5.0
SGLANG\_DISAGGREGATION\_HEARTBEAT\_MAX\_FAILURE Consecutive heartbeat failures tolerated before a peer is considered dead. 2
SGLANG\_DISAGGREGATION\_ZMQ\_SEND\_TIMEOUT Send timeout (seconds) for decode's ZMQ sockets to prefill peers. Raise if healthy sends exceed the default. 1
SGLANG\_DISAGGREGATION\_BOOTSTRAP\_ENTRY\_CLEANUP\_INTERVAL Interval (seconds) for cleaning up stale bootstrap entries. 120
SGLANG\_DISAGGREGATION\_NIXL\_BACKEND NIXL transport backend for disaggregation. UCX
SGLANG\_DISAGGREGATION\_NIXL\_BACKEND\_PARAMS JSON parameters passed to the NIXL backend.
SGLANG\_DISAGGREGATION\_ALL\_CP\_RANKS\_TRANSFER Have all context-parallel ranks participate in KV transfer. false
SGLANG\_DISAGGREGATION\_FORCE\_QUERY\_PREFILL\_DP\_RANK Force querying the prefill DP rank for routing. false
SGLANG\_DISAGGREGATION\_NUM\_PRE\_ALLOCATE\_REQS Extra slots in req\_to\_token\_pool for decode workers (effective when max\_num\_reqs greater than 32), letting more KV transfers overlap decode. 0
## Mooncake KV Store & Transfer
Environment Variable Description Default Value
SGLANG\_HICACHE\_MOONCAKE\_CONFIG\_PATH Path to the HiCache Mooncake store config file. Not set
SGLANG\_HICACHE\_MOONCAKE\_REUSE\_TE Reuse the Mooncake transfer engine across HiCache operations. true
SGLANG\_MOONCAKE\_SEND\_AUX\_TCP Send Mooncake AUX data over TCP. false
SGLANG\_ENABLE\_FAILED\_SESSION\_PROBE Probe failed Mooncake sessions for recovery. false
SGLANG\_FAILED\_SESSION\_PROBE\_INTERVAL\_S Interval (seconds) between failed-session probes. 30.0
MOONCAKE\_MASTER Address of the Mooncake master. Not set
MOONCAKE\_CLIENT Mooncake client identifier. Not set
MOONCAKE\_LOCAL\_HOSTNAME Local hostname advertised to Mooncake. localhost
MOONCAKE\_TE\_META\_DATA\_SERVER Mooncake transfer-engine metadata server. P2PHANDSHAKE
MOONCAKE\_GLOBAL\_SEGMENT\_SIZE Mooncake global segment size. 4gb
MOONCAKE\_PROTOCOL Mooncake transport protocol. rdma
MOONCAKE\_DEVICE Mooncake RDMA device(s). ""
MOONCAKE\_MASTER\_METRICS\_PORT Port for Mooncake master metrics. 9003
MOONCAKE\_CHECK\_SERVER Check connectivity to the Mooncake server on startup. false
MOONCAKE\_STANDALONE\_STORAGE Run Mooncake in standalone storage mode. false
MOONCAKE\_ENABLE\_SSD\_OFFLOAD Enable SSD offload in Mooncake. false
MOONCAKE\_OFFLOAD\_FILE\_STORAGE\_PATH File storage path for Mooncake SSD offload. Not set
ENABLE\_ASCEND\_TRANSFER\_WITH\_MOONCAKE Enable Ascend NPU transfers via Mooncake. false
ASCEND\_NPU\_PHY\_ID Physical Ascend NPU id used for Mooncake transfers. -1 auto-detects. -1
## Attention & Kernels
Environment Variable Description Default Value
SGLANG\_TRITON\_DECODE\_ATTN\_STATIC\_KV\_SPLITS Use static KV splits in the Triton decode-attention kernel. false
SGLANG\_MUSA\_FA3\_FORCE\_UPDATE\_METADATA Force FA3 metadata updates on MThreads MUSA. false
SGLANG\_SKIP\_SGL\_KERNEL\_VERSION\_CHECK Skip the sgl-kernel version compatibility check. false
## Deterministic Inference
Environment Variable Description Default Value
SGLANG\_ENABLE\_DETERMINISTIC\_INFERENCE Enable deterministic inference (fixed reduction/accumulation order). false
SGLANG\_USE\_1STAGE\_ALLREDUCE Use the 1-stage all-reduce kernel on AMD (deterministic, fixed accumulation order). If unset, it is auto-enabled when deterministic inference is on. false
SGLANG\_ENABLE\_PCIE\_IPC\_ALLREDUCE Use FlashInfer's PCIe-IPC all-reduce on switch-free intra-node hosts (no NVLink, no multicast), where the other custom all-reduce backends do not apply. Serves bf16 decode reductions that fit the workspace bound; everything else stays on NCCL. Left off under deterministic inference. false
SGLANG\_PCIE\_IPC\_MAX\_NUMEL Elements the PCIe-IPC workspace is sized for; it cannot grow after construction. 0 derives it from the widest decode batch the runner captures for. Raising it hands larger reductions to the kernels, at roughly 2 \* world\_size \* max\_numel \* itemsize bytes per rank. 0
SGLANG\_FLASHINFER\_PREFILL\_SPLIT\_TILE\_SIZE FlashInfer prefill split-tile size for deterministic attention. 4096
SGLANG\_FLASHINFER\_DECODE\_SPLIT\_TILE\_SIZE FlashInfer decode split-tile size for deterministic attention. 2048
SGLANG\_TRITON\_DECODE\_SPLIT\_TILE\_SIZE Triton decode split-tile size for deterministic attention. 256
## Speculative Decoding & Overlap
Environment Variable Description Default Value
SGLANG\_ENABLE\_OVERLAP\_PLAN\_STREAM Plan the next step on a separate stream to overlap with the current step (Overlap Spec V2). false
SGLANG\_SPEC\_SKIP\_ZERO\_STEP\_DRAFT\_EXTEND Skip draft\_extend while adaptive spec is at steps=0; saves a draft forward but the draft KV goes stale. false
SGLANG\_NGRAM\_FORCE\_GREEDY\_VERIFY Force greedy verification for the n-gram speculative path. false
SGLANG\_SANITIZE\_NAN\_LOGITS Sanitize NaN logits before sampling kernels and emit a throttled warning. true
## EPLB (Expert Parallel Load Balancing)
Environment Variable Description Default Value
SGLANG\_EXPERT\_DISTRIBUTION\_RECORDER\_DIR Output directory for the expert-distribution recorder. /tmp
SGLANG\_LOG\_EXPERT\_LOCATION\_METADATA Log expert-location metadata. false
SGLANG\_EXPERT\_LOCATION\_UPDATER\_LOG\_INPUT Log inputs to the expert-location updater. false
SGLANG\_EXPERT\_LOCATION\_UPDATER\_LOG\_METRICS Log metrics from the expert-location updater. false
## AMD & ROCm
Environment Variable Description Default Value
SGLANG\_USE\_AITER\_AG Use the AITER all-gather implementation. true
SGLANG\_USE\_AITER\_UNIFIED\_ATTN Use the AITER unified attention kernel. false
SGLANG\_USE\_AITER\_FP8\_PER\_TOKEN Use AITER FP8 per-token quantization. false
SGLANG\_AITER\_HONOR\_EXPLICIT\_MEM\_FRACTION Use an explicitly passed --mem-fraction-static as-is on AITER with context length above 8192, instead of scaling it by 0.85 to reserve non-static attention workspace. Skipping the reserve can OOM long-context serving; set it only when the scaled fraction is too small to hold the model weights. false
SGLANG\_AITER\_GFX942\_BLOCKSCALE\_USE\_CK On gfx942, run dense block-FP8 linears with the AITER CK blockscale GEMM instead of the AITER Triton one. Faster, especially at decode batch sizes, but check accuracy: it lowers GSM8K on Qwen3-32B-FP8 by about 2 points. false
SGLANG\_USE\_AITER\_MOE\_GU\_ITLV Select the AITER MoE gate/up tile layout: true interleaves, false uses the separated layout. true
SGLANG\_AITER\_FUSE\_RMSNORM\_PAD Fuse the residual-add + RMSNorm + zero-pad triplet before the MoE block via the AITER Triton kernel (TP=1, post-attention layernorm path only). false
SGLANG\_AITER\_KV\_CACHE\_LAYOUT Physical layout for the MHA KV cache on AITER: nhd or vectorized\_5d (SHUFFLE layout enabling pa\_decode\_gluon). nhd
SGLANG\_AITER\_PAGED\_PREFILL\_ASM Let gfx950 fp8 hd256 chunked prefill read the KV pool in place through AITER's paged-varlen ASM kernel, instead of gathering the prefix into a contiguous buffer first. The ASM kernel is compiled for a 4D LINEAR \[N, 64, H, D] KV cache, so it serves --page-size 64 on the NHD pool and nothing else; at any other page size this variable has no effect whichever way it is set. Where it does apply, the gain grows with context and is flat to marginally negative at moderate ISL, so set it to false for workloads that are not long-context. true
SGLANG\_ROCM\_FUSED\_DECODE\_MLA Use the fused decode MLA kernel on ROCm. false
SGLANG\_ROCM\_DISABLE\_LINEARQUANT Disable linear-layer quantization on ROCm. false
SGLANG\_HACK\_FLASHMLA\_BACKEND DeepSeek-V4 HIP decode attention kernel: auto takes aiter's gluon sparse kernel on gfx950 and the tilelang partial + combine elsewhere; aiter\_sparse, tilelang, triton, torch, comparison or unified\_kv\_triton force one. auto
SGLANG\_ROCM\_K3\_FUSE\_KDA\_INPROJ Kimi-K3 on ROCm: fold the KDA \[f\_a|b] tail into the wide \[q,k,v,g] projection so the whole input projection is one GEMM. Applies to unquantized weights only; falls back to the split projection otherwise. true
SGLANG\_ROCM\_K3\_FUSE\_KDA\_INPROJ\_MAX\_TOKENS Token count above which SGLANG\_ROCM\_K3\_FUSE\_KDA\_INPROJ stops applying and the split projection runs instead. The merged shape is only faster while the projection is bandwidth bound. 256
## NPU (Ascend)
Environment Variable Description Default Value
SGLANG\_NPU\_DISABLE\_ACL\_FORMAT\_WEIGHT Disable ACL-format weight conversion on NPU. false
SGLANG\_NPU\_USE\_MULTI\_STREAM Use multiple streams on NPU. false
SGLANG\_NPU\_USE\_MLAPO Use the MLAPO path on NPU. false
SGLANG\_NPU\_FORWARD\_NATIVE\_GELUTANH Use the native gelu-tanh activation forward (for Skywork-Reward-Gemma-2-27B-v0.2). false
SGLANG\_NPU\_FORWARD\_NATIVE\_GEMMA\_RMS\_NORM Use the native Gemma RMSNorm forward (for Skywork-Reward-Gemma-2-27B-v0.2). false
SGLANG\_USE\_AG\_AFTER\_QLORA Delay all-gather until after QLoRA for better DeepSeek V3.2 performance. false
SGLANG\_EXPERIMENTAL\_LORA\_OPTI Master switch for the experimental TRT-LLM LoRA fast path. When off, all fine-grained opt switches read false. false
SGLANG\_ZBAL\_LOCAL\_MEM\_SIZE Local memory size for the ZBAL (zero-buffer accelerate library) path (NPU only). 0
SGLANG\_ZBAL\_BOOTSTRAP\_URL Bootstrap URL for the ZBAL path (NPU only). ""
## Apple Silicon (MLX / MPS) These variables configure the SRT MLX backend. SGLang Diffusion uses PyTorch MPS and does not read them.
Environment Variable Description Default Value
SGLANG\_USE\_MLX Use the MLX backend on Apple Silicon. false
SGLANG\_MLX\_USE\_CUSTOM\_ROPE Use the custom RoPE kernel on MLX. false
SGLANG\_MLX\_FUSE\_SWIGLU Fuse the SwiGLU activation on MLX. false
SGLANG\_MLX\_CLEAR\_CACHE\_STEPS Number of decode steps between mx.clear\_cache() calls. 0 disables cache clearing. 256
## Multimodal (VLM)
Environment Variable Description Default Value
SGLANG\_VLM\_CACHE\_SIZE\_MB Size (MB) of the VLM feature cache. 100
SGLANG\_IMAGE\_MAX\_PIXELS Maximum number of pixels per image before resizing. 12845056
SGLANG\_RESIZE\_RESAMPLE Resampling filter used when resizing images (e.g. bilinear, bicubic). ""
SGLANG\_MM\_SKIP\_COMPUTE\_HASH Skip computing multimodal-item hashes. false
SGLANG\_MM\_AVOID\_RETOKENIZE For pre-tokenized (list\[int]) multimodal prompts, preserve the user's original tokens to avoid retokenization drift. true
SGLANG\_VIT\_ENABLE\_CUDA\_GRAPH Capture the vision encoder (ViT) in a CUDA graph. false
SGLANG\_USE\_CUDA\_IPC\_TRANSPORT Use CUDA IPC transport for multimodal-item tensors. false
SGLANG\_USE\_IPC\_POOL\_HANDLE\_CACHE When CUDA IPC multimodal feature transport is selected, reuse mappings to its existing bounded pool. This does not enable CUDA IPC transport or reserve another pool. true
SGLANG\_MM\_FEATURE\_CACHE\_MB Size (MB) of the multimodal feature cache. 1024
SGLANG\_MM\_ITEM\_MEM\_POOL\_RECYCLE\_INTERVAL\_SEC Interval (seconds) for recycling the multimodal-item memory pool. 0.05
## Encoder / EPD (Multimodal Disaggregation)
Environment Variable Description Default Value
SGLANG\_ENCODER\_MM\_RECEIVER\_MODE Encoder receiver selection used by EPD paths: http or grpc. http
SGLANG\_ENCODER\_GRPC\_TIMEOUT\_SECS gRPC timeout (seconds) for encoder communication. 60
SGLANG\_ENCODER\_RECV\_TIMEOUT Encoder receive timeout (seconds). 180.0
SGLANG\_ENCODER\_SEND\_TIMEOUT Encoder send timeout (seconds). 180.0
SGLANG\_ENCODER\_HTTP\_TIMEOUT Encoder HTTP timeout (seconds). 1800.0
SGLANG\_ENCODER\_REQ\_TIMEOUT Encoder per-request timeout (seconds). 180.0
SGLANG\_ENCODER\_DISPATCH\_MIN\_ITEMS Minimum items before the encoder dispatches a batch. 2
SGLANG\_ENCODER\_MAX\_BATCH\_SIZE Maximum encoder batch size. 8
SGLANG\_ENCODER\_PREPROC\_WORKERS Number of encoder preprocessing workers. 8
SGLANG\_ENCODER\_IMAGE\_PROCESSOR\_USE\_GPU Run the image processor on the GPU. false
SGLANG\_ENCODER\_BOOTSTRAP\_HEALTH\_CHECK\_INTERVAL EncoderBootstrapServer health-check interval (seconds). 0 disables it. 10.0
SGLANG\_ENCODER\_BOOTSTRAP\_HEALTH\_CHECK\_TIMEOUT EncoderBootstrapServer health-check timeout (seconds). 2.0
SGLANG\_EMBEDDING\_POOL\_SIZE\_MB Persistent receiver-side GPU embedding pool size (MB) for Mooncake EPD transport. 0 disables (per-request register/deregister). 4096
SGLANG\_ENCODER\_DP\_WORKER\_MAX\_INFLIGHT Maximum in-flight requests per encoder DP worker. 64
SGLANG\_BACKUP\_PORT\_BASE Base port for elastic-EP backup ports. 10000
## HTTP & gRPC Server
Environment Variable Description Default Value
SGLANG\_TIMEOUT\_KEEP\_ALIVE HTTP keep-alive timeout (seconds). 5
SGLANG\_UVICORN\_WORKER\_HEALTHCHECK\_TIMEOUT Uvicorn multiprocess supervisor per-worker health-check interval (seconds). 10
SGLANG\_ENABLE\_HEALTH\_ENDPOINT\_GENERATION Have the /health endpoint run a generation as part of the check. true
SGLANG\_WARMUP\_TIMEOUT If a warmup forward batch takes longer than this many seconds, the server crashes to avoid hanging. -1 disables; increase (e.g. to 1800) to accommodate kernel JIT precompile. -1
SGLANG\_GRPC\_PORT Port for the native gRPC server. Not set
SGLANG\_GRANIAN\_PARENT\_PID Parent PID for the Granian HTTP/2 worker supervisor. Not set
## NUMA & CPU
Environment Variable Description Default Value
SGLANG\_NUMA\_BIND\_V2 Select the NUMA binding implementation. true uses a pre-launch numactl wrapper; false binds in the worker process. This variable does not enable or disable NUMA binding. true
SGLANG\_AUTO\_NUMA\_BIND Automatically detect each NVIDIA GPU's local NUMA node and bind its worker when --numa-node is not specified. Set to false to leave workers unbound; an explicit --numa-node takes precedence. true
SGLANG\_CRASH\_ON\_NUMA\_BIND\_FAILURE Crash if NUMA binding fails instead of warning. false
## Metrics
Environment Variable Description Default Value
SGLANG\_ENABLE\_METRICS\_DEVICE\_TIMER Enable device-timer-based metrics. false
SGLANG\_ENABLE\_METRICS\_DP\_ATTENTION Enable data-parallel attention metrics. false
## External Models
Environment Variable Description Default Value
SGLANG\_EXTERNAL\_MODEL\_PACKAGE Python package providing external model implementations. ""
SGLANG\_EXTERNAL\_MM\_MODEL\_ARCH External multimodal model architecture name. ""
SGLANG\_EXTERNAL\_MM\_PROCESSOR\_PACKAGE Python package providing the external multimodal processor. ""
## Plugin System
Environment Variable Description Default Value
SGLANG\_PLATFORM Platform plugin name to load. ""
SGLANG\_PLUGINS Comma-separated list of plugins to load. ""
# Troubleshooting and Frequently Asked Questions Source: https://docs.sglang.io/docs/references/faq ## Troubleshooting This page lists common errors and tips for resolving them. ### CUDA Out of Memory If you encounter out-of-memory (OOM) errors, you can adjust the following parameters: * If OOM occurs during prefill, try reducing `--chunked-prefill-size` to `4096` or `2048`. This saves memory but slows down the prefill speed for long prompts. * If OOM occurs during decoding, try lowering `--max-running-requests`. * You can also decrease `--mem-fraction-static` to a smaller value, such as 0.8 or 0.7. This decreases the memory usage of the KV cache memory pool and helps prevent OOM errors during both prefill and decoding. However, it limits maximum concurrency and reduces peak throughput. * Another common case for OOM is requesting input logprobs for a long prompt as it requires significant memory. To address this, set `logprob_start_len` in your sampling parameters to include only the necessary parts. If you do need input logprobs for a long prompt, try reducing `--mem-fraction-static`. ### CUDA Error: Illegal Memory Access Encountered This error may result from kernel errors or out-of-memory issues: * If it is a kernel error, resolving it may be challenging. Please file an issue on GitHub. * If it is an out-of-memory issue, it may sometimes be reported as this error instead of "Out of Memory." Refer to the section above for guidance on avoiding OOM issues. ### The server hangs * If the server hangs during initialization or running, it can be memory issues (out of memory), network issues (nccl errors), or other bugs in sglang. * If it is out of memory, you might see that `avail mem` is very low during the initialization or right after initialization. In this case, you can try to decrease `--mem-fraction-static`, decrease `--cuda-graph-max-bs-decode`, or decrease `--chunked-prefill-size`. * Other bugs, please file an issue on GitHub. ## Frequently Asked Questions ### The results are not deterministic, even with a temperature of 0 You may notice that when you send the same request twice, the results from the engine will be slightly different, even when the temperature is set to 0. From our initial investigation, this indeterminism arises from two factors: dynamic batching and prefix caching. Roughly speaking, dynamic batching accounts for about 95% of the indeterminism, while prefix caching accounts for the remaining portion. The server runs dynamic batching under the hood. Different batch sizes can cause PyTorch/CuBLAS to dispatch to different CUDA kernels, which can lead to slight numerical differences. This difference accumulates across many layers, resulting in nondeterministic output when the batch size changes. Similarly, when prefix caching is enabled, it can also dispatch to different kernels. Even when the computations are mathematically equivalent, small numerical differences from different kernel implementations lead to the final nondeterministic outputs. To achieve more deterministic outputs in the current code, you can add `--disable-radix-cache` and send only one request at a time. The results will be mostly deterministic under this setting. **Update**: Recently, we also introduced a deterministic mode, you can enable it with `--enable-deterministic-inference`. Please find more details in this blog post: [https://lmsys.org/blog/2025-09-22-sglang-deterministic/](https://lmsys.org/blog/2025-09-22-sglang-deterministic/) # Choices Methods in SGLang Source: https://docs.sglang.io/docs/references/frontend/choices_methods This doc describes the choices methods supported by SGLang. The optional `choices_method` arg determines how options supplied to SGLang's `choices` primitive are selected. Only the `RuntimeEndpoint` backend supports the `choices_method` arg. Other backends, such as `OpenAI`, have bespoke selection implementations due to API limitations. ## Methods ### Token Length Normalized Token length normalized is the default SGLang choices method. It selects the option with the highest average logprob across all of its tokens. Usage example (alternatively, simply omit the `choices_method` arg): ```python Example theme={null} @sgl.function def example(s): s += sgl.user("What is the capital of France?") s += sgl.assistant( sgl.gen( "answer", choices=["London", "Paris", "Berlin"], choices_method=sgl.token_length_normalized, ) ) ``` This can perform poorly if an option contains many tokens, where its later tokens are predicted with high confidence based on its earlier tokens. For instance, even strong models will fail the above example if the specified options are `["Paris", "Antidisestablishmentarianism"]`. ### Greedy Token Selection Greedy token selection simply selects the option with the highest logprob for its initial token. For overlapping options where one option is a subset of a longer option, the logprobs of the shorter option are extended using its average logprob for comparison against the longer option. Usage example: ```python Example theme={null} @sgl.function def example(s): s += sgl.user("What is the capital of France?") s += sgl.assistant( sgl.gen( "answer", choices=["London", "Paris", "Berlin"], choices_method=sgl.greedy_token_selection, ) ) ``` This can perform poorly if an option misleads the model down a bad path based on an attractive initial token. For instance, greedy selection will result in an incorrect response for this example: ```python Example theme={null} @sgl.function def us_president_example(s): s += sgl.user("Name a US president.") s += sgl.assistant( sgl.gen( "answer", choices=["Donald Duck", "Millard Fillmore"], choices_method=sgl.greedy_token_selection, ) ) ``` ### Unconditional Likelihood Normalized Unconditional likelihood normalized selects the option with the highest average token logprob once normalized by the unconditional token logprobs, as described in [this EleutherAI blogpost](https://blog.eleuther.ai/multiple-choice-normalization/). This method incurs an additional LLM call to obtain the unconditional likelihoods. Usage example: ```python Example theme={null} @sgl.function def example(s): s += sgl.user("What is the capital of France?") s += sgl.assistant( sgl.gen( "answer", choices=["London", "Paris", "Berlin"], choices_method=sgl.unconditional_likelihood_normalized, ) ) ``` # Frontend Language Source: https://docs.sglang.io/docs/references/frontend/frontend_index * [Frontend Tutorial](./frontend_tutorial) * [Choices Methods](./choices_methods) # SGLang Frontend Language Source: https://docs.sglang.io/docs/references/frontend/frontend_tutorial SGLang frontend language can be used to define simple and easy prompts in a convenient, structured way. ## Launch A Server Launch the server in your terminal and wait for it to initialize. ```python Example theme={null} from sglang import assistant_begin, assistant_end from sglang import assistant, function, gen, system, user from sglang import image from sglang import RuntimeEndpoint from sglang.lang.api import set_default_backend from sglang.srt.utils import load_image from sglang.test.doc_patch import launch_server_cmd from sglang.utils import print_highlight, terminate_process, wait_for_server server_process, port = launch_server_cmd( "python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --log-level warning" ) wait_for_server(f"http://localhost:{port}", process=server_process) print(f"Server started on http://localhost:{port}") ``` Set the default backend. Note: Besides the local server, you may use also `OpenAI` or other API endpoints. ```python Example theme={null} set_default_backend(RuntimeEndpoint(f"http://localhost:{port}")) ``` ## Basic Usage The most simple way of using SGLang frontend language is a simple question answer dialog between a user and an assistant. ```python Example theme={null} @function def basic_qa(s, question): s += system(f"You are a helpful assistant than can answer questions.") s += user(question) s += assistant(gen("answer", max_tokens=512)) ``` ```python Example theme={null} state = basic_qa("List 3 countries and their capitals.") print_highlight(state["answer"]) ``` ## Multi-turn Dialog SGLang frontend language can also be used to define multi-turn dialogs. ```python Example theme={null} @function def multi_turn_qa(s): s += system(f"You are a helpful assistant than can answer questions.") s += user("Please give me a list of 3 countries and their capitals.") s += assistant(gen("first_answer", max_tokens=512)) s += user("Please give me another list of 3 countries and their capitals.") s += assistant(gen("second_answer", max_tokens=512)) return s state = multi_turn_qa() print_highlight(state["first_answer"]) print_highlight(state["second_answer"]) ``` ## Control flow You may use any Python code within the function to define more complex control flows. ```python Example theme={null} @function def tool_use(s, question): s += assistant( "To answer this question: " + question + ". I need to use a " + gen("tool", choices=["calculator", "search engine"]) + ". " ) if s["tool"] == "calculator": s += assistant("The math expression is: " + gen("expression")) elif s["tool"] == "search engine": s += assistant("The key word to search is: " + gen("word")) state = tool_use("What is 2 * 2?") print_highlight(state["tool"]) print_highlight(state["expression"]) ``` ## Parallelism Use `fork` to launch parallel prompts. Because `sgl.gen` is non-blocking, the for loop below issues two generation calls in parallel. ```python Example theme={null} @function def tip_suggestion(s): s += assistant( "Here are two tips for staying healthy: " "1. Balanced Diet. 2. Regular Exercise.\n\n" ) forks = s.fork(2) for i, f in enumerate(forks): f += assistant( f"Now, expand tip {i+1} into a paragraph:\n" + gen("detailed_tip", max_tokens=256, stop="\n\n") ) s += assistant("Tip 1:" + forks[0]["detailed_tip"] + "\n") s += assistant("Tip 2:" + forks[1]["detailed_tip"] + "\n") s += assistant( "To summarize the above two tips, I can say:\n" + gen("summary", max_tokens=512) ) state = tip_suggestion() print_highlight(state["summary"]) ``` ## Constrained Decoding Use `regex` to specify a regular expression as a decoding constraint. This is only supported for local models. ```python Example theme={null} @function def regular_expression_gen(s): s += user("What is the IP address of the Google DNS servers?") s += assistant( gen( "answer", temperature=0, regex=r"((25[0-5]|2[0-4]\d|[01]?\d\d?).){3}(25[0-5]|2[0-4]\d|[01]?\d\d?)", ) ) state = regular_expression_gen() print_highlight(state["answer"]) ``` Use `regex` to define a `JSON` decoding schema. ```python Example theme={null} character_regex = ( r"""\{\n""" + r""" "name": "[\w\d\s]{1,16}",\n""" + r""" "house": "(Gryffindor|Slytherin|Ravenclaw|Hufflepuff)",\n""" + r""" "blood status": "(Pure-blood|Half-blood|Muggle-born)",\n""" + r""" "occupation": "(student|teacher|auror|ministry of magic|death eater|order of the phoenix)",\n""" + r""" "wand": \{\n""" + r""" "wood": "[\w\d\s]{1,16}",\n""" + r""" "core": "[\w\d\s]{1,16}",\n""" + r""" "length": [0-9]{1,2}\.[0-9]{0,2}\n""" + r""" \},\n""" + r""" "alive": "(Alive|Deceased)",\n""" + r""" "patronus": "[\w\d\s]{1,16}",\n""" + r""" "bogart": "[\w\d\s]{1,16}"\n""" + r"""\}""" ) @function def character_gen(s, name): s += user( f"{name} is a character in Harry Potter. Please fill in the following information about this character." ) s += assistant(gen("json_output", max_tokens=256, regex=character_regex)) state = character_gen("Harry Potter") print_highlight(state["json_output"]) ``` ## Batching Use `run_batch` to run a batch of prompts. ```python Example theme={null} @function def text_qa(s, question): s += user(question) s += assistant(gen("answer", stop="\n")) states = text_qa.run_batch( [ {"question": "What is the capital of the United Kingdom?"}, {"question": "What is the capital of France?"}, {"question": "What is the capital of Japan?"}, ], progress_bar=True, ) for i, state in enumerate(states): print_highlight(f"Answer {i+1}: {states[i]['answer']}") ``` ## Streaming Use `stream` to stream the output to the user. ```python Example theme={null} @function def text_qa(s, question): s += user(question) s += assistant(gen("answer", stop="\n")) state = text_qa.run( question="What is the capital of France?", temperature=0.1, stream=True ) for out in state.text_iter(): print(out, end="", flush=True) ``` ## Complex Prompts You may use `{system|user|assistant}_{begin|end}` to define complex prompts. ```python Example theme={null} @function def chat_example(s): s += system("You are a helpful assistant.") # Same as: s += s.system("You are a helpful assistant.") with s.user(): s += "Question: What is the capital of France?" s += assistant_begin() s += "Answer: " + gen("answer", max_tokens=100, stop="\n") s += assistant_end() state = chat_example() print_highlight(state["answer"]) ``` ```python Example theme={null} terminate_process(server_process) ``` ## Multi-modal Generation You may use SGLang frontend language to define multi-modal prompts. See [here](../../supported-models/multimodal_language_models) for supported models. ```python Example theme={null} server_process, port = launch_server_cmd( "python -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --host 0.0.0.0 --log-level warning" ) wait_for_server(f"http://localhost:{port}", process=server_process) print(f"Server started on http://localhost:{port}") ``` ```python Example theme={null} set_default_backend(RuntimeEndpoint(f"http://localhost:{port}")) ``` Ask a question about an image. ```python Example theme={null} @function def image_qa(s, image_file, question): s += user(image(image_file) + question) s += assistant(gen("answer", max_tokens=256)) image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png" image_bytes, _ = load_image(image_url) state = image_qa(image_bytes, "What is in the image?") print_highlight(state["answer"]) ``` ```python Example theme={null} terminate_process(server_process) ``` # Deploy On Kubernetes Source: https://docs.sglang.io/docs/references/multi_node_deployment/deploy_on_k8s This document is for deploying a RoCE network-based SGLang two-node inference service on a Kubernetes (K8S) cluster. [LeaderWorkerSet (LWS)](https://github.com/kubernetes-sigs/lws) is a Kubernetes API that aims to address common deployment patterns of AI/ML inference workloads. A major use case is for multi-host/multi-node distributed inference. SGLang can also be deployed with LWS on Kubernetes for distributed model serving. Please see this guide for more details on deploying SGLang on Kubernetes using LWS. Here we take the deployment of DeepSeek-R1 as an example. ## Prerequisites 1. At least two Kubernetes nodes, each with two H20 systems and eight GPUs, are required. 2. Make sure your K8S cluster has LWS correctly installed. If it hasn't been set up yet, please follow the [installation instructions](https://lws.sigs.k8s.io/docs/installation/). **Note:** For LWS versions ≤0.5.x, you must use the Downward API to obtain `LWS_WORKER_INDEX`, as native support for this feature was introduced in v0.6.0. ## Basic example For the basic example documentation, refer to [Deploy Distributed Inference Service with SGLang and LWS on GPUs](https://github.com/kubernetes-sigs/lws/tree/main/docs/examples/sglang). However, that document only covers the basic NCCL socket mode. In this section, we’ll make some simple modifications to adapt the setup to the RDMA scenario. ## RDMA RoCE case * Check your env: ```bash Command theme={null} [root@node1 ~]# ibstatus Infiniband device 'mlx5_bond_0' port 1 status: default gid: fe80:0000:0000:0000:0225:9dff:fe64:c79a base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (2X NDR) link_layer: Ethernet Infiniband device 'mlx5_bond_1' port 1 status: default gid: fe80:0000:0000:0000:0225:9dff:fe6e:c3ec base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (2X NDR) link_layer: Ethernet Infiniband device 'mlx5_bond_2' port 1 status: default gid: fe80:0000:0000:0000:0225:9dff:fe73:0dd7 base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (2X NDR) link_layer: Ethernet Infiniband device 'mlx5_bond_3' port 1 status: default gid: fe80:0000:0000:0000:0225:9dff:fe36:f7ff base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (2X NDR) link_layer: Ethernet ``` * Prepare the `lws.yaml` file for deploying on k8s. ```yaml Config theme={null} apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet metadata: name: sglang spec: replicas: 1 leaderWorkerTemplate: size: 2 restartPolicy: RecreateGroupOnPodRestart leaderTemplate: metadata: labels: role: leader spec: dnsPolicy: ClusterFirstWithHostNet hostNetwork: true hostIPC: true containers: - name: sglang-leader image: sglang:latest securityContext: privileged: true env: - name: NCCL_IB_GID_INDEX value: "3" command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --mem-fraction-static - "0.93" - --torch-compile-max-bs - "8" - --max-running-requests - "20" - --tp - "16" # Size of Tensor Parallelism - --dist-init-addr - $(LWS_LEADER_ADDRESS):20000 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --host - "0.0.0.0" - --port - "40000" resources: limits: nvidia.com/gpu: "8" ports: - containerPort: 40000 readinessProbe: tcpSocket: port: 40000 initialDelaySeconds: 15 periodSeconds: 10 volumeMounts: - mountPath: /dev/shm name: dshm - name: model mountPath: /work/models - name: ib mountPath: /dev/infiniband volumes: - name: dshm emptyDir: medium: Memory - name: model hostPath: path: '< your models dir >' # modify it according your models dir - name: ib hostPath: path: /dev/infiniband workerTemplate: spec: dnsPolicy: ClusterFirstWithHostNet hostNetwork: true hostIPC: true containers: - name: sglang-worker image: sglang:latest securityContext: privileged: true env: - name: NCCL_IB_GID_INDEX value: "3" command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --mem-fraction-static - "0.93" - --torch-compile-max-bs - "8" - --max-running-requests - "20" - --tp - "16" # Size of Tensor Parallelism - --dist-init-addr - $(LWS_LEADER_ADDRESS):20000 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code resources: limits: nvidia.com/gpu: "8" volumeMounts: - mountPath: /dev/shm name: dshm - name: model mountPath: /work/models - name: ib mountPath: /dev/infiniband volumes: - name: dshm emptyDir: medium: Memory - name: ib hostPath: path: /dev/infiniband - name: model hostPath: path: /data1/models/deepseek_v3_moe *** apiVersion: v1 kind: Service metadata: name: sglang-leader spec: selector: leaderworkerset.sigs.k8s.io/name: sglang role: leader ports: - protocol: TCP port: 40000 targetPort: 40000 ``` * Then use `kubectl apply -f lws.yaml` you will get this output. ```text Output theme={null} NAME READY STATUS RESTARTS AGE sglang-0 0/1 Running 0 9s sglang-0-1 1/1 Running 0 9s ``` Wait for the sglang leader (`sglang-0`) status to change to 1/1, which indicates it is `Ready`. You can use the command `kubectl logs -f sglang-0` to view the logs of the leader node. Once successful, you should see output like this: ```text Output theme={null} [2025-02-17 05:27:24 TP1] Capture cuda graph end. Time elapsed: 84.89 s [2025-02-17 05:27:24 TP6] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP0] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP7] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP3] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP2] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP4] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP1] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP5] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24] INFO: Started server process [1] [2025-02-17 05:27:24] INFO: Waiting for application startup. [2025-02-17 05:27:24] INFO: Application startup complete. [2025-02-17 05:27:24] INFO: Uvicorn running on http://0.0.0.0:40000 (Press CTRL+C to quit) [2025-02-17 05:27:25] INFO: 127.0.0.1:48908 - "GET /get_model_info HTTP/1.1" 200 OK [2025-02-17 05:27:25 TP0] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, cache hit rate: 0.00%, token usage: 0.00, #running-req: 0, #queue-req: 0 [2025-02-17 05:27:32] INFO: 127.0.0.1:48924 - "POST /generate HTTP/1.1" 200 OK [2025-02-17 05:27:32] The server is fired up and ready to roll! ``` If it doesn’t start up successfully, please follow these steps to check for any remaining issues. Thanks! ### Debug * Set `NCCL_DEBUG=TRACE` to check if it is a NCCL communication problem. This should resolve most NCCL-related issues. ***Notice: If you find that NCCL\_DEBUG=TRACE is not effective in the container environment, but the process is stuck or you encounter hard-to-diagnose issues, try switching to a different container image. Some images may not handle standard error output properly.*** #### RoCE scenario * Please make sure that RDMA devices are available in the cluster environment. * Please make sure that the nodes in the cluster have Mellanox NICs with RoCE. In this example, we use Mellanox ConnectX 5 model NICs, and the proper OFED driver has been installed. If not, please refer to the document [Install OFED Driver](https://docs.nvidia.com/networking/display/mlnxofedv24102180lts/installing+the+driver) to install the driver. * Check your env: ```shell Command theme={null} $ lspci -nn | grep Eth | grep Mellanox 0000:7f:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0000:7f:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0000:c7:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0000:c7:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0001:08:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0001:08:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0001:a2:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0001:a2:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) ``` * Check the OFED driver: ```shell Command theme={null} ofed_info -s OFED-internal-23.07-0.5.0: ``` * Show RDMA link status and check IB devices: ```shell Command theme={null} $ rdma link show 8/1: mlx5_bond_0/1: state ACTIVE physical_state LINK_UP netdev reth0 9/1: mlx5_bond_1/1: state ACTIVE physical_state LINK_UP netdev reth2 10/1: mlx5_bond_2/1: state ACTIVE physical_state LINK_UP netdev reth4 11/1: mlx5_bond_3/1: state ACTIVE physical_state LINK_UP netdev reth6 $ ibdev2netdev 8/1: mlx5_bond_0/1: state ACTIVE physical_state LINK_UP netdev reth0 9/1: mlx5_bond_1/1: state ACTIVE physical_state LINK_UP netdev reth2 10/1: mlx5_bond_2/1: state ACTIVE physical_state LINK_UP netdev reth4 11/1: mlx5_bond_3/1: state ACTIVE physical_state LINK_UP netdev reth6 ``` * Test RoCE network speed on the host: ```shell Command theme={null} yum install qperf # for server： execute qperf # for client qperf -t 60 -cm1 rc_rdma_write_bw ``` * Check RDMA accessible in your container: ```shell Command theme={null} # ibv_devices # ibv_devinfo ``` ## Keys to success * In the YAML configuration above, pay attention to the NCCL environment variable. For older versions of NCCL, you should check the NCCL\_IB\_GID\_INDEX environment setting. * NCCL\_SOCKET\_IFNAME is also crucial, but in a containerized environment, this typically isn’t an issue. * In some cases, it’s necessary to configure GLOO\_SOCKET\_IFNAME correctly. * NCCL\_DEBUG is essential for troubleshooting, but I've found that sometimes it doesn't show error logs within containers. This could be related to the Docker image you're using. You may want to try switching images if needed. * Avoid using Docker images based on Ubuntu 18.04, as they tend to have compatibility issues. ## Remaining issues * In Kubernetes, Docker, or Containerd environments, we use hostNetwork to prevent performance degradation. * We utilize privileged mode, which isn’t secure. Additionally, in containerized environments, full GPU isolation cannot be achieved. ## TODO * Integrated with [k8s-rdma-shared-dev-plugin](https://github.com/Mellanox/k8s-rdma-shared-dev-plugin). # LWS Based PD Deploy Source: https://docs.sglang.io/docs/references/multi_node_deployment/lws_pd/lws_pd_deploy ## 0. Prerequisites 1. k8s >=1.26 2. lws installed on k8s. ## 1. Image Preparation `lmsysorg/sglang:deepep` ## 2. Deployment Manifest Files ***Notice: We will package all deployment files into Helm Chart format in the near future. Interested community members can contact us to contribute*** ### Prefill Prefill manifest file [prefill.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/p.yaml) *Note: The NodeSelector section, model location section, and taint toleration section can be adjusted according to your actual deployment environment* ```yaml Config theme={null} apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet metadata: name: deepseekr10528-prefill-main spec: leaderWorkerTemplate: leaderTemplate: metadata: labels: role: leader spec: containers: - command: - python3 - -m - sglang.launch_server - --port - "30000" - --host - "0.0.0.0" - --model-path - /work/models - --disaggregation-ib-device # should modify according your rdma env - mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3 - --chunked-prefill-size - "524288" - --max-prefill-tokens - "32768" - --page-size - "64" # - --init-expert-location # - /home/aiges/tuned/attachment_ep_statistics/prefill_in1024.json - --ep-dispatch-algorithm - dynamic - --eplb-algorithm - deepseek # - --deepep-config # - /home/aiges/tuned/tuned_8sms.json - --enable-dp-lm-head - --attn-dp-size - "16" - --disable-radix-cache - --moe-a2a-backend - deepep - --disaggregation-mode - prefill - --mem-fraction-static - "0.7" - --context-length - "32768" - --tp - "16" - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" - --max-running-requests - "1024" env: # - name: NVSHMEM_HCA_PE_MAPPING # value: "mlx5_bond_0:1:2,mlx5_bond_1:1:2,mlx5_bond_2:1:2,mlx5_bond_3:1:2" # - name: NVSHMEM_HCA_LIST # value: "mlx5_bond_0:1,mlx5_bond_1:1,mlx5_bond_2:1,mlx5_bond_3:1" - name: NVSHMEM_IB_GID_INDEX value: "3" - name: NVSHMEM_ENABLE_NIC_PE_MAPPING value: "1" - name: SGLANG_SET_CPU_AFFINITY value: "true" - name: SGLANG_ENABLE_JIT_DEEPGEMM value: "1" - name: NCCL_IB_QPS_PER_CONNECTION value: "8" - name: NCCL_IB_SPLIT_DATA_ON_QPS value: "1" - name: NCCL_NET_PLUGIN value: none - name: NCCL_IB_TC value: "136" - name: NCCL_MIN_NCHANNELS value: "4" - name: MC_TE_METRIC value: "false" - name: NCCL_IB_SL value: "5" - name: NCCL_IB_HCA value: ^=mlx5_0,mlx5_5,mlx5_6 - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] image: lmsysorg/sglang:deepep name: sglang-leader ports: - containerPort: 30000 protocol: TCP readinessProbe: periodSeconds: 30 tcpSocket: port: 30000 resources: limits: nvidia.com/gpu: "8" securityContext: capabilities: add: - IPC_LOCK privileged: true volumeMounts: - mountPath: /dev/shm name: dshm - mountPath: /work/models name: model - mountPath: /dev/infiniband name: ib - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs name: cf - mountPath: /root/.cache name: sgl-cache dnsPolicy: ClusterFirstWithHostNet hostIPC: true hostNetwork: true nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists - key: node-role operator: Exists volumes: - emptyDir: medium: Memory name: dshm - hostPath: # modify according to you deployment env path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528 name: model - hostPath: path: /dev/infiniband name: ib - hostPath: # modify according to you deployment env path: /data1/maas_hosted_models/models/fused_moe_triton/configs name: cf - hostPath: # modify according to you deployment env path: /data1/sgl_cache type: DirectoryOrCreate name: sgl-cache restartPolicy: RecreateGroupOnPodRestart size: 2 workerTemplate: metadata: {} spec: containers: - command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --disaggregation-ib-device - mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3 - --chunked-prefill-size - "524288" - --max-prefill-tokens - "32768" - --page-size - "64" #- --init-expert-location #- /home/aiges/tuned/attachment_ep_statistics/prefill_in1024.json - --ep-dispatch-algorithm - dynamic - --eplb-algorithm - deepseek # - --deepep-config # - /home/aiges/tuned/tuned_8sms.json - --enable-dp-lm-head - --attn-dp-size - "16" - --disable-radix-cache - --moe-a2a-backend - deepep - --disaggregation-mode - prefill - --mem-fraction-static - "0.7" - --context-length - "32768" - --tp - "16" - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" - --max-running-requests - "1024" env: - name: SGLANG_SET_CPU_AFFINITY value: "true" - name: SGLANG_HACK_DEEPEP_NUM_SMS value: "8" - name: SGLANG_HACK_DEEPEP_NEW_MODE value: "0" # - name: NVSHMEM_HCA_PE_MAPPING # value: "mlx5_bond_0:1:2,mlx5_bond_1:1:2,mlx5_bond_2:1:2,mlx5_bond_3:1:2" # - name: NVSHMEM_HCA_LIST # value: "mlx5_bond_0:1,mlx5_bond_1:1,mlx5_bond_2:1,mlx5_bond_3:1" - name: NCCL_IB_HCA value: ^=mlx5_0,mlx5_5,mlx5_6 - name: NVSHMEM_IB_TRAFFIC_CLASS value: "16" - name: NVSHMEM_IB_GID_INDEX value: "3" - name: NVSHMEM_ENABLE_NIC_PE_MAPPING value: "1" - name: CUDA_LAUNCH_BLOCKING value: "0" - name: SGLANG_MOONCAKE_TRANS_THREAD value: "8" - name: SGLANG_ENABLE_JIT_DEEPGEMM value: "1" - name: SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD value: "0" - name: NCCL_IB_QPS_PER_CONNECTION value: "8" - name: NCCL_IB_SPLIT_DATA_ON_QPS value: "1" - name: NCCL_NET_PLUGIN value: none - name: NCCL_IB_TC value: "136" - name: NCCL_MIN_NCHANNELS value: "4" - name: MC_TE_METRIC value: "true" - name: NCCL_IB_SL value: "5" - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] image: lmsysorg/sglang:deepep name: sglang-worker ports: - containerPort: 30001 protocol: TCP resources: limits: nvidia.com/gpu: "8" securityContext: capabilities: add: - IPC_LOCK privileged: true volumeMounts: - mountPath: /root/.cache name: sgl-cache - mountPath: /dev/shm name: dshm - mountPath: /work/models name: model - mountPath: /dev/infiniband name: ib - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs name: cf dnsPolicy: ClusterFirstWithHostNet hostIPC: true hostNetwork: true nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists - key: node-role operator: Exists volumes: - emptyDir: medium: Memory name: dshm - hostPath: path: /dev/infiniband name: ib - hostPath: path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528 name: model - hostPath: path: /data1/maas_hosted_models/models/fused_moe_triton/configs name: cf - hostPath: path: /data1/sgl_cache type: DirectoryOrCreate name: sgl-cache ``` ### Decode Decode node deployment manifest file [decode.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/d.yaml) *Note: The NodeSelector section, model location section, and taint toleration section can be adjusted according to your actual deployment environment* ```yaml Config theme={null} apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet metadata: name: deepseekr10528-decode-main spec: leaderWorkerTemplate: leaderTemplate: metadata: labels: role: leader spec: containers: - command: - python3 - -m - sglang.launch_server - --port - "30000" - --host - "0.0.0.0" - --model-path - /work/models - --chunked-prefill-size - "262144" - --page-size - "64" - --enable-dp-lm-head - --attn-dp-size - "16" - --moe-a2a-backend - deepep - --disaggregation-mode - decode - --mem-fraction-static - "0.849" - --context-length - "32768" - --disaggregation-ib-device - "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3" - --cuda-graph-max-bs-decode - "64" - --max-running-requests - "2048" - --tp-size - "16" # Size of Tensor Parallelism - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" env: - name: CUDA_LAUNCH_BLOCKING value: "0" - name: NVSHMEM_IB_GID_INDEX value: "3" - name: NVSHMEM_ENABLE_NIC_PE_MAPPING value: "1" - name: NCCL_IB_QPS_PER_CONNECTION value: "8" - name: NCCL_IB_SPLIT_DATA_ON_QPS value: "1" - name: NCCL_NET_PLUGIN value: "none" - name: NCCL_IB_TC value: "136" - name: NCCL_MIN_NCHANNELS value: "4" - name: NCCL_IB_SL value: "5" - name: MC_TE_METRIC value: "true" - name: SGLANG_MOONCAKE_TRANS_THREAD value: "16" - name: SGLANG_ENABLE_JIT_DEEPGEMM value: "1" - name: NCCL_IB_HCA value: ^=mlx5_0,mlx5_5,mlx5_6 - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] image: lmsysorg/sglang:deepep name: sglang-leader ports: - containerPort: 30000 protocol: TCP readinessProbe: periodSeconds: 30 tcpSocket: port: 30000 resources: limits: nvidia.com/gpu: "8" securityContext: capabilities: add: - IPC_LOCK privileged: true volumeMounts: - mountPath: /root/.cache name: sgl-cache - mountPath: /dev/shm name: dshm - mountPath: /work/models name: model - mountPath: /dev/infiniband name: ib - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs name: cf dnsPolicy: ClusterFirstWithHostNet hostIPC: true hostNetwork: true nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists - key: node-role operator: Exists volumes: - hostPath: path: /data1/sgl_cache1 type: DirectoryOrCreate name: sgl-cache - emptyDir: medium: Memory name: dshm - hostPath: path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528 name: model - hostPath: path: /dev/infiniband name: ib - hostPath: path: /data1/maas_hosted_models/models/fused_moe_triton/configs name: cf restartPolicy: RecreateGroupOnPodRestart size: 2 workerTemplate: metadata: {} spec: containers: - command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --chunked-prefill-size - "262144" - --page-size - "64" - --enable-dp-lm-head #- --enable-two-batch-overlap - --attn-dp-size - "16" - --moe-a2a-backend - deepep - --disaggregation-mode - decode - --mem-fraction-static - "0.849" - --context-length - "32768" - --disaggregation-ib-device # should modify according your rdma env - "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3" - --cuda-graph-max-bs-decode - "64" - --max-running-requests - "2048" - --tp-size - "16" # Size of Tensor Parallelism - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" env: - name: SGLANG_HACK_DEEPEP_NUM_SMS value: "24" - name: SGLANG_HACK_DEEPEP_NEW_MODE value: "0" - name: NVSHMEM_IB_TRAFFIC_CLASS value: "16" - name: NVSHMEM_IB_GID_INDEX value: "3" - name: NVSHMEM_ENABLE_NIC_PE_MAPPING value: "1" - name: NCCL_IB_QPS_PER_CONNECTION value: "8" - name: NCCL_IB_SPLIT_DATA_ON_QPS value: "1" - name: NCCL_NET_PLUGIN value: "none" - name: NCCL_IB_TC value: "136" - name: NCCL_MIN_NCHANNELS value: "4" - name: MC_TE_METRIC value: "true" - name: NCCL_IB_SL value: "5" - name: SGLANG_MOONCAKE_TRANS_THREAD value: "16" - name: SGLANG_ENABLE_JIT_DEEPGEMM value: "1" - name: NCCL_IB_HCA value: ^=mlx5_0,mlx5_5,mlx5_6 - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] image: lmsysorg/sglang:deepep name: sglang-worker ports: - containerPort: 30001 resources: limits: nvidia.com/gpu: "8" securityContext: capabilities: add: - IPC_LOCK privileged: true volumeMounts: - mountPath: /root/.cache name: sgl-cache - mountPath: /dev/shm name: dshm - mountPath: /work/models name: model - mountPath: /dev/infiniband name: ib - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs name: cf dnsPolicy: ClusterFirstWithHostNet hostIPC: true hostNetwork: true nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists - key: node-role operator: Exists volumes: - hostPath: path: /data1/sgl_cache1 type: DirectoryOrCreate name: sgl-cache - emptyDir: medium: Memory name: dshm - hostPath: path: /dev/infiniband name: ib - hostPath: # modify according to you deployment env path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528 name: model - hostPath: # modify according to you deployment env path: /data1/maas_hosted_models/models/fused_moe_triton/configs name: cf networkConfig: subdomainPolicy: Shared replicas: 1 rolloutStrategy: rollingUpdateConfiguration: maxSurge: 0 maxUnavailable: 1 type: RollingUpdate startupPolicy: LeaderCreated ``` Execute separately: ```bash Command theme={null} kubectl apply -f p.yaml kubectl apply -f d.yaml ``` At this point, we have completed the deployment of the 1P1D SGLang engine part. To allow our users to directly experience the model API, we still need a load balancer to handle sequential calls between prefill and decode. Different companies implement LBs differently, and the community will also officially release a new LB component written in Rust in the near future. Currently, we use a static K8S service + minilb approach to implement model API calls. ### Creating Service for Prefill and Decode #### Create prefill k8s service [p-svc.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/p-svc.yaml) ```yaml Config theme={null} apiVersion: v1 kind: Service metadata: name: deepseekr10528-prefill-main spec: selector: leaderworkerset.sigs.k8s.io/name: deepseekr10528-prefill-main role: leader ports: - protocol: TCP port: 30000 targetPort: 30000 ``` Execute `kubectl apply -f p-svc.yaml` #### Create decode k8s service [d-svc.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/d-svc.yaml) ```yaml Config theme={null} apiVersion: v1 kind: Service metadata: name: deepseekr10528-decode-main spec: selector: leaderworkerset.sigs.k8s.io/name: deepseekr10528-decode-main role: leader ports: - protocol: TCP port: 30000 targetPort: 30000 ``` Execute `kubectl apply -f d-svc.yaml` #### Deploy minilb and lb service [lb.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/lb.yaml) ```yaml Config theme={null} apiVersion: apps/v1 kind: Deployment metadata: name: deepseekr10528-lb-main labels: app: deepseekr10528-lb spec: replicas: 1 selector: matchLabels: app: deepseekr10528-lb template: metadata: labels: app: deepseekr10528-lb spec: nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists - key: node-role operator: Exists containers: - name: sgl-minilb image: lmsysorg/sglang:deepep command: - python - -m - sglang_router.launch_router - --pd-disaggregation - --prefill - http://deepseekr10528-prefill-main:30000 - --decode - http://deepseekr10528-decode-main:30000 - --host - 0.0.0.0 - --port - "8000" ports: - containerPort: 8000 *** apiVersion: v1 kind: Service metadata: name: deepseekr10528-lb-service spec: type: NodePort selector: app: deepseekr10528-lb ports: - protocol: TCP port: 8000 # Service Port（In-Cluster） targetPort: 8000 # Exposed Container nodePort: 30800 ``` Execute `kubectl apply -f lb.yaml` After waiting for all model deployments to succeed, you will get the following output: ```bash Command theme={null} [root@ecs-001]# kubectl get po deepseekr10528-decode-main-0 1/1 Running 0 74m deepseekr10528-decode-main-0-1 1/1 Running 0 74m deepseekr10528-lb-main-9c5dbfc57-6lcbd 1/1 Running 0 22m deepseekr10528-prefill-main-0 1/1 Running 0 74m deepseekr10528-prefill-main-0-1 1/1 Running 0 74m [root@ecs-cbm-x1-pd-cpu-001 main_doc]# kubectl get svc |grep dee deepseekr10528-decode-main ClusterIP None 97m deepseekr10528-lb-service NodePort 172.16.242.169 8000:30800/TCP 22m deepseekr10528-prefill-main ClusterIP None 97m ``` At this point, select a nodePort:30800 to access: ```bash Command theme={null} [root@ecs-001]# curl -X POST "http://{nodePort}:30800/v1/chat/completions" \ > -H "Content-Type: application/json" \ > -H "Authorization: Bearer None" \ > -d '{ > "rid":"ccccdd", > "model": "r1", > "messages": [ > {"role": "system", "content": "0: You are a helpful AI assistant"}, > {"role": "user", "content": "你是谁？."} > ], > "max_tokens":221 > }' {"id":"ccccdd","object":"chat.completion","created":1750252498,"model":"qwen2","choices":[{"index":0,"message":{"role":"assistant","content":"\n嗯，用户问了一个很基础的自我介绍问题"你是谁？"。这可能是第一次互动时的常规开场白，也可能是想确认我的身份和功能范围。\n\n用户没有提供任何背景信息，语气简洁中性。这种场景下新用户的可能性较高，需要给出清晰友好的自我介绍，同时突出实用价值来降低陌生感。\n\n考虑到中文用户，应该用简体中文回复。重点要说明三点：身份归属（深度求索）、功能定位（AI助手）、服务范围（学习/工作/生活）。结尾用开放性问题引导对话很关键——既能了解需求，又能避免让用户面对空白输入框时不知所措。\n\n用波浪线结尾可以软化语气，那个笑脸表情😊刚好能中和AI的机械感。不过要控制表情符号数量，避免显得轻浮。\n\n你好呀！我是你的AI助手，由深度求索公司（DeepSeek）开发的语言模型，名字叫 **DeepSeek-R1**。你可以把我当成一个知识丰富、随叫随到的小帮手～😊\n\n我的任务就是陪你聊天、解答问题、","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"length","matched_stop":null}],"usage":{"prompt_tokens":14,"total_tokens":235,"completion_tokens":221,"prompt_tokens_details":null}} ``` ## FAQ 1. The current deployment startup parameters may not be fully compatible with all RDMA scenarios. Different RDMA NCCL-related environment configurations may be needed in different network environments. 2. Some preset, optimized configurations for EPLB are not used here. You can adjust them according to [6017](https://github.com/sgl-project/sglang/issues/6017) as needed. # Multi-Node Deployment Source: https://docs.sglang.io/docs/references/multi_node_deployment/multi_node ## Llama 3.1 405B **Run 405B (fp16) on Two Nodes** ```bash Command theme={null} # replace 172.16.4.52:20000 with your own node ip address and port of the first node python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-405B-Instruct \ --tp 16 \ --dist-init-addr 172.16.4.52:20000 \ --nnodes 2 \ --node-rank 0 python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-405B-Instruct \ --tp 16 \ --dist-init-addr 172.16.4.52:20000 \ --nnodes 2 \ --node-rank 1 ``` Note that LLama 405B (fp8) can also be launched on a single node. ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --tp 8 ``` ## DeepSeek V3/R1 Please refer to [DeepSeek documents for reference](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-5-multi-node-deployment). ## Multi-Node Inference on SLURM This example showcases how to serve SGLang server across multiple nodes by SLURM. Submit the following job to the SLURM cluster. ```bash Command theme={null} #!/bin/bash -l #SBATCH -o SLURM_Logs/%x_%j_master.out #SBATCH -e SLURM_Logs/%x_%j_master.err #SBATCH -D ./ #SBATCH -J Llama-405B-Online-Inference-TP16-SGL #SBATCH --nodes=2 #SBATCH --ntasks=2 #SBATCH --ntasks-per-node=1 # Ensure 1 task per node #SBATCH --cpus-per-task=18 #SBATCH --mem=224GB #SBATCH --partition="lmsys.org" #SBATCH --gres=gpu:8 #SBATCH --time=12:00:00 echo "[INFO] Activating environment on node $SLURM_PROCID" if ! source ENV_FOLDER/bin/activate; then echo "[ERROR] Failed to activate environment" >&2 exit 1 fi # Define parameters model=MODEL_PATH tp_size=16 echo "[INFO] Running inference" echo "[INFO] Model: $model" echo "[INFO] TP Size: $tp_size" # Set NCCL initialization address using the hostname of the head node HEAD_NODE=$(scontrol show hostname "$SLURM_NODELIST" | head -n 1) NCCL_INIT_ADDR="${HEAD_NODE}:8000" echo "[INFO] NCCL_INIT_ADDR: $NCCL_INIT_ADDR" # Launch the model server on each node using SLURM srun --ntasks=2 --nodes=2 --output="SLURM_Logs/%x_%j_node$SLURM_NODEID.out" \ --error="SLURM_Logs/%x_%j_node$SLURM_NODEID.err" \ python3 -m sglang.launch_server \ --model-path "$model" \ --grammar-backend "xgrammar" \ --tp "$tp_size" \ --dist-init-addr "$NCCL_INIT_ADDR" \ --nnodes 2 \ --node-rank "$SLURM_NODEID" & # Wait for the NCCL server to be ready on port 30000 while ! nc -z "$HEAD_NODE" 30000; do sleep 1 echo "[INFO] Waiting for $HEAD_NODE:30000 to accept connections" done echo "[INFO] $HEAD_NODE:30000 is ready to accept connections" # Keep the script running until the SLURM job times out wait ``` Then, you can test the server by sending requests following other [documents](../../basic_usage/openai_api_completions). Thanks for [aflah02](https://github.com/aflah02) for providing the example, based on his [blog post](https://aflah02.substack.com/p/multi-node-llm-inference-with-sglang). # Multi-Node Deployment Source: https://docs.sglang.io/docs/references/multi_node_deployment/multi_node_index * [Multi Node](./multi_node) * [Deploy On K8S](./deploy_on_k8s) * [Lws Pd Deploy](./lws_pd/lws_pd_deploy) * [Deepseekv32 Pd](./rbg_pd/deepseekv32_pd) * [Deploying DeepSeek with PD Disaggregation on 96 H100 GPUs](https://lmsys.org/blog/2025-05-05-large-scale-ep/) * [Deploying Kimi K2 with PD Disaggregation on 128 H200 GPUs](https://lmsys.org/blog/2025-07-20-k2-large-scale-ep/) # DeepSeekV32-Exp RBG Based PD Deploy Source: https://docs.sglang.io/docs/references/multi_node_deployment/rbg_pd/deepseekv32_pd ## 0. Prerequisites 1. k8s >=1.26 2. lws installed on k8s. 3. rbg installed on k8s. For RBG installation, please refer to: [https://github.com/sgl-project/rbg](https://github.com/sgl-project/rbg) ## 1. Image Preparation `lmsysorg/sglang:latest` ### 2. All In One manifest file *Note: The NodeSelector section, model location section, and taint toleration section can be adjusted according to your actual deployment environment* rbg-dsv32.yml ```yaml Config theme={null} apiVersion: workloads.x-k8s.io/v1alpha1 kind: RoleBasedGroup metadata: name: deepseek-rbg-32exp namespace: default spec: roles: - name: prefill replicas: 1 workload: apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet restartPolicy: None leaderWorkerSet: size: 1 patchLeaderTemplate: metadata: labels: role: leader pd_role: prefill spec: containers: - command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --port - "30000" - --trust-remote - --host - 0.0.0.0 - --disaggregation-ib-device - mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 - --disable-radix-cache - --chunked-prefill-size - "131072" - --page-size - "64" # - --enable-eplb - --ep-dispatch-algorithm - dynamic - --eplb-algorithm - deepseek - --enable-dp-lm-head - --attn-dp-size - "8" - --moe-a2a-backend - deepep - --deepep-mode - normal - --disaggregation-mode - prefill - --mem-fraction-static - "0.8" - --max-prefill-tokens - "32768" - --context-length - "32768" - --tp - "8" - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" - --max-running-requests - "1024" env: - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] livenessProbe: failureThreshold: 3000 httpGet: path: /health port: 30000 initialDelaySeconds: 300 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 10 readinessProbe: failureThreshold: 20 httpGet: path: /health port: 30000 periodSeconds: 30 successThreshold: 1 timeoutSeconds: 10 name: sglang ports: - containerPort: 30000 name: sglang-http protocol: TCP patchWorkerTemplate: {} template: metadata: labels: inference-framework: sglang inference-stack.io/monitoring: "enabled" spec: containers: - name: sglang image: lmsysorg/sglang:latest env: - name: SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK value: "1" - name: CUDA_LAUNCH_BLOCKING value: "0" - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT value: "1000000000" - name: NVSHMEM_IB_TRAFFIC_CLASS value: "16" - name: NVSHMEM_DISABLE_P2P value: "0" - name: ENABLE_METRICS value: "true" - name: NVSHMEM_IB_GID_INDEX value: "3" - name: NVSHMEM_IB_SL value: "5" - name: SGLANG_SET_CPU_AFFINITY value: "true" - name: SGL_ENABLE_JIT_DEEPGEMM value: "1" - name: NCCL_IB_QPS_PER_CONNECTION value: "8" - name: NCCL_IB_SPLIT_DATA_ON_QPS value: "1" - name: NCCL_NET_PLUGIN value: "none" - name: NCCL_IB_TC value: "136" - name: NCCL_IB_SL value: "5" - name: NCCL_IB_TIMEOUT value: "22" - name: NCCL_IB_GID_INDEX value: "3" - name: NCCL_MIN_NCHANNELS value: "4" - name: NCCL_SOCKET_IFNAME value: bond0 - name: GLOO_SOCKET_IFNAME value: bond0 - name: NCCL_IB_HCA value: ^=mlx5_0,mlx5_5,mlx5_6 - name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME value: "bond0" - name: MC_TE_METRIC value: "false" resources: limits: nvidia.com/gpu: "8" securityContext: capabilities: add: - IPC_LOCK privileged: true volumeMounts: - mountPath: /root/.cache name: sgl-cache - mountPath: /dev/shm name: dshm - mountPath: /work/models name: model - mountPath: /dev/infiniband name: ib - mountPath: /sgl-workspace/sglang name: src dnsPolicy: ClusterFirstWithHostNet hostIPC: true hostNetwork: true nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists volumes: - hostPath: path: /var/run/sys-topology name: topo - hostPath: path: /data1/sgl_cache4 type: DirectoryOrCreate name: sgl-cache - emptyDir: medium: Memory name: dshm - hostPath: path: /data/DeepSeek-V3.2-Exp name: model - hostPath: path: /dev/infiniband name: ib - hostPath: path: /data/src/sglang type: DirectoryOrCreate name: src - name: decode replicas: 1 workload: apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet leaderWorkerSet: size: 1 patchLeaderTemplate: metadata: labels: role: leader pd_role: decode spec: containers: - command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --port - "30000" - --trust-remote - --host - 0.0.0.0 - --disaggregation-ib-device - mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 - --chunked-prefill-size - "131072" - --eplb-rebalance-layers-per-chunk - "29" - --page-size - "64" - --enable-dp-lm-head - --attn-dp-size - "8" - --moe-a2a-backend - deepep - --deepep-mode - low_latency - --disaggregation-mode - decode - --mem-fraction-static - "0.8" - --context-length - "32768" - --max-running-requests - "2048" - --tp-size - "8" # Size of Tensor Parallelism - --cuda-graph-max-bs-decode - "16" - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" env: - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] livenessProbe: failureThreshold: 30000 httpGet: path: /health port: 30000 initialDelaySeconds: 300 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 10 name: sglang readinessProbe: failureThreshold: 20 httpGet: path: /health port: 30000 periodSeconds: 30 successThreshold: 1 timeoutSeconds: 10 patchWorkerTemplate: spec: containers: - command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --crash-dump-folder - /log - --chunked-prefill-size - "262144" - --eplb-rebalance-layers-per-chunk - "29" - --page-size - "64" - --enable-dp-lm-head - --attn-dp-size - "32" - --moe-a2a-backend - "deepep" - --deepep-mode - low_latency - --disaggregation-mode - decode - --mem-fraction-static - "0.849" - --context-length - "32768" - --disaggregation-ib-device - mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 - --max-running-requests - "4096" - --cuda-graph-max-bs-decode - "16" - --tp-size - "8" # Size of Tensor Parallelism - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" env: - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] name: sglang template: metadata: labels: inference-framework: sglang-unuse inference-stack.io/monitoring: "enabled" spec: containers: - image: lmsysorg/sglang:latest name: sglang resources: limits: nvidia.com/gpu: "8" securityContext: capabilities: add: - IPC_LOCK privileged: true volumeMounts: - mountPath: /root/.cache name: sgl-cache - mountPath: /dev/shm name: dshm - mountPath: /work/models name: model - mountPath: /dev/infiniband name: ib - mountPath: /sgl-workspace/sglang name: src env: - name: SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK value: "1" - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT value: "100000000" - name: NVSHMEM_DISABLE_P2P value: "0" - name: NVSHMEM_IB_TRAFFIC_CLASS value: "16" - name: NVSHMEM_IB_SL value: "5" - name: ENABLE_METRICS value: "true" - name: CUDA_LAUNCH_BLOCKING value: "0" - name: NVSHMEM_IB_GID_INDEX value: "3" - name: NCCL_IB_QPS_PER_CONNECTION value: "8" - name: NCCL_IB_SPLIT_DATA_ON_QPS value: "1" - name: NCCL_NET_PLUGIN value: "none" - name: NCCL_IB_TC value: "136" - name: NCCL_IB_SL value: "5" - name: NCCL_IB_TIMEOUT value: "22" - name: NCCL_IB_GID_INDEX value: "3" - name: NCCL_MIN_NCHANNELS value: "4" - name: NCCL_SOCKET_IFNAME value: bond0 - name: GLOO_SOCKET_IFNAME value: bond0 - name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME value: "bond0" - name: NCCL_IB_HCA value: ^=mlx5_0,mlx5_5,mlx5_6 - name: MC_TE_METRIC value: "false" - name: SGL_ENABLE_JIT_DEEPGEMM value: "1" dnsPolicy: ClusterFirstWithHostNet hostIPC: true hostNetwork: true nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists volumes: - hostPath: path: /var/run/sys-topology name: topo - hostPath: path: /data1/sgl_cache4 type: DirectoryOrCreate name: sgl-cache - hostPath: path: /data/src/sglang type: DirectoryOrCreate name: src - emptyDir: medium: Memory name: dshm - hostPath: path: /data/DeepSeek-V3.2-Exp name: model - hostPath: path: /dev/infiniband name: ib - name: router replicas: 1 dependencies: [ "decode", "prefill" ] template: spec: containers: - name: scheduler image: lmsysorg/sglang:latest command: - sh - -c - > python3 -m sglang_router.launch_router --host 0.0.0.0 --port 8080 --pd-disaggregation --policy random --service-discovery --service-discovery-namespace ${NAMESPACE} --service-discovery-port 30000 --prefill-selector pd_role=prefill --decode-selector pd_role=decode --max-payload-size 2147483648 --worker-startup-timeout-secs 1200 env: - name: NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace *** apiVersion: v1 kind: Service metadata: labels: app: deepseek-rbg-32exp name: deepseek-rbg-32exp namespace: default spec: ports: - name: http port: 8080 protocol: TCP targetPort: 8080 nodePort: 30080 selector: rolebasedgroup.workloads.x-k8s.io/name: deepseek-rbg-32exp rolebasedgroup.workloads.x-k8s.io/role: router type: NodePort ``` ```bash Command theme={null} [root@ecs-001]# kubectl get po -n default deepseek-rbg-32exp-decode-main-0 1/1 Running 0 74m deepseek-rbg-32exp-decode-0-1 1/1 Running 0 74m deepseek-rbg-32exp-router-9c5dbfc57 1/1 Running 0 22m deepseek-rbg-32exp-prefill-0 1/1 Running 0 74m [root@ecs-cbm-x1-pd-cpu-001 main_doc]# kubectl get svc |grep dee deepseek-rbg-32exp-decode ClusterIP None 97m deepseek-rbg-32exp-router-service NodePort 172.16.242.169 8000:30800/TCP 22m deepseek-rbg-32exp-prefill ClusterIP None 97m ``` At this point, select a nodePort:30800 to access: ```bash Command theme={null} [root@ecs-001]# curl -X POST "http://{nodePort}:30800/v1/chat/completions" \ > -H "Content-Type: application/json" \ > -H "Authorization: Bearer None" \ > -d '{ > "rid":"ccccdd", > "model": "dsv32", > "messages": [ > {"role": "system", "content": "0: You are a helpful AI assistant"}, > {"role": "user", "content": "你是谁？."} > ], > "max_tokens":221 > }' {"id":"ccccdd","object":"chat.completion","created":1750252498,"model":"qwen2","choices":[{"index":0,"message":{"role":"assistant","content":"<think>\n嗯，用户问了一个很基础的自我介绍问题"你是谁？"。这可能是第一次互动时的常规开场白，也可能是想确认我的身份和功能范围。\n\n用户没有提供任何背景信息，语气简洁中性。这种场景下新用户的可能性较高，需要给出清晰友好的自我介绍，同时突出实用价值来降低陌生感。\n\n考虑到中文用户，应该用简体中文回复。重点要说明三点：身份归属（深度求索）、功能定位（AI助手）、服务范围（学习/工作/生活）。结尾用开放性问题引导对话很关键——既能了解需求，又能避免让用户面对空白输入框时不知所措。\n\n用波浪线结尾可以软化语气，那个笑脸表情😊刚好能中和AI的机械感。不过要控制表情符号数量，避免显得轻浮。\n</think>\n你好呀！我是你的AI助手，由深度求索公司（DeepSeek）开发的语言模型，名字叫 **DeepSeek-V32**。你可以把我当成一个知识丰富、随叫随到的小帮手～😊\n\n我的任务就是陪你聊天、解答问题、","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"length","matched_stop":null}],"usage":{"prompt_tokens":14,"total_tokens":235,"completion_tokens":221,"prompt_tokens_details":null}} ``` ## FAQ 1. The current deployment startup parameters may not be fully compatible with all RDMA scenarios. Different RDMA NCCL-related environment configurations may be needed in different network environments. 2. Please ensure that the sglang code in the image has incorporated the changes from [PR #10912](https://github.com/sgl-project/sglang/pull/10912). # Nightly precision regression Source: https://docs.sglang.io/docs/references/nightly_precision_regression # Nightly Precision Regression Testing ## Overview The nightly precision regression framework detects silent numerical regressions in the SGLang serving engine by comparing **per-layer hidden states** between consecutive runs. It runs as a nightly CI job on 8×H200 GPUs and can also be invoked locally for development and debugging. The framework operates on a **rolling-baseline** model: 1. **Baseline creation or comparison:** Launch the server, send a fixed prompt, dump per-layer hidden states to disk. If a previous baseline exists, compare the new tensors against it using the SGLang tensor comparator. If the comparison passes, the new tensors become the updated baseline. 2. On the first run (or when the capture shape changes), the dumped tensors are saved as a new baseline with no comparison. Baselines are stored locally on disk and synced to a **HuggingFace dataset** so they survive across CI runners and can be shared across machines. The HF dataset store is **required** — the test errors if `SGLANG_PRECISION_HF_REPO` is unset. *** ## How It Works ### Step-by-step flow ``` ┌──────────────────────────────────────────────────────────────┐ │ 1. Resolve model config (layer count, capture layers) │ │ ↓ │ │ 2. Compute capture_signature (schema, layers, TP, filter) │ │ ↓ │ │ 3. Fetch baseline from HF dataset (signature-matched) │ │ ↓ │ │ 4. Launch SGLang server with DUMPER enabled │ │ ↓ │ │ 5. POST /dumper/configure (set layer filter + cleanup) │ │ ↓ │ │ 6. POST /v1/chat/completions (fixed prompt, 2 tokens, │ │ ignore_eos=true to force decode path) │ │ ↓ │ │ 7. Kill server; assert decode tensors were captured │ │ ↓ │ │ 8. Baseline exists (with matching signature)? │ │ ├── YES → Run comparator → pass/fail │ │ │ ├── PASS → update baseline, push to HF │ │ │ └── FAIL → push diagnostics to HF │ │ └── NO → copy today's tensors as initial baseline │ │ → push to HF as "baseline_established" │ │ ↓ │ │ 9. Report summary (stdout + GitHub Step Summary) │ └──────────────────────────────────────────────────────────────┘ ``` ### Key components | Component | File | Purpose | | - | - | - | | Test entry point | `test/registered/debug_utils/test_nightly_precision_regression.py` | Orchestrates server launch, dump, compare, and reporting | | HF baseline store | `python/sglang/test/precision_baseline_store.py` | Push / fetch / prune baselines on a HuggingFace dataset | | Tensor comparator | `python/sglang/srt/debug_utils/comparator/` | Compares two directories of `.pt` tensors, emits JSONL report | | Dumper infrastructure | `python/sglang/srt/debug_utils/dumper.py` | Captures per-layer hidden states at runtime | | CI workflow | `.github/workflows/nightly-test-nvidia.yml` | Schedules the nightly job on 8×H200 | *** ## What Gets Dumped and Compared ### Strided layer capture Not every layer is dumped — the framework uses a **strided capture** to reduce I/O and storage overhead. By default, it captures: * Layer 0 (always) * The last layer (always) * Every 8th layer in between (configurable via `LAYER_CAPTURE_STRIDE`) The layer count is resolved automatically from the model's HuggingFace `config.json` (`num_hidden_layers` or `num_layers`). If resolution fails, all layers are captured as a safe fallback. The dumper filter is built dynamically as a regex matching only the selected layer indices, e.g.: ``` match(r'^non_intrusive__model\.layers\.(0|7|15|23)\.(inputs\.1|self_attn\.inputs\.hidden_states)$', name) ``` ### Decode-path verification The test generates **2 tokens** with `ignore_eos=True` to ensure the model's decode path is exercised. After the dump, `_assert_decode_captured()` verifies that tensors from the decode step were actually captured (not just prefill). If only prefill tensors are found, the test fails immediately — this catches misconfigurations where `--max-total-tokens` is too low for the decode loop to run. ### Comparator The comparator computes **relative differences** (`rel_diff`) for each tensor and checks them against a configurable threshold (default `1e-3`). The comparison is restricted to the post-fusion attention input, which `--override-dims` declares replicated across TP ranks: ``` --filter self_attn\.inputs\.hidden_states --override-dims ^non_intrusive__model\.layers\.\d+\.self_attn\.inputs\.hidden_states$:bs h # tp:replicated ``` Declaring the axis replicated makes the comparator check that every rank holds the same value, rather than summing partial contributions across ranks. Layer-entry `inputs.1` tensors are still captured, but they are excluded from the comparison by `--filter` (see below). If the comparator returns exit code 0 but compared **zero layers** (baseline/target name mismatch), the test fails with a diagnostic message rather than silently passing. ### Capture signature A `capture_signature` (SHA-1 hash of schema version, max\_tokens, ignore\_eos, TP size, dumper filter, comparator filter, and fusion backend) is computed per run. The HF store uses this signature during fetch to ensure only baselines with an identical capture and comparison contract are considered. If the signature changes (e.g. you add layers to the capture set or change TP), the framework establishes a fresh baseline instead of erroring on incompatible tensors. ### Fusion and what gets compared The harness pins FlashInfer all-reduce fusion **on** (`--flashinfer-allreduce-fusion-backend trtllm`; the SM90 auto-enable was dropped in #23402). Two names are captured per layer: * `inputs.1` — hidden states entering the layer. The FFN boundary defers the cross-layer all-reduce, so with fusion active these are rank-local TP-partial sums. CPU-side reduction of them is not the kernel's semantic output and drifts across self-hosted runners, so they are **not** compared. `_assert_fused_tp_layout()` keeps them only as a guard: if they come back replicated, fusion silently fell back and the test fails. * `self_attn.inputs.hidden_states` — the value attention consumes after `attn_boundary.prepare()`, i.e. post fused all-reduce and residual RMSNorm. This is replicated across ranks and is what the comparator actually diffs. This is why a stale baseline captured with fusion *not* initialized is incompatible: it holds replicated layer-entry tensors where the target holds TP-partial ones, which reads as an approximately `tp_size`x mismatch. Fusion initialization failures are tracked in [FlashInfer #3676](https://github.com/flashinfer-ai/flashinfer/issues/3676) and [SGLang #30875](https://github.com/sgl-project/sglang/issues/30875). *** ## Environment Variables | Variable | Default | Description | | - | - | - | | `SGLANG_PRECISION_MODELS` | `zai-org/GLM-5.1-FP8` | Comma-separated HuggingFace model IDs to test | | `SGLANG_PRECISION_BASELINE_DIR` | `/tmp/sglang_precision_baselines` | Local directory for baseline tensors | | `SGLANG_PRECISION_DIFF_THRESHOLD` | `1e-3` | Per-tensor relative diff threshold | | `SGLANG_PRECISION_FORCE_UPDATE` | `0` | Set to `1` to skip comparison and unconditionally refresh baseline | | `SGLANG_PRECISION_COMMIT` | *(auto-detected from git)* | Override the sglang commit SHA tagged on push | | `SGLANG_PRECISION_HF_REPO` | *(required)* | HuggingFace dataset repo for cross-runner baseline storage | | `SGLANG_PRECISION_HF_REVISION` | `main` | Branch/revision of the HF dataset | | `SGLANG_PRECISION_HF_TOKEN` | *(required in CI)* | HuggingFace token with write access to the dataset. Kept off `HF_TOKEN`, which already carries the runner's gated-model read token | | `SGLANG_PRECISION_HF_READ_ONLY` | `0` | Set to `1` to fetch and compare without updating the shared baseline store | *** ## CI Integration ### Workflow job The test is registered on the `nightly-8-gpu-h200` stage in `.github/workflows/nightly-test-nvidia.yml`, which runs through `_pr-test-stage.yml` like every other CUDA stage. The baseline env below is exported on every scheduled stage; only this test reads it. Key CI configuration: ```yaml theme={null} - name: Export precision baseline env if: inputs.scheduled env: BASELINE_HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }} run: | { echo "SGLANG_PRECISION_BASELINE_DIR=/tmp/sglang_precision_baselines" echo "SGLANG_PRECISION_HF_REPO=${{ vars.SGLANG_PRECISION_HF_REPO }}" echo "SGLANG_PRECISION_HF_REVISION=${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }}" echo "SGLANG_PRECISION_COMMIT=${{ github.sha }}" echo "SGLANG_PRECISION_HF_TOKEN=${BASELINE_HF_TOKEN}" } >> "$GITHUB_ENV" ``` `SGLANG_PRECISION_HF_TOKEN` rather than `HF_TOKEN`: the latter already carries the runner's gated-model read token, and overwriting it would turn every gated model on the job into a 401. Standalone `/rerun-test` runs fetch the nightly baseline without changing it. Maintainers can refresh it from an in-repo PR with `/rerun-test --refresh-precision-baseline test/registered/debug_utils/test_nightly_precision_regression.py`. ### Required GitHub secrets/variables | Name | Type | Purpose | | - | - | - | | `SGLANG_PRECISION_HF_REPO` | Repository variable | HF dataset repo ID (e.g. `org/sglang-precision-baselines`) — **required**, the test errors if unset | | `SGLANG_PRECISION_HF_REVISION` | Repository variable (optional) | Dataset branch (defaults to `main`) | | `HF_TOKEN_PRECISION_STORE` | Repository secret | HF token with write access to the dataset; exported to the job as `SGLANG_PRECISION_HF_TOKEN` | ### GitHub Step Summary When running in CI, the test writes a Markdown table to the GitHub Actions job summary showing each model's status (`PASSED`, `FAILED`, `BASELINE_ESTABLISHED`, or `ERROR`). *** ## HF Dataset Storage Layout Baselines are organized in the HF dataset as: ``` ///
/run-/ ├── meta.json # Run metadata (model, commit, hardware, thresholds, stats) ├── comparator_report.jsonl # Per-tensor comparison results └── tensors/ ├── layer_0_inputs_1.pt ├── layer_7_inputs_1.pt └── ... ``` A top-level `manifest.jsonl` tracks all runs with one JSON object per line. Each manifest row carries a `capture_signature` field so that fetch selects only baselines with a matching capture shape. The `prune_old_runs()` function (callable manually) retains daily runs for 30 days and keeps one run per week beyond that window. *** ## How to Add a New Model ### Option A: Add to the default model list (CI) Edit the default in `test/registered/debug_utils/test_nightly_precision_regression.py`: ```python theme={null} DEFAULT_MODELS_FOR_NIGHTLY_PRECISION = "zai-org/GLM-5.1-FP8,your-org/your-model" ``` Or set the `SGLANG_PRECISION_MODELS` environment variable in the CI workflow to override the default. ### Option B: Run locally for a specific model ```bash theme={null} export SGLANG_PRECISION_MODELS="your-org/your-model" export SGLANG_PRECISION_BASELINE_DIR="/tmp/my_precision_baselines" export SGLANG_PRECISION_DIFF_THRESHOLD="1e-3" export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines" export SGLANG_PRECISION_HF_TOKEN="hf_..." cd test python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v ``` ### Step-by-step: adding a model to the nightly CI 1. **Verify the model works with the dumper.** Run locally first to ensure hidden states are captured correctly: ```bash theme={null} export SGLANG_PRECISION_MODELS="your-org/your-model" export SGLANG_PRECISION_BASELINE_DIR="/tmp/test_baselines" export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines" export SGLANG_PRECISION_HF_TOKEN="hf_..." export SGLANG_PRECISION_FORCE_UPDATE="1" # first run: establish baseline cd test python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v -k test_precision ``` 2. **Run a comparison pass** (remove `FORCE_UPDATE`): ```bash theme={null} unset SGLANG_PRECISION_FORCE_UPDATE python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v -k test_precision ``` This should report `PASSED` if the engine is numerically stable for the model. 3. **Set the tensor-parallelism size.** If the model requires TP > 1, the test harness defaults to `tp_size=8` for all models. To customize, modify the `ModelLaunchSettings` construction in the test or pass extra server arguments: ```python theme={null} # In setUpClass or via env-driven logic cls.models = [ModelLaunchSettings("your-org/your-model", tp_size=4)] ``` 4. **Adjust the diff threshold if needed.** FP8 or quantized models may exhibit larger numerical differences. Set `SGLANG_PRECISION_DIFF_THRESHOLD` to an appropriate value (e.g., `1e-2` for FP8). 5. **Add to the default model list** or configure `SGLANG_PRECISION_MODELS` in the CI workflow. ### Considerations for model-specific adjustments | Concern | How to handle | | - | - | | TP size != 8 | Override `tp_size` in `ModelLaunchSettings` or add model-specific logic | | Quantized models (FP8, GPTQ) | Loosen `SGLANG_PRECISION_DIFF_THRESHOLD` (e.g., `1e-2`) | | Model needs extra server args | Pass them via `ModelLaunchSettings(model, extra_args=["--quantization", "fp8"])` | | Model needs different prompt | Modify `PROMPT` constant or make it model-configurable | | MoE models with TP partial sums | Already handled by `--override-dims` (`bs h[tp:partial]`) | | Fewer/more capture layers | Adjust `LAYER_CAPTURE_STRIDE` (default 8); set lower for smaller models | | Decode not captured | Ensure `--max-total-tokens` is well above the scheduler's decode reservation (default 512); the test uses 4096 | *** ## Running Locally ### Prerequisites * SGLang installed in development mode * GPUs matching the model's requirements * `huggingface_hub` installed * A **HuggingFace dataset** for baseline storage and a write-capable `SGLANG_PRECISION_HF_TOKEN`. The HF store is **mandatory** — `SGLANG_PRECISION_HF_REPO` must be set or the test will error at startup. This is because the nightly CI runners are ephemeral (no persistent local disk), so baselines must survive across runs via the HF dataset. There is currently no local-only fallback. ### Quick local test ```bash theme={null} # All three are required — the test errors if SGLANG_PRECISION_HF_REPO is unset. export SGLANG_PRECISION_MODELS="Qwen/Qwen2.5-0.5B-Instruct" export SGLANG_PRECISION_BASELINE_DIR="/tmp/precision_baselines" export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines" export SGLANG_PRECISION_HF_TOKEN="hf_..." # First run: establish baseline cd test python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v # Second run: compare against baseline python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v ``` ### Force-refresh a baseline ```bash theme={null} export SGLANG_PRECISION_FORCE_UPDATE="1" python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v ``` *** ## Interpreting Results ### Status codes | Status | Meaning | | - | - | | `BASELINE_ESTABLISHED` | No prior baseline with a matching signature existed; today's tensors saved as the new baseline | | `PASSED` | All per-layer hidden states are within the diff threshold; baseline updated | | `FAILED` | One or more layers exceeded the diff threshold, or 0 layers were compared (baseline/target mismatch); diagnostic data pushed to HF | | `ERROR` | Server launch, inference, or comparison encountered an unexpected error | ### Output example ``` ============================================================ Nightly Precision Regression Summary ============================================================ Model Status Details ------------------------------------------------------------ zai-org/GLM-5.1-FP8 PASSED comparison ok, baseline updated Qwen/Qwen2.5-0.5B-Instruct FAILED tensor=layer_23.inputs_1 rel_diff=0.0152 ============================================================ ``` ### When a failure is detected 1. The comparator output is saved to `/tmp/nightly_precision__*.log` 2. The failing tensors and comparator report are pushed to the HF dataset with `pass_label="failed"` for offline diagnosis 3. The GitHub Step Summary includes the failure details 4. The CI job exits with a non-zero status *** ## Baseline Management ### Local baselines Baselines are stored at: ``` $SGLANG_PRECISION_BASELINE_DIR//nightly_precision/*.pt ``` A `baseline_meta.json` next to the tensors records the timestamp and commit that produced the baseline. ### HF dataset baselines * **Fetch:** At test start, if no local baseline exists, the latest signature-matched baseline is downloaded from the HF dataset. * **Push:** After each run, tensors and metadata are uploaded to the dataset. * **Prune:** Use `prune_old_runs()` to garbage-collect old baselines (keeps 30 days of daily runs, one per week after that). ### Refreshing a stale baseline If an intentional numerical change (e.g., kernel optimization, model refactor) causes a comparison failure: 1. Verify the change is intentional 2. Set `SGLANG_PRECISION_FORCE_UPDATE=1` and run the test once to establish a new baseline 3. Commit any necessary threshold adjustments If you change the capture configuration (stride, TP size, etc.), the `capture_signature` will differ and the framework automatically establishes a fresh baseline — no manual intervention needed. *** ## Known Limitations ### Baseline drift The framework uses a **rolling baseline**: every successful comparison updates the baseline to the current run's tensors. This means the reference shifts forward each day. While individual day-to-day diffs stay within the configured threshold, tiny numerical differences can **accumulate over time**, causing the baseline to silently drift away from the original golden values. **Implications:** * The framework detects **regressions** (a sudden, large numerical change between consecutive runs), not **absolute accuracy** relative to a fixed reference. * Over weeks or months, the cumulative drift may become significant enough to mask a real regression that happened gradually, or to cause a false-positive failure when the drift eventually crosses the threshold. **Mitigation strategies (not yet implemented):** * Periodically re-establish a fresh anchor baseline from a known-good reference commit. * Track the cumulative drift in the manifest metadata and alert when it exceeds a long-term budget. * Compare against a fixed "epoch" baseline in addition to the rolling one. ### No local-only mode The test requires a HuggingFace dataset (`SGLANG_PRECISION_HF_REPO`) and a write-capable `SGLANG_PRECISION_HF_TOKEN`. There is no local-only fallback. This is by design — CI runners have no persistent local disk, so the HF dataset is the only way to carry baselines across runs. If you need to run the test locally, you must set up a HF dataset (even a private one) and provide the corresponding token. *** ## File Reference | File | Role | | - | - | | `test/registered/debug_utils/test_nightly_precision_regression.py` | Main test — server lifecycle, dump, compare, report | | `python/sglang/test/precision_baseline_store.py` | HF dataset store — push, fetch, prune baselines | | `python/sglang/srt/debug_utils/comparator/` | Tensor comparison engine | | `python/sglang/srt/debug_utils/dumper.py` | Runtime hidden-state capture | | `.github/workflows/nightly-test-nvidia.yml` | CI workflow definition | | `test/run_suite.py` | Test suite registration (includes `nightly-test-8-gpu-h200`) | # References Source: https://docs.sglang.io/docs/references/overview FAQ, environment variables, production metrics, deployment guides, and more. * [FAQ](./faq) * [Environment Variables](./environment_variables) * [Production Metrics](./production_metrics) * [Production Request Trace](./production_request_trace) * [Multi-Node Deployment](./multi_node_deployment/multi_node) * [Custom Chat Template](./custom_chat_template) * [Frontend Language](./frontend/frontend_tutorial) * [Post-Training Integration](./post_training_integration) # Post-Training Integration Source: https://docs.sglang.io/docs/references/post_training_integration SGLang has become the de facto inference backend for modern LLM training frameworks, powering state-of-the-art models across the industry. From GLM-4.6 to Qwen3, leading models leverage SGLang's high-performance inference during reinforcement learning and post-training workflows. What makes SGLang essential for post-training? * Open-To-Use Refit Functionality: diverse method for colocate or disaggregate * Easy To Postpone Generation: enable partial rollout and dedicated rollout control * Fine-Grained Engine Sleep And Wake Up: facilitate maxium-powered rollout and training * Training Serving Alignment: ensure the performance consistency in training and serving * Load Balancing Router: cache-aware load-balancing for high-throughput rollout * Deterministic Inference: ensure zero kl divergence between rollout and training These capabilities, combined with native integration support across major frameworks, have established SGLang as the infrastructure backbone for modern LLM/VLMs post-training. We also share our latest work in this slide, [Optimizing Large-Scale RL with SGLang](https://gamma.app/docs/Optimizing-RL-with-SGLang-y0kqgj877k34779). ## Adoption * [**Miles**](https://github.com/radixark/miles): Enterprise-scale RL framework for large MoE models with SGLang-native rollout, speculative training, and production-grade stability * [**slime**](https://github.com/THUDM/slime): Post-training framework combining Megatron and SGLang, used to train GLM-4.6 * [**AReaL**](https://github.com/inclusionAI/AReaL): Fully asynchronous RL system achieving 2.77x speedup with SGLang backend for continuous rollout generation * [**ROLL**](https://github.com/alibaba/ROLL): ROLL is an efficient and user-friendly RL library designed for Large Language Models utilizing Large Scale GPU resources * [**verl**](https://github.com/volcengine/verl): Full-stack RLHF framework supporting PPO, GRPO, and ReMax with modular SGLang integration * [**Unsloth**](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide): 2x faster fine-tuning with optimized kernels, deploys seamlessly with SGLang inference * [**LLaMA Factory**](https://github.com/hiyouga/LLaMA-Factory): Unified framework for training 100+ LLMs with LoRA, QLoRA, and full fine-tuning methods * [**Tunix**](https://github.com/google/tunix): Google's JAX-native library for LLM post-training with SFT, DPO, PPO, and GRPO support * [**RL2**](https://github.com/ChenmienTan/RL2): Ray Less Reinforcement Learning, a concise library of post-training for large language models ## Collaboration Due to the privacy of the design parternes, we cannot list the companies that adopt SGLang for post-training. However, we are happy to share the details with you if you are interested and trust the choice among 10+ top companies and frontier labs across US and China. If you are interested in integrating SGLang with your training framework or need technical support, we're here to help! Reach out to us at **[rl\_team@lmsys.org](mailto:rl_team@lmsys.org)** for partnerships, integration guidance, and custom feature development. # Production Metrics Source: https://docs.sglang.io/docs/references/production_metrics SGLang exposes the following metrics via Prometheus. You can enable it by adding `--enable-metrics` when you launch the server. An example of the monitoring dashboard is available in [examples/monitoring/grafana.json](https://github.com/sgl-project/sglang/blob/main/examples/monitoring/grafana/dashboards/json/sglang-dashboard.json). ## Language model metrics `sglang:time_to_first_token_seconds` measures the time from request creation in the API server to the first token. For streaming requests it ends when the first output arrives after detokenization, which may contain no printable text. Non-streaming clients only see the final response, so for them it ends when the scheduler produced the first token. Here is an example of the metrics: ```text Output theme={null} $ curl http://localhost:30000/metrics # HELP sglang:prompt_tokens_total Number of prefill tokens processed. # TYPE sglang:prompt_tokens_total counter sglang:prompt_tokens_total{model_name="meta-llama/Llama-3.1-8B-Instruct"} 8.128902e+06 # HELP sglang:generation_tokens_total Number of generation tokens processed. # TYPE sglang:generation_tokens_total counter sglang:generation_tokens_total{model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.557572e+06 # HELP sglang:token_usage The token usage # TYPE sglang:token_usage gauge sglang:token_usage{model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.28 # HELP sglang:cache_hit_rate The cache hit rate # TYPE sglang:cache_hit_rate gauge sglang:cache_hit_rate{model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.007507552643049313 # HELP sglang:time_to_first_token_seconds Histogram of time to first token in seconds. # TYPE sglang:time_to_first_token_seconds histogram sglang:time_to_first_token_seconds_sum{model_name="meta-llama/Llama-3.1-8B-Instruct"} 2.3518979474117756e+06 sglang:time_to_first_token_seconds_bucket{le="0.001",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 sglang:time_to_first_token_seconds_bucket{le="0.005",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 sglang:time_to_first_token_seconds_bucket{le="0.01",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 sglang:time_to_first_token_seconds_bucket{le="0.02",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 sglang:time_to_first_token_seconds_bucket{le="0.04",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1.0 sglang:time_to_first_token_seconds_bucket{le="0.06",model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.0 sglang:time_to_first_token_seconds_bucket{le="0.08",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:time_to_first_token_seconds_bucket{le="0.1",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:time_to_first_token_seconds_bucket{le="0.25",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:time_to_first_token_seconds_bucket{le="0.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:time_to_first_token_seconds_bucket{le="0.75",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:time_to_first_token_seconds_bucket{le="1.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 27.0 sglang:time_to_first_token_seconds_bucket{le="2.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 140.0 sglang:time_to_first_token_seconds_bucket{le="5.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 314.0 sglang:time_to_first_token_seconds_bucket{le="7.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 941.0 sglang:time_to_first_token_seconds_bucket{le="10.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1330.0 sglang:time_to_first_token_seconds_bucket{le="15.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1970.0 sglang:time_to_first_token_seconds_bucket{le="20.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 2326.0 sglang:time_to_first_token_seconds_bucket{le="25.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 2417.0 sglang:time_to_first_token_seconds_bucket{le="30.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 2513.0 sglang:time_to_first_token_seconds_bucket{le="+Inf",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11008.0 sglang:time_to_first_token_seconds_count{model_name="meta-llama/Llama-3.1-8B-Instruct"} 11008.0 # HELP sglang:e2e_request_latency_seconds Histogram of End-to-end request latency in seconds # TYPE sglang:e2e_request_latency_seconds histogram sglang:e2e_request_latency_seconds_sum{model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.116093850019932e+06 sglang:e2e_request_latency_seconds_bucket{le="0.3",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 sglang:e2e_request_latency_seconds_bucket{le="0.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:e2e_request_latency_seconds_bucket{le="0.8",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:e2e_request_latency_seconds_bucket{le="1.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:e2e_request_latency_seconds_bucket{le="1.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:e2e_request_latency_seconds_bucket{le="2.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:e2e_request_latency_seconds_bucket{le="2.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:e2e_request_latency_seconds_bucket{le="5.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.0 sglang:e2e_request_latency_seconds_bucket{le="10.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 10.0 sglang:e2e_request_latency_seconds_bucket{le="15.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11.0 sglang:e2e_request_latency_seconds_bucket{le="20.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 14.0 sglang:e2e_request_latency_seconds_bucket{le="30.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 247.0 sglang:e2e_request_latency_seconds_bucket{le="40.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 486.0 sglang:e2e_request_latency_seconds_bucket{le="50.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 845.0 sglang:e2e_request_latency_seconds_bucket{le="60.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1513.0 sglang:e2e_request_latency_seconds_bucket{le="+Inf",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11228.0 sglang:e2e_request_latency_seconds_count{model_name="meta-llama/Llama-3.1-8B-Instruct"} 11228.0 # HELP sglang:func_latency_seconds Function latency in seconds # TYPE sglang:func_latency_seconds histogram sglang:func_latency_seconds_sum{name="generate_request"} 4.514771912145079 sglang:func_latency_seconds_bucket{le="0.05",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.07500000000000001",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.1125",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.16875",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.253125",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.3796875",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.56953125",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.8542968750000001",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="1.2814453125",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="1.9221679687500002",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="2.8832519531250003",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="4.3248779296875",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="6.487316894531251",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="9.730975341796876",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="14.596463012695313",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="21.89469451904297",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="32.84204177856446",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="49.26306266784668",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="+Inf",name="generate_request"} 14007.0 sglang:func_latency_seconds_count{name="generate_request"} 14007.0 # HELP sglang:num_running_reqs The number of running requests # TYPE sglang:num_running_reqs gauge sglang:num_running_reqs{model_name="meta-llama/Llama-3.1-8B-Instruct"} 162.0 # HELP sglang:num_used_tokens The number of used tokens # TYPE sglang:num_used_tokens gauge sglang:num_used_tokens{model_name="meta-llama/Llama-3.1-8B-Instruct"} 123859.0 # HELP sglang:gen_throughput The generate throughput (token/s) # TYPE sglang:gen_throughput gauge sglang:gen_throughput{model_name="meta-llama/Llama-3.1-8B-Instruct"} 86.50814177726902 # HELP sglang:num_queue_reqs The number of requests in the waiting queue # TYPE sglang:num_queue_reqs gauge sglang:num_queue_reqs{model_name="meta-llama/Llama-3.1-8B-Instruct"} 2826.0 # HELP sglang:spec_num_steps Currently active speculative_num_steps. # TYPE sglang:spec_num_steps gauge sglang:spec_num_steps{model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.0 # HELP sglang:spec_num_draft_tokens Currently active speculative_num_draft_tokens (decouples from steps under topk>1). # TYPE sglang:spec_num_draft_tokens gauge sglang:spec_num_draft_tokens{model_name="meta-llama/Llama-3.1-8B-Instruct"} 4.0 ``` ## Request-level time per output token (TPOT) `sglang:request_time_per_output_token_seconds` records TPOT once per successfully finished request: ```text theme={null} TPOT = (finished_time - first_token_time) / (completion_tokens - 1) = 1 / meta_info["decode_throughput"] ``` The first token is excluded because its latency belongs to TTFT. Requests with fewer than two output tokens, streaming requests that finish in their first output, aborted requests, and PD prefill workers do not contribute. The histogram is split by `is_streaming` and reuses the `--bucket-inter-token-latency` buckets. Unlike the token-weighted `sglang:inter_token_latency_seconds`, this metric gives every qualifying request equal weight. p95 TPOT per model: ```promql theme={null} histogram_quantile( 0.95, sum by (model_name, is_streaming, le) ( rate(sglang:request_time_per_output_token_seconds_bucket[5m]) ) ) ``` Mean TPOT is `rate(..._sum[5m]) / rate(..._count[5m])`. Per-request decode tokens/s is its reciprocal, so p5 TPS is `1 / p95 TPOT`; the reciprocal of mean TPOT is not mean TPS. ## Diffusion metrics SGLang Diffusion exposes request, queue, stage and LoRA metrics with `--enable-metrics`. See [Diffusion production metrics](/docs/sglang-diffusion/production_metrics) for the metric reference, counting semantics and disaggregated scraping setup. ## Setup Guide This section describes how to set up the monitoring stack (Prometheus + Grafana) provided in the `examples/monitoring` directory. ### Prerequisites * Docker and Docker Compose installed * SGLang server running with metrics enabled ### Usage 1. **Start your SGLang server with metrics enabled:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path \ --port 30000 \ --enable-metrics \ --enable-mfu-metrics ``` Replace `` with the actual path to your model (e.g., `meta-llama/Meta-Llama-3.1-8B-Instruct`). Ensure the server is accessible from the monitoring stack (you might need `--host 0.0.0.0` if running in Docker). By default, the metrics endpoint will be available at `http://:30000/metrics`. 2. **Navigate to the monitoring example directory:** ```bash Command theme={null} cd examples/monitoring ``` 3. **Start the monitoring stack:** ```bash Command theme={null} docker compose up -d ``` This command will start Prometheus and Grafana in the background. 4. **Access the monitoring interfaces:** * **Grafana:** Open your web browser and go to [http://localhost:3000](http://localhost:3000). * **Prometheus:** Open your web browser and go to [http://localhost:9090](http://localhost:9090). 5. **Log in to Grafana:** * Default Username: `admin` * Default Password: `admin` You will be prompted to change the password upon your first login. 6. **View the Dashboard:** The SGLang dashboard is pre-configured and should be available automatically. Navigate to `Dashboards` -> `Browse` -> `SGLang Monitoring` folder -> `SGLang Dashboard`. ### Troubleshooting * **Port Conflicts:** If you encounter errors like "port is already allocated," check if other services (including previous instances of Prometheus/Grafana) are using ports `9090` or `3000`. Use `docker ps` to find running containers and `docker stop ` to stop them, or use `lsof -i :` to find other processes using the ports. You might need to adjust the ports in the `docker-compose.yaml` file if they permanently conflict with other essential services on your system. To modify Grafana's port to the other one(like 3090) in your Docker Compose file, you need to explicitly specify the port mapping under the grafana service. Option 1: Add GF\_SERVER\_HTTP\_PORT to the environment section: ``` environment: - GF_AUTH_ANONYMOUS_ENABLED=true - GF_SERVER_HTTP_PORT=3090 # <-- Add this line ``` Option 2: Use port mapping: ``` grafana: image: grafana/grafana:latest container_name: grafana ports: - "3090:3000" # <-- Host:Container port mapping ``` * **Connection Issues:** * Ensure both Prometheus and Grafana containers are running (`docker ps`). * Verify the Prometheus data source configuration in Grafana (usually auto-configured via `grafana/datasources/datasource.yaml`). Go to `Connections` -> `Data sources` -> `Prometheus`. The URL should point to the Prometheus service (e.g., `http://prometheus:9090`). * Confirm that your SGLang server is running and the metrics endpoint (`http://:30000/metrics`) is accessible *from the Prometheus container*. If SGLang is running on your host machine and Prometheus is in Docker, use `host.docker.internal` (on Docker Desktop) or your machine's network IP instead of `localhost` in the `prometheus.yaml` scrape configuration. * **No Data on Dashboard:** * Generate some traffic to your SGLang server to produce metrics. For example, run a benchmark: ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang --dataset-name random --num-prompts 100 --random-input 128 --random-output 128 ``` * Check the Prometheus UI (`http://localhost:9090`) under `Status` -> `Targets` to see if the SGLang endpoint is being scraped successfully. * Verify the `model_name` and `instance` labels in your Prometheus metrics match the variables used in the Grafana dashboard. You might need to adjust the Grafana dashboard variables or the labels in your Prometheus configuration. ### Configuration Files The monitoring setup is defined by the following files within the `examples/monitoring` directory: * `docker-compose.yaml`: Defines the Prometheus and Grafana services. * `prometheus.yaml`: Prometheus configuration, including scrape targets. * `grafana/datasources/datasource.yaml`: Configures the Prometheus data source for Grafana. * `grafana/dashboards/config/dashboard.yaml`: Tells Grafana to load dashboards from the specified path. * `grafana/dashboards/json/sglang-dashboard.json`: The actual Grafana dashboard definition in JSON format. You can customize the setup by modifying these files. For instance, you might need to update the `static_configs` target in `prometheus.yaml` if your SGLang server runs on a different host or port. #### Check if the metrics are being collected Run: ```text Output theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 3000 \ --random-input 1024 \ --random-output 1024 \ --random-range-ratio 0.5 ``` to generate some requests. Then you should be able to see the metrics in the Grafana dashboard. ## Estimated Performance Metrics (MFU-related) SGLang exports the following estimated per-GPU counters that can be used to derive Model FLOPs Utilization (MFU)-related signals: * `sglang:estimated_flops_per_gpu_total`: Estimated floating-point operations. * `sglang:estimated_read_bytes_per_gpu_total`: Estimated bytes read from memory. * `sglang:estimated_write_bytes_per_gpu_total`: Estimated bytes written to memory. These metrics are available when both `--enable-metrics` and `--enable-mfu-metrics` are enabled. These are cumulative counters. Use Prometheus `rate(...)` to get per-second values. ### PromQL examples Average TFLOPS per GPU: ```promql theme={null} rate(sglang:estimated_flops_per_gpu_total[1m]) / 1e12 ``` Average estimated memory bandwidth in GB/s: ```promql theme={null} (rate(sglang:estimated_read_bytes_per_gpu_total[1m]) + rate(sglang:estimated_write_bytes_per_gpu_total[1m])) / 1e9 ``` ### Notes * These metrics are estimates intended for observability and trend analysis. * Estimated memory bytes reflect modeled traffic and are not a direct hardware counter from GPU profilers. # Production Request Tracing Source: https://docs.sglang.io/docs/references/production_request_trace SGLang exports request trace data based on the OpenTelemetry Collector. You can enable tracing by adding the `--enable-trace` and configure the OpenTelemetry Collector endpoint using `--otlp-traces-endpoint` when launching the server. You can find example screenshots of the visualization in [https://github.com/sgl-project/sglang/issues/8965](https://github.com/sgl-project/sglang/issues/8965). ## Setup Guide This section explains how to configure the request tracing and export the trace data. 1. Install the required packages and tools * install Docker and Docker Compose * install the dependencies ```bash Command theme={null} # enter the SGLang root directory pip install -e "python[tracing]" # or manually install the dependencies using pip pip install opentelemetry-sdk opentelemetry-api opentelemetry-exporter-otlp opentelemetry-exporter-otlp-proto-grpc ``` 2. Launch OpenTelemetry collector and Jaeger ```bash Command theme={null} docker compose -f examples/monitoring/tracing_compose.yaml up -d ``` 3. Start your SGLang server with tracing enabled ```bash Command theme={null} # set env variables export SGLANG_OTLP_EXPORTER_SCHEDULE_DELAY_MILLIS=500 export SGLANG_OTLP_EXPORTER_MAX_EXPORT_BATCH_SIZE=64 # start the prefill and decode server python -m sglang.launch_server --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 # start the model-gate-way python -m sglang_router.launch_router --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 ``` Replace `0.0.0.0:4317` with the actual endpoint of the OpenTelemetry collector. If you launched the openTelemetry collector with tracing\_compose.yaml, the default receiving port is 4317. To use the HTTP/protobuf span exporter, set the following environment variable and point to an HTTP endpoint, for example, `http://0.0.0.0:4318/v1/traces`. ```bash Command theme={null} export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf ``` 4. Raise some requests 5. Observe whether trace data is being exported * Access port 16686 of Jaeger using a web browser to visualize the request traces. * The OpenTelemetry Collector also exports trace data in JSON format to /tmp/otel\_trace.json. In a follow-up patch, we will provide a tool to convert this data into a Perfetto-compatible format, enabling visualization of requests in the Perfetto UI. 6. Dynamically adjust trace level The trace level accepts configurable values from `0` to `3`. The meanings of different trace level values are as follows: ``` 0: disable tracing 1: Trace important slices 2: Trace all slices except nested ones 3: Trace all slices (default) ``` **At startup** — set `SGLANG_TRACE_LEVEL` before launching the server: ```bash Command theme={null} SGLANG_TRACE_LEVEL=2 python -m sglang.launch_server --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 ``` **At runtime** — dynamically adjust via HTTP API without restarting: ```bash Command theme={null} curl http://0.0.0.0:30000/set_trace_level?level=2 ``` Replace `0.0.0.0:30000` with your actual server address, and replace `level=2` with the level you want to set. **Note**: You must set the parameter `--enable-trace`; otherwise, the trace capability will not be enabled regardless of any dynamic adjustments to the trace level. ## Async Tracing (Reducing Performance Overhead) When batch sizes are large, synchronous OTel span creation can degrade inference throughput due to thread-safe locking and background export threads. Async tracing moves all span creation to a dedicated exporter process via ZMQ, keeping the scheduler and tokenizer hot paths free of OTel overhead. **Enable async tracing:** ```bash theme={null} SGLANG_TRACE_ASYNC=1 python -m sglang.launch_server --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 ``` **How it works:** * A daemon exporter process is started per worker process (scheduler, tokenizer, etc.). * The root span is still created in the caller process (one per request, negligible cost), preserving cross-process span linking via `traceparent`. * Thread spans and slice spans are buffered as lightweight operation dicts and flushed to the exporter via ZMQ PUSH/PULL. * Span IDs are pre-generated in the caller process using `TraceCustomIdGenerator.preset_next_span_id()`, so the exported span tree is identical to synchronous mode. * Thread info (scheduler label, TP/DP/PP ranks) is registered once via a callback on `trace_set_thread_info()`. **Tuning:** | Environment Variable | Description | Default | | - | - | - | | `SGLANG_TRACE_ASYNC` | Enable async tracing | `false` | | `SGLANG_TRACE_ASYNC_FLUSH_THRESHOLD` | Max buffered ops before auto-flush | `100` | ## How to add Tracing for slices you're interested in?(API introduction) We have already inserted instrumentation points in the tokenizer and scheduler main threads. If you wish to trace additional request execution segments or perform finer-grained tracing, please use the APIs from the tracing package as described below. **All of the following implementations are done in python/sglang/srt/observability/req\_time\_stats.py. If you want to add another slice, please do it here.** 1. Initialization Every process involved in tracing during the initialization phase should execute: ```python Example theme={null} process_tracing_init(otlp_traces_endpoint, server_name) ``` The otlp\_traces\_endpoint is obtained from the arguments, and you can set server\_name freely, but it should remain consistent across all processes. Every thread involved in tracing during the initialization phase should execute: ```python Example theme={null} trace_set_thread_info("thread label", tp_rank, dp_rank) ``` The "thread label" can be regarded as the name of the thread, used to distinguish different threads in the visualization view. 2. Create a trace context for a request Each request needs to call `TraceReqContext()` to initialize a request context, which is used to generate slice spans and record request stage info. You can either store it within the request object or maintain it as a global variable. 3. Mark the beginning and end of a request ``` trace_ctx.trace_req_start(). trace_ctx.trace_req_finish() ``` trace\_req\_start() and trace\_req\_finish() must be called within the same process, for example, in the tokenizer. 4. Add tracing for a slice * Add slice tracing normally: ```python Example theme={null} trace_ctx.trace_slice_start(RequestStage.TOKENIZER.stage_name) trace_ctx.trace_slice_end(RequestStage.TOKENIZER.stage_name) or trace_ctx.trace_slice(slice: TraceSliceContext) ``` - The end of the last slice in a thread must be marked with thread\_finish\_flag=True, or explicitly call trace\_ctx.abort(); otherwise, the thread's span will not be properly generated. ```python Example theme={null} trace_ctx.slice_end(RequestStage.D.stage_name, thread_finish_flag = True) trace_ctx.abort() ``` 5. When the request execution flow transfers to another thread, the thread context needs to be explicitly rebuilt. * receiver: Execute the following code after receiving the request via ZMQ ```python Example theme={null} trace_ctx.rebuild_thread_context() ``` ## How to Extend the Tracing Framework to Support Complex Tracing Scenarios The currently provided tracing package still has potential for further development. If you wish to build more advanced features upon it, you must first understand its existing design principles. The core of the tracing framework's implementation lies in the design of the span structure and the trace context. To aggregate scattered slices and enable concurrent tracking of multiple requests, we have designed a three-level trace context structure or span structure: `TraceReqContext`, `TraceThreadContext` and `TraceSliceContext`. Their relationship is as follows: ``` TraceReqContext (req_id="req-123") ├── TraceThreadContext(thread_label="scheduler", tp_rank=0) | └── TraceSliceContext(slice_name="prefill") | └── TraceThreadContext(thread_label="scheduler", tp_rank=1) └── TraceSliceContext(slice_name="prefill") ``` Each traced request maintains a global `TraceReqContext` and creates a corresponding request span. For every thread that processes the request, a `TraceThreadContext` is recorded and a thread span is created. The `TraceThreadContext` is nested within the `TraceReqContext`, and each currently traced code slice—potentially nested—is stored in its associated `TraceThreadContext`. In addition to the above hierarchy, each slice also records its previous slice via Span.add\_link(), which can be used to trace the execution flow. # CLI reference Source: https://docs.sglang.io/docs/sglang-diffusion/api/cli Run one-off generation tasks and launch the HTTP server from the command line. Use the CLI for one-off generation with `sglang generate` or to start a persistent HTTP server with `sglang serve`. ### Overlay repos for non-diffusers models If `--model-path` points to a supported non-diffusers source repo, SGLang can resolve it through a self-hosted overlay repo. SGLang first checks a built-in overlay registry. Concrete built-in mappings can be added over time without changing the CLI surface. Override example: ```bash Command theme={null} export SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY='{ "Wan-AI/Wan2.2-S2V-14B": { "overlay_repo_id": "your-org/Wan2.2-S2V-14B-overlay", "overlay_revision": "main" } }' sglang generate \ --model-path Wan-AI/Wan2.2-S2V-14B \ --config configs/wan_s2v.yaml ``` The overlay repo should be a complete diffusers-style/componentized repo You can also pass the overlay repo itself as `--model-path` if it contains `_overlay/overlay_manifest.json`. Notes: 1. `SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY` is only an optional override for development and debugging. It accepts either a JSON object or a path to a JSON file, and can extend or replace built-in entries for the current process. 2. On the first load, SGLang will: * download overlay metadata from the overlay repo * download the required files from the original source repo * materialize a local standard component repo under `~/.cache/sgl_diffusion/materialized_models/` 3. Later loads reuse the materialized local repo. The materialized repo is what the runtime loads as a normal componentized model directory. ## Quick Start ### Generate ```bash Command theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --prompt "A beautiful sunset over the mountains" \ --save-output ``` ### Serve ```bash Command theme={null} sglang serve \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --num-gpus 4 \ --ulysses-degree 2 \ --ring-degree 2 \ --port 30010 ``` For request and response examples, see [OpenAI-Compatible API](/docs/sglang-diffusion/api/openai_api). Use `sglang generate --help` and `sglang serve --help` for the full argument list. The CLI help output is the source of truth for exhaustive flags. ## Common Options ### Model and runtime * `--model-path {MODEL}`: model path or Hugging Face model ID * `--served-model-name {NAME}`: stable model name exposed by serving APIs. Defaults to `--model-id` when set, otherwise `--model-path`. * `--model-variant {NAME}`: semantic checkpoint variant to load when one model repository contains multiple weight partitions. The pipeline maps this stable name to the repository layout before loading; for example, MiniMax-H3 accepts `fl2va` and `ref2va`, or `hybrid` with explicit merged transformer weights to serve all three tasks (see the [H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#singularity-hybrid-weights)). This is a server/load-time choice, unlike a request's `task`. * `--minimax-h3-adaln-cache-path {FILE}`: advanced MiniMax-H3-only inference cache. It replaces the checkpoint's AdaLN projection weights with precomputed outputs and only accepts requests whose exact FP32 timestep plan is included in the cache. It requires unquantized weights and the matching model variant. * `--minimax-h3-adaln-online {true,false}`: rebuild MiniMax-H3 AdaLN outputs from the checkpoint on demand instead of keeping the 24.2 GiB of `adaln_proj` weights resident. Works with any step count or schedule; requires the unquantized native-layout checkpoint. Built plans live in a GPU slab with per-plan LRU eviction and, by default, a pinned-host cache so previously seen schedules swap back in over PCIe instead of re-reading the checkpoint. * `--minimax-h3-adaln-plan-width {N}`: widest timestep plan the online slab is sized for (default 4 covers every task; t2va needs 2, fl2va 3). * `--minimax-h3-adaln-host-cache-gb {GB}`: pinned host memory per rank caching built AdaLN plans (default 8; 0 disables). A 50-step schedule needs about 0.9 (t2va) / 1.33 (fl2va) / 1.77 (ref2va) GB; over-capacity plan sets simply recompute. Expert escape hatches (GPU slot count, experimental fp32 rebuild) are the `SGLANG_DIFFUSION_MINIMAX_H3_ADALN_*` environment variables. * `--model-subfolder {PATH}`: advanced direct override for a component subfolder inside the model repository. Prefer `--model-variant` when the pipeline exposes semantic routing. If both are supplied, they must resolve to the same weight partition. * `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter from a local path, Hugging Face repo/subfolder, or exact Hub file URL * `--lora-weight-name {FILE}`: select one adapter file from a repository that contains multiple LoRA revisions. The Hub download is filtered to that file plus JSON metadata, so unused weights are not downloaded. * `--lora-alpha {N}`: supply the training alpha when a single-file adapter omits both per-layer alpha tensors and `adapter_config.json`. Do not set it when the adapter already records alpha metadata. * PEFT `adapter_config.json` semantics are applied automatically for named adapter slots, RSLoRA, and per-layer `alpha_pattern`. Adapters that require unsupported auxiliary parameters or runtime behavior, such as DoRA, fail before weight injection instead of silently using ordinary LoRA math. * `--lora-merge-mode {auto|merge|dynamic}`: choose how LoRA is applied. `auto` statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks. * `--num-gpus {N}`: number of GPUs to use * `--numa-node {N} [N ...]`: NUMA node for each GPU worker, the i-th value for the worker on local GPU i. When unset, each worker binds its CPUs and host memory to its GPU's NUMA node on multi-socket hosts, as srt does; set `SGLANG_AUTO_NUMA_BIND=0` to turn that off. Unbound workers on a two-socket host (e.g. GB200/GB300) can drift to the other socket, which slows host-bound steps and makes latency vary run to run. * `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and dispatches residency from selected-GPU headroom and workload type: image DiTs stay resident above the 45 GiB threshold, while video DiT placement remains model-specific. It uses FSDP only for validated DiT-offload replacement paths. `speed` keeps `torch.compile` disabled unless a model-specific deployment config opts in after validation; pass `--enable-torch-compile true` to enable it explicitly. Use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes. * `--direct-gpu-weight-loading {true|false}`: opt into direct GPU loading for an unquantized, GPU-resident, TP=1 DiT by materializing its complete checkpoint state dict on GPU. Compatible tensors become parameter storage without another GPU copy, while tensors requiring transforms can still need temporary allocations. It is disabled by default; startup time and peak GPU memory remain model-dependent, so benchmark the target model before deployment. It is incompatible with DiT CPU/layerwise offload and FSDP. * `--tp-size {N}`: tensor parallelism size. Depending on the pipeline, it can shard the DiT, one or more encoders, or both. * `--sp-degree {N}`: sequence parallelism size * `--dp-size {N}` (alias `--data-parallel-size`): number of data-parallel replicas. Each replica is a full copy of the engine on `num_gpus / N` GPUs with its own ingress; generation requests round-robin across replicas, realtime sessions stick to the replica holding their state, and control operations (weights, LoRA, memory occupation, shutdown) apply to every replica. Combines with the other parallelism axes (`num_gpus = dp × cfg × tp × sp`); monolithic serving only. * `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls * `--kv-gather-degree {N}`: sequence-parallel degree that splits rows inside attention and exchanges with one K/V all-gather (queries stay local) instead of Ulysses all-to-all. Non-causal attention only; does not compose with `--ulysses-degree`/`--ring-degree` yet. When no SP degree is set explicitly, `sp_degree=2` defaults to `kv_gather_degree=2` (its measured-win zone) and higher degrees default to Ulysses; under that auto assignment, attention calls the gather path cannot take fall back to the Ulysses exchange, while an explicit degree fails instead of degrading. * `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism. Requests that disable CFG remain valid on a CFG-parallel server, but extra CFG ranks redundantly recompute the single active branch. * `--encoder-parallel {auto|fold|dp|replicate}`: how native encoders use the GPUs in each DiT replica. `auto` TP-folds a native text/image encoder wide enough to benefit, selects batch DP for an explicitly supported native text encoder when it can engage, and otherwise keeps the existing encoder TP layout; `fold` shards native text/image encoders across the full replica whenever dimensions allow; `dp` splits a batched encode across supported native text encoder copies and composes with encoder TP; `replicate` disables folding and batch DP. Encoder collectives never cross `--dp-size` replicas. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel). * `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic * `--enable-torch-compile {true|false}`: compile native diffusion hot paths. When no warmup mode is configured, this also enables server warmup so first real requests do not pay compile latency. * `--offload-during-compile {true|false}`: when compile warmup is active, temporarily layerwise-offload DiT weights and move resident non-DiT components off-device so `max-autotune` fits on tighter-memory GPUs; the configured serving residency is restored before real traffic. Skipped under existing layerwise offload, Cache-DiT, or FSDP. * `--enable-breakable-cuda-graph {true|false}`: capture supported DiT forwards as breakable CUDA graph segments to reduce launch overhead. Requires `--warmup-resolutions` for every served resolution because each resolution is captured separately. On a model outside the validated list it stays enabled with a warning; compare outputs against eager. If the startup warmup captures no graph (the model's denoising stage does not use BCG), the server warns that it runs eager. An `lossless` or `high` request is rejected when it would mount request-scoped DiT fusions that were not present during lossless graph capture; VAE-only request-gated paths remain compatible. * `--bcg-text-buckets {N...}`: prompt-length padding buckets for breakable CUDA graph capture/replay reuse. * `--attention-backend {BACKEND}`: attention backend for native SGLang and diffusers pipelines * `--enable-attention-backend-autotune {true|false}`: for SGLang-native pipelines, benchmark compatible attention backends on each layer's first sufficiently large input and keep a backend only when it is both numerically compatible and measurably faster. Disabled by default and currently validated on SM90 and SM12x. Explicit `--attention-backend`, component overrides, and model-required backends are never replaced. * `--component-attention-backends {MAP}`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa`. A key may carry a `.self` or `.cross` role suffix to override one kind of attention only, for example `transformer.cross=fa` * `--attention-backend-config {CONFIG}`: attention backend configuration * `--srt-encoder-url {HTTPADDRESS}`: address of SGLang srt server with AR model for GLM-Image like models. See [Models with AR Stage](/docs/sglang-diffusion/models_with_ar). * `--srt-encoder-timeout {SECONDS}`: Timeout in seconds for HTTP requests to the SGLang encoder server * `--srt-encoder-connection-timeout {SECONDS}`: TCP connection timeout in seconds for SGLang encoder server * `--scheduler-rpc-timeout {SECONDS}`: optional end-to-end deadline for an internal scheduler RPC, including scheduler queue time. It is unset by default so valid long-running and queued video jobs are not failed by the transport layer. Set it only when the deployment requires a bounded request deadline; caller cancellation and server shutdown remain effective without it. * `--enable-trace`: export OpenTelemetry traces for the diffusion scheduler and worker stages. * `--otlp-traces-endpoint {HOST:PORT}`: OTLP collector endpoint used with `--enable-trace` (default: `localhost:4317`). * `--otlp-service-name {NAME}`: `service.name` attached to exported traces. Resolution order is this flag, `OTEL_SERVICE_NAME`, then `sglang-diffusion`. * `--enable-metrics`: expose Prometheus metrics at `/metrics` (default: disabled). Includes request counts, queue time, host-side stage timing and LoRA state, separated by role and DP replica. See [Production metrics](/docs/sglang-diffusion/production_metrics) for metric semantics and disaggregated scraping. * `--pe-server-url {HTTPADDRESS}`: url of SGLang server hosting the PE model (e.g., for ERNIE-Image). See [Models with Prompt Enhancement](/docs/sglang-diffusion/models_with_pe). * `--prompt-enhancer-config {JSON_FILE}`: configure an independent SRT LLM/VLM for opt-in HTTP prompt rewriting. Requests use `enhance_prompt=true`; no enhancer runs by default. See [Prompt enhancement](/docs/sglang-diffusion/models_with_pe). ### Sampling and output * `--prompt {PROMPT}` and `--negative-prompt {PROMPT}` * `--image-path {PATH} [{PATH} ...]`: input image(s) for image-to-video or image-to-image generation * `--num-inference-steps {STEPS}` and `--seed {SEED}` * `--num-outputs-per-prompt {N}` / `--num-outputs {N}`: generate multiple outputs for each prompt. A scalar seed expands as `seed + output_index`. * `--quality {exact,lossless,high}`: cumulative request-level optimization tier. `lossless` (default) runs every fast path that keeps the reference math and the precision of every operand, moving only where the rounding happens; it does not enable quantization, sparsity, caching, or another approximate path. `exact` restricts it to the fast paths that reproduce the reference's rounding exactly, so output is bit-identical in the same environment -- ask for it when a run has to be reproducible. `high` includes the complete `lossless` set and may also enable model-owned approximate optimizations. `extra-high` is accepted as the former name of `lossless`. Separately configured quantization, attention, or caching options still apply. Support and validation constraints are model-specific. * `skip_softmax_params` (online request only): explicitly enables lossy BLASST/Skip-Softmax attention for compatible FA self-attention layers. See [Attention Backends](../attention_backends#request-scoped-skip-softmax). * `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}` * `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames` For frame interpolation and upscaling, see [Post-Processing](/docs/sglang-diffusion/api/post_processing). ### Quantization Component checkpoint paths are selected separately, so changing DiT precision never silently changes prompt embeddings. The component-scoped forms are the canonical interface for any component key from `model_index.json` or a native pipeline's registered module name: | Intent | Canonical option | Convenience alias | Behavior | | - | - | - | - | | Replace a component | `--component-paths. {MODEL}` | `---path {MODEL}` | Load the replacement component's configuration and weights | | Replace only its weights | `--component-weights-paths. {WEIGHTS}` | `---weights-path {WEIGHTS}` | Retain the base component configuration and replace its weights | | Select component precision | `--component-precisions. {DTYPE}` | — | Use a loader-supported exact parameter and execution dtype | | Direct-load an eligible component | `--component-direct-gpu-weight-loading.` | None | Use that component's audited direct-GPU loader; it must stay resident | | Quantize an unquantized component online | `--component-quantizations. {METHOD}` | `---quantization {METHOD}` | Apply a method supported by that component's native loader | | Keep selected component layers unquantized | `--component-quantization-ignored-layers. {PATTERN...}` | None | Pass component-local ignored-layer patterns to its online quantizer | For example, pair any replacement text-encoder configuration with a separate single-file checkpoint as follows: ```bash theme={null} --component-paths.text_encoder COMPONENT_REPO_OR_DIRECTORY \ --component-weights-paths.text_encoder WEIGHTS_FILE_OR_REPO_FILE ``` The transformer-specific `--transformer-weights-path` spelling remains supported for the primary DiT. Do not mechanically replace it with `--component-weights-paths.transformer`: the component-scoped form requires an actual component name, which is pipeline-specific. Use `--quantization` only to override the method inferred by the transformer loader, and `--quantization-ignored-layers` to keep matching transformer layers unquantized during online quantization. For a native text encoder: * `--component-paths.text_encoder {MODEL}` replaces the text-encoder checkpoint; `--text-encoder-path {MODEL}` is its shorter alias * `--component-quantizations.text_encoder {METHOD}` applies supported online quantization to an unquantized native encoder; pair it with `--component-quantization-ignored-layers.text_encoder {PATTERN...}` when selected layers must remain unquantized * Quantization metadata is auto-detected from that checkpoint. The native loader accepts compatible serialized formats without a model-name allowlist and rejects implementations that do not construct the required quantized layers. The same contract applies to every weighted component: path routing is generic, while quantized materialization is capability-based. Native auxiliary loaders whose current materializer expects plain state dicts reject unsupported quantization metadata before model construction. See [Quantized Component Repositories](/docs/sglang-diffusion/quantization#quantized-component-repositories) for the current component matrix. A model cookbook is the source of truth for published, model-specific checkpoint examples; for example, all H3 sources and their exact overlays are kept in one [MiniMax-H3 compatibility table](/cookbook/diffusion/MiniMax/MiniMax-H3#checkpoint-and-adapter-formats). Exact precision overrides are capability-based. Native text and image encoders, standard VAE components, and native plain-state components support them; other component loaders reject the option instead of accepting a dtype that their execution stage would not honor. Direct-GPU loading is also capability-based. The existing `--direct-gpu-weight-loading` remains the primary DiT path. The component form currently supports standard native `vae` and `video_vae` state dicts on CUDA: it streams each safetensors tensor directly to the resident module, rather than materializing a complete CPU state dict. It rejects custom Diffusers `auto_map` classes, quantized checkpoints, tied state entries, and any component-offload or layerwise-offload placement. Component overrides accept a local component directory, a standalone Hub repository, or a Hub component subfolder written as `owner/repo/subfolder`. For transformer and native encoder loaders, an explicit weight filename keeps the base component configuration and replaces only its weights; the selected file format must still be supported by that loader. For supported realtime causal video models, `--kv-cache-quant {off|int4|int2}` compresses completed KV-cache chunks independently of transformer weight quantization. It is lossy and disabled by default. See [Realtime and Causal Video Models](/docs/sglang-diffusion/realtime_models) for the runtime and model scope, and [Quantization](/docs/sglang-diffusion/quantization) for supported quantization families and examples. ### Request logging * `--log-requests`: Log user-facing fields of all requests (default: `False`). The verbosity is decided by `--log-requests-level`. * `--log-requests-level {0|1|2|3}`: Verbosity level for request logging (default: `2`). 0: Log metadata (request id). 1: Log metadata and sampling config (seed, steps, guidance, resolution, frames, fps, ...). 2: Log metadata, sampling config and prompt (truncated to 2 KiB). 3: Log metadata, sampling config and full prompt. * `--log-requests-format {text|json}`: Format for request logging (default: `text`). `text` is human-readable; `json` outputs structured JSON lines. * `--log-requests-target {TARGET...}`: Target(s) for request logging. Use `stdout` for console output and/or directory path(s) for file output. Can specify multiple targets, e.g., `--log-requests-target stdout /my/log/dir`. ## Configuration Files Use `--config` to load JSON or YAML configuration. Command-line flags override values from the config file. ```bash Command theme={null} sglang generate --config config.yaml ``` Example: ```yaml Config theme={null} model_path: FastVideo/FastHunyuan-diffusers prompt: A beautiful woman in a red dress walking down a street output_path: outputs/ num_gpus: 2 sp_degree: 2 tp_size: 1 num_frames: 45 height: 720 width: 1280 num_inference_steps: 6 seed: 1024 fps: 24 precision: bf16 vae_precision: fp16 vae_tiling: true vae_sp: true enable_torch_compile: false ``` HunyuanVideo and FastHunyuan use tiled VAE decode by default so multi-GPU runs distribute VAE tiles instead of selecting spatial-shard decode. At HunyuanVideo's supported 960×544×77 shape, spatial-shard decode can consume 99.8 GiB per rank before requesting another 49.61 GiB causal mask. At FastHunyuan's default 1280×720×125 shape, the mask alone would require 197.75 GiB. You can still override the policy with `--vae-config.parallel-decode-mode`, but `spatial` and `spatial_shard` should only be used for smaller validated shapes. ## Generate `sglang generate` runs a single generation job and exits when the job finishes. ```bash Command theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --text-encoder-cpu-offload \ --pin-cpu-memory \ --num-gpus 4 \ --ulysses-degree 2 \ --ring-degree 2 \ --prompt "A curious raccoon" \ --save-output \ --output-path outputs \ --output-file-name "a-curious-raccoon.mp4" ``` HTTP server-only arguments are ignored by `sglang generate`. For supported native pipelines, set `SGLANG_CACHE_DIT_ENABLED=true` to enable Cache-DiT. It can run with DiT layerwise offload; it cannot run with FSDP. For the diffusers backend, use `--backend diffusers --cache-dit-config ...`. See [Cache-DiT](/docs/sglang-diffusion/cache_dit). Native conditioning reuse is enabled by default, independently of Cache-DiT: * `--disable-conditioning-cache`: disable cross-request text/image conditioning reuse and VAE posterior caching. Reuse within a grouped stage remains enabled. * `--conditioning-cache-max-size-mb {MiB}`: shared CPU and device LRU capacity per executor/rank, default `512`; `0` disables cross-request caching. Consumed negative text conditioning stays on its original device; other entries live on the CPU. The cache holds at most 128 entries and bypasses oversized outputs. See [conditioning cache](/docs/sglang-diffusion/caching-acceleration#conditioning-cache) for model coverage, invalidation, and the distinction between cold and repeated-input latency. For supported image pipelines, breakable CUDA graph can be enabled with `--enable-breakable-cuda-graph`, but you must declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures. LongCat-Image is supported by this path. Its DiT always consumes a fixed 512-token prompt body, so different prompt lengths reuse the same graph without model-specific padding. For a 1024x1024 deployment, use: ```bash Command theme={null} sglang serve --model-path meituan-longcat/LongCat-Image \ --enable-breakable-cuda-graph \ --warmup-resolutions 1024x1024 \ --enable-torch-compile false \ --port 30010 ``` SANA-Video supports the same path for fixed-shape serving. Its default text stage emits a fixed 300-token prompt shape, so the runtime reuses one graph across prompt lengths without padding it to a generic text bucket. BCG also requires the serving frame count to match warmup; the example captures the model's default 81-frame signature: ```bash Command theme={null} sglang serve \ --model-path Efficient-Large-Model/SANA-Video_2B_480p_diffusers \ --enable-breakable-cuda-graph \ --warmup-resolutions 832x480 \ --enable-torch-compile false \ --port 30010 ``` ### Component Residency Use `--component-residency COMPONENT=MODE` to assign one runtime residency mode to each native pipeline component: ```bash Command theme={null} sglang generate \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --component-residency all=resident text_encoder=layerwise-offload vae=component-offload \ --prompt "A quiet city street after rain" ``` The available modes are: * `resident`: keep the complete component on the accelerator. * `component-offload`: keep the complete component on CPU between uses, moving it to the accelerator before each declared use and back to CPU afterward. * `snapshot-offload`: retain CPU parameter storage while a complete component runs on GPU. At the end of a use, restore the CPU parameters instead of copying GPU weights back. Mutable buffers still move back to CPU. * `layerwise-offload`: keep component weights on CPU and stream its declared layers during execution. Selectors match exact loaded component keys from `model_index.json`, including names such as `transformer_2`, `audio_vae`, and `connectors`. The group selectors `dit`, `text_encoder`, `image_encoder`, and `vae` are also available, together with `all`. An exact key overrides a matching group, and a group overrides `all`. Components without a matching canonical selector retain their explicit legacy setting or automatic/model default. The existing `--dit-cpu-offload`, `--text-encoder-cpu-offload`, `--image-encoder-cpu-offload`, `--vae-cpu-offload`, and `--cpu-offload-components` options remain supported. New and legacy options may be mixed: `--component-residency` wins only for components it matches, while unmatched legacy settings remain effective. Legacy layerwise selectors take precedence over legacy component-offload selectors for the same component. Explicit `--dit-layerwise-offload false` makes the DiT resident unless another explicit DiT selector, such as `--dit-cpu-offload true` or `--component-residency dit=component-offload`, selects a different mode. Explicit component placement also applies during warmup and request cleanup. Pipeline preferences cannot keep or reload an explicitly offloaded component merely to prepare for the next request. An explicitly `resident` component remains resident. This applies to matching exact, group, and `all` selectors, as well as legacy offload options. Components without an explicit placement may still use the pipeline's warmup retention and request-end preload preferences. Selecting one component does not disable those preferences for other components. On-demand loading and prefetch for upcoming uses within a request are unchanged; explicit offload may therefore add weight-transfer latency to the first real request after warmup. Layerwise selection is strict. A native weighted component selected for `layerwise-offload` must declare its layer structure; otherwise startup fails with the unsupported component name instead of silently changing modes. Explicit non-resident placement also requires a request-time component-use declaration, so it cannot silently select a module that the pipeline does not manage. FSDP applies only to resident components. The Diffusers backend supports only pipeline-wide `all=resident` and `all=component-offload`. #### Snapshot offload On NVIDIA CUDA GPUs with separate host and device memory, explicitly select `--component-residency vae=snapshot-offload` to avoid repeated weight D2H transfers. Existing defaults and `component-offload` behavior are unchanged. Snapshot offload retains a full host copy during GPU execution. With `--pin-cpu-memory` enabled (the default), it shares a per-worker host pin budget with layerwise offload. Layerwise initialization claims its allowance first; snapshot weights use the remainder. Shared parameter storage is pinned once and reused across requests; its allowance is returned when the storage is released. Pinning also checks current host headroom, including the cgroup limit. Weights that do not fit the budget, or have pinning disabled, retain their existing CPU storage. Checkpoint mmap stays file-backed unless pinning, dtype conversion, or weight mutation materializes a copy. Pageable H2D can be slower, and pinning temporarily needs space for both the original and pinned storage. This budget limits new pinned weight allocations, not total process RAM or the CUDA host allocator's cache. Benchmark repeated requests with the actual host storage and workload; avoiding D2H does not guarantee an end-to-end speedup. The whole component must still fit on GPU. Unlike layerwise offload with all layers retained across denoising steps, snapshot offload does not require a layer-structure declaration and also covers parameters outside transformer blocks. Weight updates, LoRA merge/unmerge, and sleep restore the CPU weights before mutation or release. FSDP-managed components, shared-memory GPUs, and snapshot-offloaded DiTs with breakable CUDA graphs are not supported. ### Layerwise Offload Tuning Use layerwise offload when a component does not fit comfortably in GPU memory. The compatibility options `--dit-layerwise-offload` and `--layerwise-offload-components` remain available (`--layerwise-offload-modules` is an alias), while new deployments can select the mode directly: ```bash Command theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --component-residency transformer=layerwise-offload text_encoder=layerwise-offload \ --dit-offload-prefetch-size 0 \ --prompt "A quiet city street after rain" ``` Values passed to the compatibility option `--layerwise-offload-components` must match loaded component keys, such as `transformer`, `text_encoder`, `image_encoder`, `vae`, `condition_image_encoder`, `spatial_upsampler`, or `vocoder`. Its `default` group selects text encoders, image encoders, and VAEs. Use `all` to select every layerwise-offloadable component. Layerwise tuning options such as `--dit-offload-prefetch-size`, `--dit-layerwise-resident-layers`, `--dit-layerwise-residency-policy`, and `--dit-layerwise-residency-lifetime` continue to control the streamed layer working set. Prefer the smallest component set that solves the memory issue because layerwise offload can increase latency. DiT layerwise offload can run with Cache-DiT: skipped blocks are not streamed, and the first layer after a skip may sync-load. Cache-DiT remains incompatible with FSDP. Those four set the default for every streamed component. To give one component its own values, use the `component=value` forms, which also accept JSON: ```bash theme={null} sglang serve --model-path \ --layerwise-offload-components dit,text_encoder \ --layerwise-prefetch-size text_encoder=2 \ --layerwise-resident-layers transformer=4 \ --layerwise-residency-policy transformer=strided \ --layerwise-residency-lifetime transformer=permanent ``` The `component=value` maps take the loaded component's name — `transformer`, `text_encoder`, `video_vae` — not a group alias: `--layerwise-offload-components` understands `dit`, but `--layerwise-resident-layers dit=4` matches no component and is silently ignored. * `--layerwise-prefetch-size`: how many layers to fetch ahead. Fractional values are a share of the stack, `>= 1` an absolute count. Deeper prefetch overlaps more of the transfer with compute, at the cost of staging buffers. * `--layerwise-resident-layers`: how many layers stay on the GPU instead of being streamed. Fractional values are a share of the stack. Resident layers pay for a component that runs its layers many times per request — a DiT across the denoise steps, a video VAE across the latent chunks — and do nothing for one that runs them once per request, such as a text encoder. * `--layerwise-residency-policy`: `leading` keeps the first layers, `strided` spreads them across the stack so the transfers do not arrive as one burst. * `--layerwise-residency-lifetime`: how long the resident layers stay. `forward` (default) places them when the component starts running its layers for a request and releases them when it finishes, so they are transferred once per request and keep a copy in host memory. `permanent` places them at load time, keeps no host copy, and never releases them: one transfer for the life of the server. A component without an entry keeps the group default, so adding these options changes nothing until one is set. All of these trade VRAM for transfer. The lifetime decides *whose* VRAM: with `forward`, only the component that is running holds its resident layers, so the peak is the largest single component's set; with `permanent`, every component's resident layers are on the GPU at once, so the peak is their sum. Pick `permanent` when that sum fits alongside the activations — it also frees the pinned host memory the copies would have held — and `forward` when it does not. Measure before raising any of them. ## Serve `sglang serve` starts the HTTP server and keeps the model loaded for repeated requests. ```bash Command theme={null} sglang serve \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --text-encoder-cpu-offload \ --pin-cpu-memory \ --num-gpus 4 \ --ulysses-degree 2 \ --ring-degree 2 \ --port 30010 ``` ### Asynchronous output saving Use `--async-output-save` to overlap output saving with the next request during native NVIDIA CUDA serving. It is disabled by default. See [Asynchronous output saving](/docs/sglang-diffusion/performance-optimization#asynchronous-output-saving) for an example, eligible requests, and the response latency and memory tradeoffs. ### Health endpoints SGLang Diffusion separates process liveness from inference readiness: | Endpoint | Success condition | Recommended use | | - | - | - | | `GET /liveness` | The HTTP server is accepting requests. It remains `200` during server warmup. | Kubernetes liveness probe | | `GET /health` | The server is ready for normal inference traffic. It returns `503` while server-based synthetic warmup is running and `200` after it completes. | Startup and readiness probes | | `GET /health_generate` | Compatibility alias for `/health`. It does not currently issue a generation request in SGLang Diffusion. | Existing integrations only | `/health` gates only server-based warmup. With `--warmup-mode off` or `--warmup-mode request`, it returns `200` once the HTTP server starts; those modes do not promise that compilation or other first-request work has completed. If server-based warmup fails, the server terminates instead of reporting ready. Do not use `/health` as a liveness probe: a long server warmup can legitimately keep it at `503` for several minutes. ### Cloud Storage SGLang Diffusion can upload generated images and videos to S3-compatible object storage after generation. ```bash Command theme={null} export SGLANG_CLOUD_STORAGE_TYPE=s3 export SGLANG_S3_BUCKET_NAME=my-bucket export SGLANG_S3_ACCESS_KEY_ID=your-access-key export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key export SGLANG_S3_ENDPOINT_URL=https://minio.example.com ``` See [Environment Variables](/docs/sglang-diffusion/environment_variables) for the full set of storage options. ## Component Path Overrides Override individual pipeline components such as `vae`, `transformer`, or `text_encoder` with `---path`. ```bash Command theme={null} sglang serve \ --model-path black-forest-labs/FLUX.2-dev \ --vae-path fal/FLUX.2-Tiny-AutoEncoder ``` The component key must match a key in the model's `model_index.json` or a native pipeline's registered module name. The path must be either a Hugging Face repo ID or a complete component directory. Path selection and quantized checkpoint support are separate capabilities. A pre-quantized component repo must carry its quantization metadata, and the selected loader must support that serialized format. Native plain-state loaders fail closed; library-managed components inherit the corresponding Transformers or Diffusers support. The transformer-specific `--quantization` flag does not select the format of component checkpoints; their own metadata does. See [Quantized Component Repositories](/docs/sglang-diffusion/quantization#quantized-component-repositories). ## Component Attention Backend Overrides Use `--component-attention-backends` when one pipeline component needs a different native attention backend from the global `--attention-backend`. ```bash Command theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --attention-backend fa \ --component-attention-backends text_encoder=torch_sdpa ``` The component key must match a pipeline module key such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`. Component overrides take precedence over the global `--attention-backend` while that component is being constructed and fail if the component cannot satisfy them. A native component may explicitly defer backend selection until first use; components with fixed attention reject the override. Sparse self-attention backends use a compatible dense backend for cross-attention layers. The global backend remains strict for DiT components, while auxiliary components may fall back to a compatible backend. The Diffusers backend supports only the global backend passthrough. You can also pass dotted CLI entries: ```bash Command theme={null} sglang generate \ --model-path \ --component-attention-backends.text_encoder torch_sdpa \ --component-attention-backends.transformer fa ``` A component key may end in `.self` or `.cross` to override one kind of attention only. This helps when a backend's per-call overhead pays off for self-attention but not for the lighter cross-attention: ```bash Command theme={null} sglang generate \ --model-path \ --component-attention-backends.transformer.self sage_attn \ --component-attention-backends.transformer.cross fa ``` A role override outranks the component-wide backend for layers of that role and is equally strict: it fails the load if it never reaches a layer of that role. See [Attention Backends](/docs/sglang-diffusion/attention_backends#per-role-overrides). ## Diffusers Backend Use `--backend diffusers` to force vanilla diffusers pipelines when no native SGLang implementation exists or when a model requires a custom pipeline class. ### Key Options
Argument Values Description
--backend auto, sglang, diffusers Choose native SGLang, force native, or force diffusers
--attention-backend flash, \_flash\_3\_hub, sage, xformers, native Attention backend for diffusers pipelines
--trust-remote-code flag Required for models with custom pipeline classes
--vae-tiling and --vae-slicing flag Lower memory usage for VAE decode
--dit-precision and --vae-precision fp16, bf16, fp32 Precision controls
--enable-torch-compile flag Enable torch.compile
--cache-dit-config Cache-DiT config for diffusers pipelines
### Example ```bash theme={null} sglang generate \ --model-path AIDC-AI/Ovis-Image-7B \ --backend diffusers \ --trust-remote-code \ --attention-backend flash \ --prompt "A serene Japanese garden with cherry blossoms" \ --height 1024 \ --width 1024 \ --num-inference-steps 30 \ --save-output \ --output-path outputs \ --output-file-name ovis_garden.png ``` For pipeline-specific arguments not exposed in the CLI, pass `diffusers_kwargs` in a config file. # OpenAI API Source: https://docs.sglang.io/docs/sglang-diffusion/api/openai_api Image and video generation endpoints with LoRA adapter management. The SGLang diffusion HTTP server implements an OpenAI-compatible API for image and video generation, as well as LoRA adapter management. ## Prerequisites * Python 3.11+ if you plan to use the OpenAI Python SDK. ## Serve Launch the server using the `sglang serve` command. ### Start the server ```bash theme={null} SERVER_ARGS=( --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers --served-model-name wan-t2v --text-encoder-cpu-offload --pin-cpu-memory --num-gpus 4 --ulysses-degree=2 --ring-degree=2 --port 30010 ) sglang serve "${SERVER_ARGS[@]}" ``` * **--model-path**: Path to the model or model ID. * **--served-model-name**: Stable model name exposed by the serving APIs. It defaults to `--model-id` when set, otherwise `--model-path`. * **--port**: HTTP port to listen on (default: `30000`). ### Served model name `--served-model-name` separates the public API identity from the checkpoint location. This is useful when replicas use different local mount paths or when a gateway needs one stable model name: ```bash theme={null} sglang serve \ --model-path /models/Wan2.1-T2V-1.3B-Diffusers \ --served-model-name wan-t2v \ --port 30010 ``` `--model-id` is not a free-form deployment alias: it selects the registered model configuration for checkpoints whose local path cannot be identified. `--served-model-name` only controls the name exposed by serving APIs. When both are set, the served name takes precedence for API responses. ### Discover the served model **Endpoint:** `GET /v1/models` Returns the public model name together with diffusion-specific runtime information. **Curl Example:** ```bash curl theme={null} curl -sS "http://localhost:30010/v1/models" ``` **Response Example:** ```json theme={null} { "object": "list", "data": [ { "id": "wan-t2v", "object": "model", "created": 1786348800, "owned_by": "sglang", "root": "wan-t2v", "parent": null, "max_model_len": null, "num_gpus": 4, "task_type": "T2V", "supported_task_types": ["T2V"], "dit_precision": "bf16", "vae_precision": "fp16", "pipeline_name": "WanPipeline", "pipeline_class": "WanPipeline" } ] } ``` Retrieve the same model by its served name: ```bash curl theme={null} curl -sS "http://localhost:30010/v1/models/wan-t2v" ``` `GET /server_info` also reports `served_model_name` for gateway discovery. Video and action responses use this name when the request does not provide a model explicitly. *** ## Endpoints ### Select a request task `GET /v1/models` reports `task_type` (the configured default) and `supported_task_types` (the tasks implemented by that configured pipeline). `GET /model_info` also reports the supported `output_types`. A pipeline may support image and video output on one server when its implementation and loaded components provide both paths. Send the optional SGLang extension `task_type` as a case-insensitive enum name: | Interface | Task field | Output constraint | | - | - | - | | `POST /v1/images/generations` JSON | `"task_type": "t2i"` | Image | | `POST /v1/images/edits` multipart | `-F task_type=i2i` | Image | | `POST /v1/videos` JSON or multipart | `"task_type": "i2v"` or `-F task_type=i2v` | Video | | OpenAI Python client | `extra_body={"task_type": "t2i"}` | Selected endpoint | | Native Python `generate` sampling parameters | `"task_type": "t2i"` | Selected task | | `sglang generate` | `--task-type t2i` | Selected task | Choose only a task in the server's advertised set. An unsupported task, a task with the wrong output type for the endpoint, or missing required conditioning returns HTTP 400. For example, `i2v` needs image input, while `v2v` and `f2v` need video input. Model-specific size, frame count, and reference constraints still apply. If you omit `task_type`, the server uses the endpoint and supplied inputs to choose a compatible task, preferring its default when compatible. It asks for an explicit task if several alternatives remain (for example `v2v` versus `f2v`). Pipelines with one task keep that task as their default. Sending a request never changes the server default or subsequent requests. ### Prompt enhancement Image generation, image edits and video creation accept `enhance_prompt=true` when the server is configured with `--prompt-enhancer-config`. An independent SRT service rewrites the prompt before diffusion runs. The default is `false`; image responses return `data[].revised_prompt`, and video responses return `revised_prompt`. See [deployment and model compatibility](/docs/sglang-diffusion/models_with_pe). ### Model-specific request fields The image and video request schemas contain only OpenAI fields and stable SGLang extensions shared across model families. Put model-specific controls in `extra_body` when using the OpenAI Python client: ```python Python theme={null} from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:30010/v1") response = client.images.generate( model="ideogram-ai/ideogram-4-nf4", prompt="A typeset poster", extra_body={"preset": "V4_QUALITY_48"}, ) ``` Raw JSON requests may send these fields at the top level. Multipart video requests may send individual declared form fields or one JSON object in `extra_params`. SGLang forwards model-specific extensions only when the active model declares them, so a field for one model family does not silently change another model. ### Image Generation The server implements an OpenAI-compatible Images API under the `/v1/images` namespace. **Create an image** **Endpoint:** `POST /v1/images/generations` #### Request quality `quality` selects a cumulative request-level optimization tier: `exact` keeps the selected deployment's reference path and all unconditional bit-exact replacements, so output is bit-identical to it in the same environment; `lossless` additionally enables the request-gated DiT/VAE fusions that keep the reference math and every operand's precision and only move the rounding; `high` includes the full `lossless` set and may also enable model-owned sparse, caching, lower-precision, or other approximate paths. `extra-high` is accepted as the former name of `lossless`. The tier does not override separately configured quantization, attention, or caching options. Omit it (or send OpenAI's default `auto`) to keep the `lossless` runtime default. It is distinct from `output_quality`, which controls only output-file compression. The same extension is accepted by image edits and video requests. **Python Example (b64\_json response):** ```python Python theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1") img = client.images.generate( prompt="A calico cat playing a piano on stage", size="1024x1024", n=1, response_format="b64_json", ) image_bytes = base64.b64decode(img.data[0].b64_json) with open("output.png", "wb") as f: f.write(image_bytes) ``` **Curl Example:** ```bash curl theme={null} curl -sS -X POST "http://localhost:30010/v1/images/generations" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer sk-proj-1234567890" \ -d '{ "prompt": "A calico cat playing a piano on stage", "size": "1024x1024", "n": 1, "response_format": "b64_json" }' ``` > **Note** > If `response_format=url` is used and cloud storage is not configured, the API returns > a relative URL like `/v1/images//content`. **Edit an image** **Endpoint:** `POST /v1/images/edits` This endpoint accepts a multipart form upload with input images and a text prompt. The server can return either a base64-encoded image or a URL to download the image. Uploaded filenames are sanitized, and saved uploads are constrained to the server's upload directory. This also applies to image uploads for video and mesh generation. **Curl Example (b64\_json response):** ```bash Command theme={null} curl -sS -X POST "http://localhost:30010/v1/images/edits" \ -H "Authorization: Bearer sk-proj-1234567890" \ -F "image=@local_input_image.png" \ -F "url=image_url.jpg" \ -F "prompt=A calico cat playing a piano on stage" \ -F "size=1024x1024" \ -F "response_format=b64_json" ``` **Curl Example (URL response):** ```bash Command theme={null} curl -sS -X POST "http://localhost:30010/v1/images/edits" \ -H "Authorization: Bearer sk-proj-1234567890" \ -F "image=@local_input_image.png" \ -F "url=image_url.jpg" \ -F "prompt=A calico cat playing a piano on stage" \ -F "size=1024x1024" \ -F "response_format=url" ``` **Download image content** When `response_format=url` is used with `POST /v1/images/generations` or `POST /v1/images/edits`, the API returns a relative URL like `/v1/images//content`. **Endpoint:** `GET /v1/images/{image_id}/content` **Curl Example:** ```bash theme={null} curl -sS -L "http://localhost:30010/v1/images//content" \ -H "Authorization: Bearer sk-proj-1234567890" \ -o output.png ``` ### Video Generation The server implements a subset of the OpenAI Videos API under the `/v1/videos` namespace. **Create a video (text-to-video)** **Endpoint:** `POST /v1/videos` **Python Example:** ```python Python theme={null} from openai import OpenAI client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1") video = client.videos.create( prompt="A calico cat playing a piano on stage", size="1280x720" ) print(f"Video ID: {video.id}, Status: {video.status}") ``` **Curl Example:** ```bash curl theme={null} curl -sS -X POST "http://localhost:30010/v1/videos" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer sk-proj-1234567890" \ -d '{ "prompt": "A calico cat playing a piano on stage", "size": "1280x720" }' ``` **Create a video (image-to-video)** For I2V or TI2V models (e.g., Wan2.1 I2V, LTX-2.3 two-stage), pass an input image via multipart form upload or a reference URL. **Curl Example (multipart form upload):** ```bash Command theme={null} curl -sS -X POST "http://localhost:30010/v1/videos" \ -H "Authorization: Bearer sk-proj-1234567890" \ -F "prompt=A cat playing a piano" \ -F "input_reference=@input_image.png" \ -F "size=1280x720" ``` **Curl Example (reference URL):** ```bash Command theme={null} curl -sS -X POST "http://localhost:30010/v1/videos" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer sk-proj-1234567890" \ -d '{ "prompt": "A cat playing a piano", "reference_url": "https://example.com/input_image.png", "size": "1280x720" }' ``` `size` is the literal output resolution for most pipelines. Wan-Animate-2 treats it as a pixel-area budget instead: the output keeps the reference image's aspect ratio and uses the largest frame with at most `width × height` pixels whose sides are multiples of 16, so `size=640x800` with a 9:16 reference image yields a 528×944 video. Request a budget with the reference image's aspect ratio to get a specific size. Wan-Animate-2 also ignores `seconds` and `num_frames`: the output has as many frames as the (resampled) reference video, generated in chunks of `clip_len` frames; see the [Wan-Animate-2 cookbook](/cookbook/diffusion/Wan/Wan-Animate-2#1-2-defaults). Wan-Animate-2 keeps the reference video's audio track on the output; the model-specific `enable_audio` field (form field or `extra_body`, default `true`) set to `false` gives a silent output. #### MP4 encoding `x264_preset` picks the libx264 preset for the returned MP4: `ultrafast`, `superfast`, `veryfast`, `faster`, `fast` (the default), `medium`, `slow`, `slower`, `veryslow` or `placebo`. Faster presets finish encoding sooner and trade file size and fidelity at the same `output_compression`/`output_quality`. Send it at the top level of a JSON body or as a multipart form field; `sglang generate` takes `--x264-preset`. **List videos** **Endpoint:** `GET /v1/videos` **Python Example:** ```python Python theme={null} videos = client.videos.list() for item in videos.data: print(item.id, item.status) ``` **Curl Example:** ```bash curl theme={null} curl -sS -X GET "http://localhost:30010/v1/videos" \ -H "Authorization: Bearer sk-proj-1234567890" ``` **Download video content** **Endpoint:** `GET /v1/videos/{video_id}/content` **Python Example:** ```python Python theme={null} import time # Poll for completion while True: page = client.videos.list() item = next((v for v in page.data if v.id == video_id), None) if item and item.status == "completed": break time.sleep(5) # Download content resp = client.videos.download_content(video_id=video_id) with open("output.mp4", "wb") as f: f.write(resp.read()) ``` **Curl Example:** ```bash curl theme={null} curl -sS -L "http://localhost:30010/v1/videos//content" \ -H "Authorization: Bearer sk-proj-1234567890" \ -o output.mp4 ``` *** ### LoRA Management The server supports dynamic loading, merging, and unmerging of LoRA adapters. **Important Notes:** * Mutual Exclusion: Only one LoRA configuration can be active per target at a time * Switching: To switch LoRAs, deactivate the current LoRA with `unmerge_lora_weights`, then `set` the new one * Caching: The server caches loaded LoRA weights in memory. Switching back to a previously loaded LoRA (same path) has little cost **Set LoRA Adapter** Loads one or more LoRA adapters and applies them to the model. By default, regular weights are statically merged, while FSDP-sharded weights use dynamic LoRA to avoid full-gather memory peaks. **Endpoint:** `POST /v1/set_lora` **Parameters:** * `lora_nickname` (string or list of strings, required): A unique identifier for the LoRA adapter(s). Can be a single string or a list of strings for multiple LoRAs * `lora_path` (string or list of strings/None, optional): Path to the `.safetensors` file(s) or Hugging Face repo ID(s). Required for the first load; optional if re-activating a cached nickname. If a list, must match the length of `lora_nickname` * `target` (string or list of strings, optional): Which transformer(s) to apply the LoRA to. If a list, must match the length of `lora_nickname`. Valid values: * `"all"` (default): Apply to all transformers * `"transformer"`: Apply only to the primary transformer (high noise for Wan2.2) * `"transformer_2"`: Apply only to transformer\_2 (low noise for Wan2.2) * `"critic"`: Apply only to the critic model * `strength` (float or list of floats, optional): LoRA strength for merge, default 1.0. If a list, must match the length of `lora_nickname`. Values \< 1.0 reduce the effect, values > 1.0 amplify the effect * `merge_mode` (string, optional): `"auto"` (default server policy), `"merge"` (force static merge), or `"dynamic"` (apply LoRA at forward time) **Single LoRA Example:** ```bash Command theme={null} curl -X POST http://localhost:30010/v1/set_lora \ -H "Content-Type: application/json" \ -d '{ "lora_nickname": "lora_name", "lora_path": "/path/to/lora.safetensors", "target": "all", "strength": 0.8 }' ``` **Multiple LoRA Example:** ```bash Command theme={null} curl -X POST http://localhost:30010/v1/set_lora \ -H "Content-Type: application/json" \ -d '{ "lora_nickname": ["lora_1", "lora_2"], "lora_path": ["/path/to/lora1.safetensors", "/path/to/lora2.safetensors"], "target": ["transformer", "transformer_2"], "strength": [0.8, 1.0] }' ``` **Multiple LoRA with Same Target:** ```bash Command theme={null} curl -X POST http://localhost:30010/v1/set_lora \ -H "Content-Type: application/json" \ -d '{ "lora_nickname": ["style_lora", "character_lora"], "lora_path": ["/path/to/style.safetensors", "/path/to/character.safetensors"], "target": "all", "strength": [0.7, 0.9] }' ``` When using multiple LoRAs: * All list parameters (`lora_nickname`, `lora_path`, `target`, `strength`) must have the same length * If `target` or `strength` is a single value, it will be applied to all LoRAs * Multiple LoRAs applied to the same target are applied in order **Merge LoRA Weights** Manually merges the currently set LoRA weights into the base model. With FSDP-sharded weights, manual merge may require a full-gather and can OOM. Use `set_lora` with `merge_mode="auto"` or `"dynamic"` for the lower-peak path. **Endpoint:** `POST /v1/merge_lora_weights` **Parameters:** * `target` (string, optional): Which transformer(s) to merge. One of "all" (default), "transformer", "transformer\_2", "critic" * `strength` (float, optional): LoRA strength for merge, default 1.0. Values \< 1.0 reduce the effect, values > 1.0 amplify the effect **Curl Example:** ```bash theme={null} curl -X POST http://localhost:30010/v1/merge_lora_weights \ -H "Content-Type: application/json" \ -d '{"strength": 0.8}' ``` **Unmerge LoRA Weights** Unmerges the currently active LoRA weights from the base model, restoring it to its original state. This **must** be called before setting a different LoRA. **Endpoint:** `POST /v1/unmerge_lora_weights` **Curl Example:** ```bash theme={null} curl -X POST http://localhost:30010/v1/unmerge_lora_weights \ -H "Content-Type: application/json" ``` **List LoRA Adapters** Returns loaded LoRA adapters and current application status per module. **Endpoint:** `GET /v1/list_loras` **Curl Example:** ```bash theme={null} curl -sS -X GET "http://localhost:30010/v1/list_loras" ``` **Response Example:** ```json theme={null} { "loaded_adapters": [ { "nickname": "lora_a", "path": "/weights/lora_a.safetensors" }, { "nickname": "lora_b", "path": "/weights/lora_b.safetensors" } ], "active": { "transformer": [ { "nickname": "lora2", "path": "tarn59/pixel_art_style_lora_z_image_turbo", "merged": true, "mode": "merged", "strength": 1.0 } ] } } ``` Notes: * If LoRA is not enabled for the current pipeline, the server will return an error. * `num_lora_layers_with_weights` counts only layers that have LoRA weights applied for the active adapter. ### Example: Switching LoRAs 1. Set LoRA A: ```bash Command theme={null} curl -X POST http://localhost:30010/v1/set_lora -d '{"lora_nickname": "lora_a", "lora_path": "path/to/A"}' ``` 2. Generate with LoRA A... 3. Unmerge LoRA A: ```bash Command theme={null} curl -X POST http://localhost:30010/v1/unmerge_lora_weights ``` 4. Set LoRA B: ```bash Command theme={null} curl -X POST http://localhost:30010/v1/set_lora -d '{"lora_nickname": "lora_b", "lora_path": "path/to/B"}' ``` 5. Generate with LoRA B... ### Adjust Output Quality The server supports adjusting output quality and compression levels for both image and video generation through the `output-quality` and `output-compression` parameters. #### Parameters * **`output-quality`** (string, optional): Preset quality level that automatically sets compression. **Default is `"default"`**. Valid values: * `"maximum"`: Highest quality (100) * `"high"`: High quality (90) * `"medium"`: Medium quality (55) * `"low"`: Lower quality (35) * `"default"`: Auto-adjust based on media type (50 for video, 75 for image) * **`output-compression`** (integer, optional): Direct compression level override (0-100). **Default is `None`**. When provided (not `None`), takes precedence over `output-quality`. * `0`: Lowest quality, smallest file size * `100`: Highest quality, largest file size #### Notes * **Precedence**: When both `output-quality` and `output-compression` are provided, `output-compression` takes precedence * **Format Support**: Quality settings apply to JPEG, and video formats. PNG uses lossless compression and ignores these settings * **File Size vs Quality**: Lower compression values (or "low" quality preset) produce smaller files but may show visible artifacts # Post-Processing Source: https://docs.sglang.io/docs/sglang-diffusion/api/post_processing SGLang diffusion supports optional post-processing steps that run after generation to improve temporal smoothness (frame interpolation) or spatial resolution (upscaling). These steps are independent of the diffusion model and can be combined in a single run. When both are enabled, **frame interpolation runs first** (increasing the frame count), then **upscaling runs on every frame** (increasing the spatial resolution). *** ## Frame Interpolation (video only) Frame interpolation synthesizes new frames between each pair of consecutive generated frames, producing smoother motion without re-running the diffusion model. The `--frame-interpolation-exp` flag controls how many rounds of interpolation to apply: each round inserts one new frame into every gap between adjacent frames, so the output frame count follows the formula: > **(N − 1) × 2^exp + 1** > > e.g. 5 original frames with `exp=1` → 4 gaps × 1 new frame + 5 originals = **9** frames; > with `exp=2` → **17** frames. ### CLI Arguments
Argument Description
--enable-frame-interpolation Enable frame interpolation. Model weights are downloaded automatically on first use.
--frame-interpolation-exp \{EXP} Interpolation exponent — 1 = 2× temporal resolution, 2 = 4×, etc. (default: 1)
--frame-interpolation-scale \{SCALE} RIFE inference scale; use 0.5 for high-resolution inputs to save memory (default: 1.0)
--frame-interpolation-model-path \{PATH} Local directory or HuggingFace repo ID containing RIFE flownet.pkl weights (default: elfgum/RIFE-4.22.lite, downloaded automatically)
### Supported Models Frame interpolation uses the [RIFE](https://github.com/hzwer/Practical-RIFE) (Real-Time Intermediate Flow Estimation) architecture. Only **RIFE 4.22.lite** (`IFNet` with 4-scale `IFBlock` backbone) is supported. The network topology is hard-coded, so custom weights provided via `--frame-interpolation-model-path` must be a `flownet.pkl` checkpoint that is compatible with this architecture. Other RIFE versions (e.g., older `v4.x` variants with different block counts) or entirely different frame interpolation methods (FILM, AMT, etc.) are **not supported**.
Weight HuggingFace Repo Description
RIFE 4.22.lite *(default)* elfgum/RIFE-4.22.lite Lightweight model, downloaded automatically on first use
### Example Generate a 5-frame video and interpolate to 9 frames ((5 − 1) × 2¹ + 1 = 9): ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --prompt "A dog running through a park" \ --num-frames 5 \ --enable-frame-interpolation \ --frame-interpolation-exp 1 \ --save-output ``` *** ## Upscaling (image and video) Upscaling increases the spatial resolution of generated images or video frames using [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN). The model weights are downloaded automatically on first use and cached for subsequent runs. ### CLI Arguments
Argument Description
--enable-upscaling Enable post-generation upscaling using Real-ESRGAN.
--upscaling-scale \{SCALE} Desired upscaling factor (default: 4). The 4× model is used internally; if a different scale is requested, a bicubic resize is applied after the network output.
--upscaling-model-path \{PATH} Local .pth file, HuggingFace repo ID, or repo\_id:filename for Real-ESRGAN weights (default: ai-forever/Real-ESRGAN with RealESRGAN\_x4.pth, downloaded automatically). Use the repo\_id:filename format to specify a custom weight file from a HuggingFace repo (e.g. my-org/my-esrgan:weights.pth).
### Supported Models Upscaling supports two Real-ESRGAN network architectures. The correct architecture is **auto-detected** from the checkpoint keys, so you only need to point `--upscaling-model-path` at a valid `.pth` file:
Architecture Example Weights Description
RRDBNet RealESRGAN\_x4plus.pth Heavier model with higher quality; best for photos
SRVGGNetCompact RealESRGAN\_x4.pth *(default)*, realesr-animevideov3.pth, realesr-general-x4v3.pth Lightweight model; faster inference, good for video
The default weight file is [`ai-forever/Real-ESRGAN`](https://huggingface.co/ai-forever/Real-ESRGAN) with `RealESRGAN_x4.pth` (SRVGGNetCompact, 4× native scale). Other super-resolution models (e.g., SwinIR, HAT, BSRGAN) are **not supported** — only Real-ESRGAN checkpoints using the two architectures above are compatible. ### Examples Generate a 1024×1024 image and upscale to 4096×4096: ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.2-dev \ --prompt "A cat sitting on a windowsill" \ --output-size 1024x1024 \ --enable-upscaling \ --save-output ``` Generate a video and upscale each frame by 4×: ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --prompt "A curious raccoon" \ --enable-upscaling \ --upscaling-scale 4 \ --save-output ``` *** ## Combining Frame Interpolation and Upscaling Frame interpolation and upscaling can be combined in a single run. Interpolation is applied first (increasing the frame count), then upscaling is applied to every frame (increasing the spatial resolution). Example — generate 5 frames, interpolate to 9 frames, and upscale each frame by 4×: ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --prompt "A curious raccoon" \ --num-frames 5 \ --enable-frame-interpolation \ --frame-interpolation-exp 1 \ --enable-upscaling \ --upscaling-scale 4 \ --save-output ``` # Attention Backends Source: https://docs.sglang.io/docs/sglang-diffusion/attention_backends Select and configure attention backends for SGLang diffusion pipelines. This document describes the attention backends available in sglang diffusion (`sglang.multimodal_gen`) and how to select them. ## Overview Attention backends are defined by `AttentionBackendEnum` (`sglang.multimodal_gen.runtime.platforms.interface.AttentionBackendEnum`) and selected via the CLI flag `--attention-backend`. Backend selection is performed by the shared attention layers (e.g. `LocalAttention` / `USPAttention` / `UlyssesAttention` in `sglang.multimodal_gen.runtime.layers.attention.layer`). `--attention-backend` is strict for the diffusion transformer / DiT. Auxiliary components such as encoders and VAEs use it when compatible, then fall back to a component default or a platform-compatible backend. Use `--component-attention-backends` when an auxiliary component must use a specific backend; incompatible overrides, including an override for a component that constructs no SGLang attention layer, fail unless a sparse backend is being replaced for cross-attention. A component key may carry a `.self` or `.cross` role suffix to override just one kind of attention; see [Per-role overrides](#per-role-overrides). When using the diffusers backend, `--attention-backend` is passed through to diffusers' `set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`). * **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA. On SM100/B200, dense non-causal fp16/bf16 native attention prefers cuDNN SDPA and falls back to FA4 if cuDNN has no compatible kernel. * **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA. * **Intel XPU**: uses XPU Flash Attention backend (fp16/bf16, head sizes 64/96/128/192/256); otherwise falls back to PyTorch SDPA. * **MUSA**: uses FlashAttention when available; also supports Sage Attention when installed; otherwise falls back to PyTorch SDPA. * **MPS**: always uses PyTorch SDPA. * **NPU**: for ring attention uses FA otherwise uses PyTorch SDPA. ## Backend options For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBackendEnum`. The table below lists the backends implemented by the built-in platforms. `fa3`/`fa4` are accepted as aliases for `fa`.
CLI value Enum value Notes
`fa` / `fa3` / `fa4` `FA` FlashAttention. fa3/fa4 are normalized to fa during argument parsing (ServerArgs.**post\_init**).
`torch_sdpa` `TORCH_SDPA` PyTorch scaled\_dot\_product\_attention.
`sliding_tile_attn` `SLIDING_TILE_ATTN` Sliding Tile Attention (STA). Requires st\_attn. Configure via --attention-backend-config.
`sage_attn` `SAGE_ATTN` Requires sageattention. On Hopper (SM90), PyPI sageattention==2.2.0 is unsupported because it lacks the upstream SM90 binding fix. Install pip install --force-reinstall git+[https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4](https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4) --no-build-isolation. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120; see upstream setup.py.
`sage_attn_3` `SAGE_ATTN_3` Requires SageAttention3 installed per upstream instructions.
`sparge_attn` `SPARGE_ATTN` Training-free sparse SageAttention2. CUDA SM80/86/87/89/90, FP16/BF16, head dim 64/128, and square self-attention with sequence length at least 128. Other attention shapes use dense SDPA. Install pip install git+[https://github.com/thu-ml/SpargeAttn.git](https://github.com/thu-ml/SpargeAttn.git) --no-build-isolation; configure retained block ratio with topk.
`sol_attn` `SOL_ATTN` Requires the upstream sol-attn package. Install with pip install git+[https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse\_backends](https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends). BF16, head dim 128. Configure via --attention-backend-config.
`video_sparse_attn` `VIDEO_SPARSE_ATTN` Requires vsa. Configure sparsity via --attention-backend-config.
`subblock_sparse_attn` `SUBBLOCK_SPARSE_ATTN` Training-free SubBlock sparsity for MiniMax-H3 on CUDA SM90, SM100, and SM120. The default compute mode is BF16; compute\_mode=sage\_fp8 is an approximate SM90-only path that requires SpargeAttn. See the MiniMax-H3 recipe.
`video_sparse_attn_h3` `VIDEO_SPARSE_ATTN_H3` Video Sparse Attention for MiniMax-H3 / FastH3 (VSA-H3). In-tree kernels, no external package: FastVideo's native tcgen05 block-sparse forward on SM100 / SM103, the Triton tile-64 kernel on SM90. Configure via --attention-backend-config.
`hybrid_window_attn_h3` `HYBRID_WINDOW_ATTN_H3` VDN-H3 hybrid attention: chunk-aligned window softmax (exact, gated) plus the Video Delta linear branch over the window complement. The window runs as a union of FlashAttention varlen calls (FA4 on SM100 / SM103 / SM120, FA3 on SM90; on SM80 / SM86 / SM89 the same FA3 build runs its Sm80 mainloop at FA2-class speed). Configure via --attention-backend-config.
`vmoba_attn` `VMOBA_ATTN` Requires kernel.attn.vmoba\_attn.vmoba. Configure via --attention-backend-config.
`aiter` `AITER` Requires aiter.
aiter\_sage AITER\_SAGE Requires aiter.
sla\_attn SLA\_ATTN Sparse Linear Attention. Requires SpargeAttn. Install with pip install git+[https://github.com/thu-ml/SpargeAttn.git](https://github.com/thu-ml/SpargeAttn.git) --no-build-isolation.
sage\_sla\_attn SAGE\_SLA\_ATTN SageAttention + Sparse Linear Attention. Requires SpargeAttn (same install as SLA).
`sparse_video_gen_2_attn` `SPARSE_VIDEO_GEN_2_ATTN` Requires svg. See installation instructions at [https://github.com/svg-project/Sparse-VideoGen](https://github.com/svg-project/Sparse-VideoGen).
laser\_attn LASER\_ATTN Requires attentions which can be installed with sgl\_kernel\_npu; available only for NPU.
block\_sparse\_attn BLOCK\_SPARSE\_ATTN Requires attentions which can be installed with sgl\_kernel\_npu; available only for NPU.
rain\_fusion\_attn RAIN\_FUSION\_ATTN Requires attentions which can be installed with sgl\_kernel\_npu; available only for NPU.
fp8\_fa\_sm120 FP8\_FA\_SM120 FP8 (E4M3) dense attention for SM120 GPUs (GeForce RTX 50, RTX PRO Blackwell), written in CuTe-DSL. Q/K/V are quantized per head on the fly; the output stays BF16. Non-causal, batch 1, head dim 128, BF16 inputs; other calls use torch\_cudnn\_sdpa. Opt-in only: outputs differ from BF16 attention by about 5% relative RMS per call. Each new sequence length compiles once (about 10 s).
## Selection priority The selection order in `runtime/layers/attention/selector.py` is: 1. `global_force_attn_backend(...)` / `global_force_attn_backend_context_manager(...)` 2. Component *and role* override from `--component-attention-backends..` while that component is being constructed 3. Component override from `--component-attention-backends` while that component is being constructed 4. CLI `--attention-backend` (`ServerArgs.attention_backend`) 5. Layer or component default, when declared 6. Auto selection (platform capability, dtype, and installed packages) An explicit global backend mismatch fails for DiT self-attention. Auxiliary components may fall back to their declared default or another compatible backend. Sparse backends selected for self-attention similarly fall back to a compatible dense backend for cross-attention. Explicit component overrides are otherwise strict. ### Per-role overrides A role is the kind of attention a layer performs: `self` or `cross`. Appending a role to a component key selects a backend for just that role, which is useful when a quantized or otherwise expensive backend pays off for self-attention but not for the much lighter cross-attention: ```bash Command theme={null} sglang generate \ --model-path \ --component-attention-backends.transformer.self sage_attn \ --component-attention-backends.transformer.cross fa ``` A role override outranks the component-wide backend for layers of that role and leaves the other role alone. Layers of a role with no override fall through to the component-wide backend as usual, and `_2` inherits ``'s role overrides unless it declares its own. Role overrides are as strict as component-wide ones: a role override that never reaches a layer fails the load, including when the component builds no layers of that role at all. The one tolerated divergence is the shared sparse rule above — a sparse backend named for `cross` is replaced by a compatible dense backend and logged, rather than failing. Model and layer backend sets guide automatic selection; they are not positive allowlists for explicit requests. A backend selected with `--attention-backend` or `--component-attention-backends` can therefore be used without first adding it to every model's set. It still fails closed when the platform cannot resolve it or when it lacks a required operation such as dense cross-attention, packed variable-length attention, or ring-attention LSE. Model paths that require one backend for correctness declare it as required; those layers keep that backend even when the surrounding component is overridden. For an automatic, measurement-based choice, pass `--enable-attention-backend-autotune true`. On each layer's first sufficiently large input, the native runtime times compatible candidates and switches only when the output remains within its numerical guard and the measured gain exceeds the noise margin. The tuner is disabled by default and currently validated on SM90 and SM12x. It does not replace a backend selected explicitly through `--attention-backend` or `--component-attention-backends`, or a backend required by the model for correctness. ## Configuration Some backends require additional configuration. You can pass these parameters via `--attention-backend-config`. This argument accepts: * A path to a JSON or YAML configuration file. * A JSON string (e.g., `'{"sparsity": 0.5}'`). * Key-value pairs (e.g., `"sparsity=0.5,enable_x=true"`). ### Supported Configuration Parameters **Sliding Tile Attention (`sliding_tile_attn`)**
Parameter Type Description Default
`mask_strategy_file_path` `str` **Required.** Path to the mask strategy JSON file. -
`sta_mode` `str` Mode of STA. `STA_inference`
`skip_time_steps` `int` Number of steps to use full attention before switching to sparse attention. `15`
**Video Sparse Attention (`video_sparse_attn`)**
Parameter Type Description Default
`sparsity` `float` Validation sparsity (0.0 - 1.0). `0.0`
**SpargeAttention (`sparge_attn`)** | Parameter | Type | Description | Default | | - | - | - | - | | `topk` | `float` | Fraction of predicted attention blocks retained per head. Higher values preserve more attention work and generally improve quality. Must be in `(0, 1]`. | `0.5` | SpargeAttention is approximate even when `topk=1`: the recommended upstream kernel quantizes attention through SageAttention2. Validate output quality and end-to-end latency on the target model and resolution before deployment. **Video Sparse Attention for H3 (`video_sparse_attn_h3`)**
Parameter Type Description Default
`VSA_sparsity` `float` Fraction of video tiles excluded from the top-k selection (0.0 - 1.0). FastH3 defaults to its trained `0.8`. `0.8` (FastH3), `0.9` (base H3)
`vsa_mode` `str` `exempt`: non-video keys (text/audio prefix tiles) are always selected. `compete`: they compete with video tiles in the top-k. `exempt`
`vsa_dense_first_n_steps` `int` Use dense attention for the first N denoising steps. `0`
`vsa_dense_layers` `list[int]` Layer indices kept dense, e.g. `[0, 1]`. `[]`
`vsa_tile_size` `int` Kernel tile size. Only `64` (the trained (4, 4, 4) geometry) is accepted. `64`
VSA-H3 constraints: * Only the DiT runs sparse; the token refiner, text encoder, and VAEs keep their dense defaults. An explicit `--component-attention-backends text_encoder=fa` is rejected because the H3 text encoder has SDPA-only layers. * Uses the checkpoint's trained `to_gate_compress` compression branch. Base MiniMax-H3 weights load zero gates and run pure sparse. * Ulysses sequence parallelism is supported; `--ring-degree` greater than 1, `torch.compile`, and breakable CUDA graph execution are rejected. **Hybrid window attention for VDN-H3 (`hybrid_window_attn_h3`)** [VDN-H3](/cookbook/diffusion/MiniMax/MiniMax-H3#7-vdn-h3-hybrid-attention-8-step-distill) replaces every DiT block's dense self-attention with two branches. The softmax branch is an exact softmax over a chunk-aligned frame window (frame `t` belongs to chunk `t // 5` and attends to chunks `c - 1 .. c + 1`; frames 0 and F-1 are dense as rows and columns; every text / audio pair stays dense) scaled by a per-head sigmoid gate. The linear branch (a frame-wise Video Delta rule recurrence, forward and reverse over frames) covers exactly the window's complement and is driven by the attention module. The mask is request-static, so the metadata is built once per request. The checkpoint's `transformer/config.json` carries the window geometry; the backend reads it.
Parameter Type Description Default
`vdn_h3_dense_smoke` `bool` Allows a dense transformer backend (`fa`) on the VDN-H3 weights for the base-H3 + LoRA equivalence smoke. The gates and the linear branch are skipped: a study, not a sample. `false`
`vdn_max_gather_rows` `int` Upper bound on the gathered K/V rows per window FlashAttention call; consecutive chunk groups fill one call up to it. Splitting changes no query's kept key set, only the pass count and the gather's peak memory. `200000`
Hybrid window attention constraints: * VDN-H3 requires this backend for the transformer; a dense backend on these weights would silently skip the linear branch and the gates, so it is rejected unless `vdn_h3_dense_smoke` is set. The token refiner, text encoder, and VAEs keep dense attention. * Base MiniMax-H3 and FastH3 checkpoints have no linear branch and are rejected by this backend. * Ulysses sequence parallelism is supported (QK-norm + RoPE run after the all-to-all on the head shard); `--ring-degree` greater than 1, `torch.compile`, and breakable CUDA graph execution are rejected. **V-MoBA (`vmoba_attn`)**
Parameter Type Description Default
`temporal_chunk_size` `int` Chunk size for temporal dimension. -
`temporal_topk` `int` Top-K tokens to select in temporal dimension. -
`spatial_chunk_size` `list[int]` Chunk size for spatial dimension (H, W). -
`spatial_topk` `int` Top-K tokens to select in spatial dimension. -
`st_chunk_size` `list[int]` Chunk size for spatiotemporal dimension (T, H, W). -
`st_topk` `int` Top-K tokens to select in spatiotemporal dimension. -
`moba_select_mode` `str` Selection mode (e.g., `threshold`). `threshold`
`moba_threshold` `float` Threshold value for selection. `0.25`
`moba_threshold_type` `str` Type of thresholding (e.g., `query_head`). `query_head`
`first_full_step` `int` Number of initial steps to use full attention. `12`
`first_full_layer` `int` Number of initial layers to use full attention. `0`
`temporal_layer` `int` Number of temporal layers. `1`
`spatial_layer` `int` Number of spatial layers. `1`
`st_layer` `int` Number of spatiotemporal layers. `1`
**Block Sparse Attention (`block_sparse_attn`)**
Parameter Type Description Default
`skip_first_steps` `int` Number of steps to use laser attention before switching to sparse attention. `10`
`sparsity` `float` The sparsity coefficient must be in the range (0, 1). `0.2`
**Sol-Attn (`sol_attn`)**
Parameter Type Description Default
`tau` `float` Routing threshold scale. Higher values select fewer exact KV blocks. `1.0`
`thresh_type` `str` Threshold mode: `diag` or `exact`. `diag`
`sink_tokens` `int` Exact KV sink length for prefix tokens such as text/audio rows. `0`
`sink_start` `int` Start index of the exact KV sink range. `0`
`dense_steps` `int` Use dense attention for the first N denoising steps. `10`
`dense_layers` `str` Layer indices kept dense, e.g. `0,1` or `0-2`. `0,1`
`dense_backend` `str` Backend used for the dense prefix: `fa` (default) or `sage_attn`. `sage_attn` is approximate. `fa`
`kv_splits` `int | str` KV split factor passed to the Sol-Attn kernel. Use `auto` on long sequences. `auto`
## Platform support matrix
Backend CUDA ROCm XPU MUSA MPS NPU Notes
`fa` Yes Yes ✅ ✅ ❌ ✅ CUDA requires SM80+ and fp16/bf16. XPU uses its own flash attention backend. FlashAttention is only used when the required runtime is installed; otherwise it falls back to torch\_sdpa. No extra installations are required for NPU
`torch_sdpa` Yes Yes Yes Yes ✅ ✅ Most compatible option across platforms.
`sliding_tile_attn` Yes No No No ❌ ❌ CUDA-only. Requires st\_attn. Configure via --attention-backend-config.
`sage_attn` Yes No No Yes ❌ ❌ Optional dependency on CUDA and MUSA. On Hopper, also falls back to FlashAttention when the installed package lacks the SM90 binding fix.
`sage_attn_3` Yes No No No ❌ ❌ CUDA-only (optional dependency).
`sparge_attn` Yes No No No ❌ ❌ CUDA SM80/86/87/89/90 only. Requires SpargeAttn; head dim 64/128 and square self-attention with sequence length at least 128.
`sol_attn` Yes No No No ❌ ❌ CUDA-only. Requires sol-attn. Install with pip install git+[https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse\_backends](https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends). Configure via --attention-backend-config.
`video_sparse_attn` Yes No No No ❌ ❌ CUDA-only. Requires vsa. Configure sparsity via --attention-backend-config.
`video_sparse_attn_h3` Yes No No No ❌ ❌ CUDA-only (SM90 / SM100 / SM103). In-tree kernels (native tcgen05 forward on Blackwell, Triton on Hopper), no external dependency. Configure via --attention-backend-config.
sla\_attn Yes No No No ❌ ❌ CUDA-only. Requires SpargeAttn.
sage\_sla\_attn Yes No No No ❌ ❌ CUDA-only. Requires SpargeAttn.
vmoba\_attn Yes No No No ❌ ❌ CUDA-only. Requires kernel.attn.vmoba\_attn.vmoba. Configure via --attention-backend-config.
aiter ❌ ✅ No No ❌ ❌ Requires aiter.
aiter\_sage ❌ ✅ No No ❌ ❌ Requires aiter.
`sparse_video_gen_2_attn` Yes No No No ❌ ❌ CUDA-only. Requires svg.
laser\_attn ❌ ❌ ❌ ❌ ❌ ✅ NPU-only. Requires attentions from sgl\_kernel\_npu. Uses SDPA if seqlen less than 2048.
block\_sparse\_attn ❌ ❌ ❌ ❌ ❌ ✅ NPU-only. Requires attentions from sgl\_kernel\_npu. Configuration via --attention-backend-config.
rain\_fusion\_attn ❌ ❌ ❌ ❌ ❌ ✅ NPU-only. Requires attentions from sgl\_kernel\_npu Configuration via --attention-backend-config.
fp8\_fa\_sm120 Yes ❌ ❌ ❌ ❌ ❌ CUDA SM120 only; falls back to torch\_cudnn\_sdpa on other devices and for unsupported calls.
## Usage ### Select a backend via CLI ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend fa ``` ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend torch_sdpa ``` ### Override one component Use component overrides when a specific module needs different attention semantics from the main transformer: ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend fa \ --component-attention-backends text_encoder=torch_sdpa ``` Component keys match pipeline module names from `model_index.json`, such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`. Use this override when the fallback must be pinned: unlike the global backend, an incompatible component override raises an error instead of selecting another backend. The one role-based exception is a sparse self-attention backend, which uses a compatible dense backend for cross-attention layers in the same component. The component must construct SGLang-selectable attention or explicitly defer selection until first use; components with fixed attention reject the override. Per-component overrides apply only to native pipelines. The Diffusers backend accepts the global `--attention-backend` passthrough instead. ### Per-request override (denoise loop) A single server can serve exact and approximate attention side by side: requests may switch the DiT denoise attention backend via the `attention_backend_override` sampling param. Valid values are the exact/drop-in dense kernels — `fa`, `torch_sdpa`, `sage_attn`, `sage_attn_3`. The field participates in the dynamic-batch signature, so requests with different backends never share a batch. ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend-override sage_attn ``` ```python theme={null} client.images.generate( model="", prompt="...", extra_body={"attention_backend_override": "sage_attn"}, ) ``` Incompatible server settings **reject the request** (with a server log) instead of silently falling back: breakable CUDA graphs and `torch.compile` bake the attention kernel into a captured/traced graph; sparse server backends (`sliding_tile_attn`, `video_sparse_attn`, ...) cannot be mixed with per-request dense switching; under ring parallelism the target must be ring-capable. Note `sage_attn` / `sage_attn_3` are lossy (quantized attention) — validate quality on your workload. ### Request-scoped Skip Softmax Skip Softmax (BLASST) keeps the QK matmul, but skips the exponential, softmax-state update, V load, and PV matmul for attention tiles whose estimated softmax mass is below a threshold. It is an explicit lossy optimization backed by the FlashInfer kernels shipped with SGLang's pinned dependencies. See the [BLASST paper](https://arxiv.org/abs/2512.12087) and NVIDIA's [video-generation study](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog28_Accelerating_Video_Generation_with_GEMM_Quantization_Attention_Quantization_and_Skip_Softmax_Attention_in_TensorRT-LLM.md) for the algorithm and measured quality/performance trade-offs. Pass `skip_softmax_params` in one image or video request. The runtime routes compatible self-attention layers through the FA/FlashInfer path for that batch; cross-attention keeps its normal backend, and the next request restores the server default. ```python theme={null} client.videos.create( model="", prompt="...", extra_body={ "skip_softmax_params": { "threshold_scale_factor": 500.0, "start_step": 14, } }, ) ``` * `threshold_scale_factor` is required and must be positive. The kernel uses `threshold_scale_factor / context_length`; larger values skip more work and usually increase quality loss. * `start_step` is the zero-based denoising step at which sparse execution starts. It defaults to `0`; keeping early high-noise steps dense is generally safer. There is intentionally no default threshold. Calibrate the threshold and start step against output-quality metrics for each model, resolution, step count, and deployment backend; values tuned for one workload are not portable. Current support is FP16/BF16, head dimension 128 or 256, and unmasked self-attention on Hopper SM90 and Blackwell SM100/SM103/SM107. Ulysses sequence parallelism is supported because the kernel runs after its all-to-all. Ring Attention, attention masks, `torch.compile`, and breakable CUDA graphs reject the request instead of silently running dense attention. Models that merely offer an FA backend do not automatically qualify: the runtime still checks the GPU, dtype, head dimension, attention role, and execution mode. ### Using SpargeAttention Install the optional CUDA extension, then select the backend explicitly: ```bash theme={null} pip install ninja pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation sglang generate \ --model-path \ --prompt "..." \ --attention-backend sparge_attn \ --attention-backend-config topk=0.5 ``` `sparge_attn` applies to compatible self-attention layers. Cross-attention uses the existing dense fallback; short self-attention and asymmetric Q/KV also use dense SDPA. LTX-2, LTX-2.3, and LTX-2.5 have compatible video (128) and audio (64) head dimensions, so their sufficiently long, unmasked self-attention uses SpargeAttention while prompt and audio-video cross-attention remain dense. Ulysses sequence parallelism is supported, but ring attention is not because the upstream kernel does not expose the softmax LSE needed for ring merging. K/V-gather SP therefore follows the normal sparse-backend rule and uses Ulysses instead. ### Sage then Sol hybrid `sol_attn` keeps the first `dense_steps` steps dense. Set `dense_backend=sage_attn` to run that prefix on SageAttention and the tail on Sol sparse attention. Keep the text encoder on `torch_sdpa`: ```bash theme={null} sglang generate \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --attention-backend sol_attn \ --attention-backend-config dense_backend=sage_attn,dense_steps=10 \ --component-attention-backends text_encoder=torch_sdpa,transformer=sol_attn \ --prompt "A cat walking on a sunny beach, gentle waves." \ --save-output ``` Both `sage_attn` and `sol_attn` are approximate. The default DiT backend remains `fa`. ### Using Sliding Tile Attention (STA) ```bash theme={null} # Pass the mask strategy file path via config sglang generate \ --model-path \ --prompt "..." \ --attention-backend sliding_tile_attn \ --attention-backend-config "mask_strategy_file_path=/abs/path/to/mask_strategy.json" ``` ### Notes for ROCm / MPS * ROCm: use `--attention-backend torch_sdpa` or `fa` depending on what is available in your environment. On gfx1151 / Strix Halo (wave32), do **not** use `aiter` — prefer `torch_sdpa` (image sets `SGLANG_USE_AITER=0`). * MPS: the platform implementation always uses `torch_sdpa`. # Cache-DiT Acceleration Source: https://docs.sglang.io/docs/sglang-diffusion/cache_dit Configure Cache-DiT acceleration for diffusion inference. SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to **1.69x inference speedup** with minimal quality loss. ## Overview **Cache-DiT** uses intelligent caching strategies to skip redundant computation in the denoising loop: * **DBCache (Dual Block Cache)**: Dynamically decides when to cache transformer blocks based on residual differences * **TaylorSeer**: Uses Taylor expansion for calibration to optimize caching decisions * **DMD Calibrator**: An **exponential-basis** forecasting calibrator (Dynamic Mode Decomposition, not Distribution Matching Distillation) that serves as a drop-in alternative to TaylorSeer's polynomial basis; strongest on flow-matching models * **SCM (Step Computation Masking)**: Step-level caching control for additional speedup Cache-DiT also ships **SVDQuant** W4A4 (int4 / NVFP4) dynamic quantization, which can be combined with DBCache caching (see [Quantization](#quantization)). ## Basic Usage Cache-DiT is a **per-request** switch: each request decides whether to run cached or lossless, and requests with different Cache-DiT settings never share a batch. The `SGLANG_CACHE_DIT_*` environment variables remain available as server-wide defaults for requests that leave the switch unset. Enable it for a single generation: ```bash theme={null} sglang generate --model-path Qwen/Qwen-Image \ --prompt "A beautiful sunset over the mountains" \ --enable-cache-dit true ``` Or per request against a running server, via the OpenAI-compatible API: ```python theme={null} client.images.generate( model="Qwen/Qwen-Image", prompt="A beautiful sunset over the mountains", extra_body={ "enable_cache_dit": True, # optional knob overrides for this request only "cache_dit_params": {"residual_diff_threshold": 0.12, "scm_preset": "fast"}, }, ) ``` `enable_cache_dit` accepts three states: `true` (on for this request), `false` (off for this request, overriding the server default), and unset (follow the `SGLANG_CACHE_DIT_ENABLED` server default). `cache_dit_params` accepts the DBCache knobs (`Fn_compute_blocks`, `Bn_compute_blocks`, `max_warmup_steps`, `residual_diff_threshold`, `max_continuous_cached_steps`, `enable_taylorseer`, `taylorseer_order`), the DMD knobs (`enable_dmd`, `dmd_history`, `dmd_rank`, `dmd_ridge`, `dmd_svd_precision`; DMD and TaylorSeer are mutually exclusive calibrators and cannot be enabled together), the SCM knobs (`scm_preset`, `scm_compute_bins`, `scm_cache_bins`, `scm_policy`), and a nested `secondary` dict with the DBCache knobs for the second transformer of dual-DiT models (unset secondary keys inherit the request's primary values, then the `SGLANG_CACHE_DIT_SECONDARY_*` defaults). To make Cache-DiT the default for every request instead, export the environment variable when launching: ```bash theme={null} SGLANG_CACHE_DIT_ENABLED=true \ sglang serve --model-path Qwen/Qwen-Image ``` ## Diffusers Backend Cache-DiT supports loading acceleration configs from a custom YAML file. For diffusers pipelines (`diffusers` backend), pass the YAML/JSON path via `--cache-dit-config`. This flow requires cache-dit >= 1.2.0 (`cache_dit.load_configs`). ### Single GPU inference Define a `cache.yaml` file that contains: * DBCache + TaylorSeer ```yaml theme={null} cache_config: max_warmup_steps: 8 warmup_interval: 2 max_cached_steps: -1 max_continuous_cached_steps: 2 Fn_compute_blocks: 1 Bn_compute_blocks: 0 residual_diff_threshold: 0.12 enable_taylorseer: true taylorseer_order: 1 ``` Then apply the config with: ```bash theme={null} sglang generate \ --backend diffusers \ --model-path Qwen/Qwen-Image \ --cache-dit-config cache.yaml \ --prompt "A beautiful sunset over the mountains" ``` * DBCache + TaylorSeer + SCM (Step Computation Mask) ```yaml Config theme={null} cache_config: max_warmup_steps: 8 warmup_interval: 2 max_cached_steps: -1 max_continuous_cached_steps: 2 Fn_compute_blocks: 1 Bn_compute_blocks: 0 residual_diff_threshold: 0.12 enable_taylorseer: true taylorseer_order: 1 # Must set the num_inference_steps for SCM. The SCM will automatically # generate the steps computation mask based on the num_inference_steps. # Reference: https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#scm-steps-computation-masking num_inference_steps: 28 steps_computation_mask: fast ``` * DBCache + TaylorSeer + SCM (Step Computation Mask) + Cache CFG ```yaml Config theme={null} cache_config: max_warmup_steps: 8 warmup_interval: 2 max_cached_steps: -1 max_continuous_cached_steps: 2 Fn_compute_blocks: 1 Bn_compute_blocks: 0 residual_diff_threshold: 0.12 enable_taylorseer: true taylorseer_order: 1 num_inference_steps: 28 steps_computation_mask: fast enable_sperate_cfg: true # e.g, Qwen-Image, Wan, Chroma, Ovis-Image, etc. ``` * DBCache + DMD Calibrator Instead of TaylorSeer, you can use the DMD calibrator: an **exponential-basis** forecasting calibrator that serves as a drop-in alternative to TaylorSeer's polynomial basis. DMD models the cached feature stream as a linear dynamical system (`Y_{t+1} ~= A @ Y_t`), forecasts cached features from the fitted eigen-modes, and stays accurate over longer cache skips where polynomial extrapolation diverges. DMD here refers to Dynamic Mode Decomposition (Schmid 2010\), **not** Distribution Matching Distillation. DMD works best on flow-matching models (e.g., FLUX), while TaylorSeer is often better on DDPM-style models — try both. DMD and TaylorSeer are mutually exclusive — enable only one of `enable_dmd` / `enable_taylorseer`: ```yaml Config theme={null} cache_config: max_warmup_steps: 8 warmup_interval: 2 max_cached_steps: -1 max_continuous_cached_steps: 2 Fn_compute_blocks: 1 Bn_compute_blocks: 0 # Bn=0 since the DMD calibrator replaces the Bn calibrator residual_diff_threshold: 0.12 enable_dmd: true dmd_history: 6 # snapshot window length, 5-6 typical dmd_svd_precision: "medium" # "low", "medium" or "high" ``` A `dmd_history` window of 5–6 snapshots is typically the sweet spot — longer histories do not always help, because the feature dynamics drift across timesteps. With fewer than 4 uniformly spaced snapshots available, DMD transparently falls back to the Taylor expansion it maintains internally. See the [Cache-DiT DMD documentation](https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#dmd-calibrator-dynamic-mode-decomposition) for the mathematical principle and quantitative comparisons. A ready-made config is available at [examples/configs/cache\_dmd.yaml](https://github.com/vipshop/cache-dit/blob/main/examples/configs/cache_dmd.yaml) in the Cache-DiT repository. Apply it with the same `--cache-dit-config` flag: ```bash theme={null} sglang generate \ --backend diffusers \ --model-path Qwen/Qwen-Image \ --cache-dit-config cache_dmd.yaml \ --prompt "A beautiful sunset over the mountains" ``` ### Distributed inference * 1D Parallelism Define a parallelism only config yaml `parallel.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: auto attention_backend: native ``` Then, apply the distributed inference acceleration config from yaml. `ulysses_size: auto` means that cache-dit will auto detect the `world_size` as the ulysses\_size. Otherwise, you should manually set it as specific int number, e.g, 4. Then apply the distributed config with: (Note: please add `--num-gpus N` to specify the number of gpus for distributed inference) ```bash theme={null} sglang generate \ --backend diffusers \ --num-gpus 4 \ --model-path Qwen/Qwen-Image \ --cache-dit-config parallel.yaml \ --prompt "A futuristic cityscape at sunset" ``` * 2D Parallelism You can also define a 2D parallelism config yaml `parallel_2d.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: auto tp_size: 2 attention_backend: native ``` Then, apply the 2D parallelism config from yaml. Here `tp_size: 2` means using tensor parallelism with size 2. The `ulysses_size: auto` means that cache-dit will auto detect the `world_size // tp_size` as the ulysses\_size. * 3D Parallelism You can also define a 3D parallelism config yaml `parallel_3d.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: 2 ring_size: 2 tp_size: 2 attention_backend: native ``` Then, apply the 3D parallelism config from yaml. Here `ulysses_size: 2`, `ring_size: 2`, `tp_size: 2` means using ulysses parallelism with size 2, ring parallelism with size 2 and tensor parallelism with size 2. * Ulysses Anything Attention To enable Ulysses Anything Attention, you can define a parallelism config yaml `parallel_uaa.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: auto attention_backend: native ulysses_anything: true ``` * Ulysses FP8 Communication For device that don't have NVLink support, you can enable Ulysses FP8 Communication to further reduce the communication overhead. You can define a parallelism config yaml `parallel_fp8.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: auto attention_backend: native ulysses_float8: true ``` * Async Ulysses CP You can also enable async ulysses CP to overlap the communication and computation. Define a parallelism config yaml `parallel_async.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: auto attention_backend: native ulysses_async: true # Now, only support for FLUX.1, Qwen-Image, Ovis-Image and Z-Image. ``` Then, apply the config from yaml. Here `ulysses_async: true` means enabling async ulysses CP. * TE-P and VAE-P You can also specify the extra parallel modules in the yaml config. For example, define a parallelism config yaml `parallel_extra.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: auto attention_backend: native extra_parallel_modules: ["text_encoder", "vae"] ``` ### Hybrid Cache and Parallelism Define a hybrid cache and parallel acceleration config yaml `hybrid.yaml` file that contains: ```yaml Config theme={null} cache_config: max_warmup_steps: 8 warmup_interval: 2 max_cached_steps: -1 max_continuous_cached_steps: 2 Fn_compute_blocks: 1 Bn_compute_blocks: 0 residual_diff_threshold: 0.12 enable_taylorseer: true taylorseer_order: 1 parallelism_config: ulysses_size: auto attention_backend: native extra_parallel_modules: ["text_encoder", "vae"] ``` Then, apply the hybrid cache and parallel acceleration config from yaml. ```bash theme={null} sglang generate \ --backend diffusers \ --num-gpus 4 \ --model-path Qwen/Qwen-Image \ --cache-dit-config hybrid.yaml \ --prompt "A beautiful sunset over the mountains" ``` ### Attention Backend In some cases, users may want to only specify the attention backend without any other optimization configs. In this case, you can define a yaml file `attention.yaml` that only contains: ```yaml Config theme={null} attention_backend: "flash" # '_flash_3' for Hopper ``` ### Quantization You can also specify the quantization config in the yaml file, required `torchao>=0.16.0`. For example, define a yaml file `quantize.yaml` that contains: ```yaml Config theme={null} quantize_config: # quantization configuration for transformer modules # float8 (DQ), float8_weight_only, float8_blockwise, int8 (DQ), int8_weight_only, etc. quant_type: "float8" # layers to exclude from quantization (transformer). layers that contains any of the # keywords in the exclude_layers list will be excluded from quantization. This is useful # for some sensitive layers that are not robust to quantization, e.g., embedding layers. exclude_layers: - "embedder" - "embed" verbose: false # whether to print verbose logs during quantization ``` Then, apply the quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization. For example: ```bash Command theme={null} sglang generate \ --backend diffusers \ --model-path Qwen/Qwen-Image \ --warmup-mode request \ --cache-dit-config quantize.yaml \ --enable-torch-compile \ --dit-cpu-offload false \ --text-encoder-cpu-offload false \ --prompt "A beautiful sunset over the mountains" ``` #### SVDQuant (W4A4 int4 / NVFP4) SVDQuant is Cache-DiT's built-in W4A4 PTQ quantization (weights and activations in int4 or NVFP4, with smoothed low-rank branches). It can be freely combined with DBCache caching and the DMD calibrator for the largest speedups. ::::note SVDQuant requires a cache-dit build **with CUDA extension support** — a plain `pip install cache-dit` does NOT include it. Install one of: ```bash Command theme={null} # Option 1: prebuilt CUDA 13 wheel pip install cache-dit-cu13== --no-deps # Option 2: build from source with SVDQuant enabled git clone https://github.com/vipshop/cache-dit cd cache-dit export CUDA_HOME=/usr/local/cuda CACHE_DIT_BUILD_SVDQUANT=1 pip install ".[quantization]" --no-build-isolation ``` :::: Valid `quant_type` values are `svdq_int4_r{32,64,128,256}_dq` (int4 W4A4) and `svdq_nvfp4_r{32,64,128,256}_dq` (NVFP4 W4A4; requires a Blackwell GPU). Example config combining SVDQuant NVFP4 with DBCache + DMD (see [examples/configs/blackwell/cache\_dmd\_svdq.yaml](https://github.com/vipshop/cache-dit/blob/main/examples/configs/blackwell/cache_dmd_svdq.yaml)): ```yaml Config theme={null} cache_config: max_warmup_steps: 8 warmup_interval: 2 max_cached_steps: -1 max_continuous_cached_steps: 2 Fn_compute_blocks: 1 Bn_compute_blocks: 0 residual_diff_threshold: 0.12 enable_dmd: true dmd_history: 6 dmd_svd_precision: "medium" quantize_config: quant_type: "svdq_nvfp4_r128_dq" # nvfp4 for Blackwell; use svdq_int4_r128_dq for int4 svdq_kwargs: quantize_device: "cuda" fused_mlp: true exclude_layers: - "embedder" - "embed" verbose: false ``` For int4 W4A4 (pre-Blackwell GPUs), the same config with `quant_type: "svdq_int4_r128_dq"` is available at [examples/configs/cache\_dmd\_svdq.yaml](https://github.com/vipshop/cache-dit/blob/main/examples/configs/cache_dmd_svdq.yaml) (add `runtime_kernel: "v2"` to `svdq_kwargs`). Enable `torch.compile` for the best SVDQuant performance, and make sure `--warmup-steps` covers the compile warmup (use the same value as `--num-inference-steps`): ```bash Command theme={null} sglang generate \ --backend diffusers \ --model-path black-forest-labs/FLUX.1-dev \ --num-inference-steps=28 \ --warmup-mode request \ --warmup-steps 28 \ --cache-dit-config cache_dmd_svdq.yaml \ --enable-torch-compile \ --dit-cpu-offload false \ --text-encoder-cpu-offload false \ --prompt "A beautiful sunset over the mountains" ``` You can verify from the log that the quantization is active: `[Cache-DiT] SVDQuant Type: svdq_nvfp4_r128_dq, Rank: 128`. ### Combined Configs: Cache + Parallelism + Quantization You can also combine all the above configs together in a single yaml file `combined.yaml` that contains: ```yaml Config theme={null} cache_config: max_warmup_steps: 8 warmup_interval: 2 max_cached_steps: -1 max_continuous_cached_steps: 2 Fn_compute_blocks: 1 Bn_compute_blocks: 0 residual_diff_threshold: 0.12 enable_taylorseer: true taylorseer_order: 1 parallelism_config: ulysses_size: auto attention_backend: native extra_parallel_modules: ["text_encoder", "vae"] quantize_config: quant_type: "float8" exclude_layers: - "embedder" - "embed" verbose: false ``` Then, apply the combined cache, parallelism and quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization. ## Advanced Configuration ### DBCache Parameters DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
### TaylorSeer Configuration TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
### DMD Calibrator Configuration DMD (Dynamic Mode Decomposition, Schmid 2010 — **not** Distribution Matching Distillation) is an **exponential-basis** forecasting calibrator and a drop-in alternative to TaylorSeer's polynomial basis. At each full-compute step it records a snapshot of the computed features; at a cached step it identifies a linear propagator from the recent snapshot window (one economy SVD with rank truncation, then eigendecomposition) and forecasts the current features via eigenvalue powers — cheap to advance, and stable over longer cache skips where polynomial extrapolation diverges. It typically improves both speed and quality over pure DBCache. **DMD and TaylorSeer are mutually exclusive** (enabling both raises a `ValueError`); DMD is best for flow-matching models, TaylorSeer for DDPM-style ones. See the [Cache-DiT DMD documentation](https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#dmd-calibrator-dynamic-mode-decomposition) for details:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_DMD` false Enable the DMD calibrator
History `SGLANG_CACHE_DIT_DMD_HISTORY` 6 Snapshot window length; 5-6 typical. Needs >= 4 uniformly spaced snapshots, otherwise DMD falls back to TaylorSeer
Rank `SGLANG_CACHE_DIT_DMD_RANK` 0 SVD truncation rank; 0 = automatic (drop modes below 1e-4 of the leading singular value)
Ridge `SGLANG_CACHE_DIT_DMD_RIDGE` 1e-8 Tikhonov regularization added to the inverted singular values
SVD Precision `SGLANG_CACHE_DIT_DMD_SVD_PRECISION` medium SVD precision: "low", "medium" or "high"
Usage (SGLD backend, env-driven): ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_DMD=true \ sglang generate --model-path black-forest-labs/FLUX.1-dev \ --prompt "A curious raccoon in a forest" ``` On the diffusers backend, enable DMD from the yaml config instead (`enable_dmd: true` in `cache_config`, see [Diffusers Backend](#diffusers-backend)); DMD can also be set per request via `cache_dit_params: {"enable_dmd": true}`. ### Combined Configuration Example DBCache and TaylorSeer are complementary strategies that work together, you can configure both sets of parameters simultaneously: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang generate --model-path black-forest-labs/FLUX.1-dev \ --prompt "A curious raccoon in a forest" ``` ### SCM (Step Computation Masking) SCM provides step-level caching control for additional speedup. It decides which denoising steps to compute fully and which to use cached results. **SCM Presets** SCM is configured with presets:
Preset Compute Ratio Speed Quality
`none` 100% Baseline Best
`slow` \~75% \~1.3x High
`medium` \~50% \~2x Good
`fast` \~35% \~3x Acceptable
`ultra` \~25% \~4x Lower
**Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_SCM_PRESET=medium \ sglang generate --model-path Qwen/Qwen-Image \ --prompt "A futuristic cityscape at sunset" ``` **Custom SCM Bins** For fine-grained control over which steps to compute vs cache: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_SCM_COMPUTE_BINS="8,3,3,2,2" \ SGLANG_CACHE_DIT_SCM_CACHE_BINS="1,2,2,2,3" \ sglang generate --model-path Qwen/Qwen-Image \ --prompt "A futuristic cityscape at sunset" ``` **SCM Policy**
Policy Env Variable Description
`dynamic` `SGLANG_CACHE_DIT_SCM_POLICY=dynamic` Adaptive caching based on content (default)
`static` `SGLANG_CACHE_DIT_SCM_POLICY=static` Fixed caching pattern
## Environment Variables All Cache-DiT parameters can also be configured via environment variables, which act as the server-wide defaults for requests that don't set `enable_cache_dit` / `cache_dit_params`. See [Environment Variables](./environment_variables) for the complete list. ## Supported Models SGLang Diffusion x Cache-DiT supports almost all models originally supported in SGLang Diffusion:
Model Family Example Models
Wan Wan2.1, Wan2.2
Flux FLUX.1-dev, FLUX.2-dev
Z-Image Z-Image-Turbo
Qwen Qwen-Image, Qwen-Image-Edit, Qwen-Image 2.1
Hunyuan HunyuanVideo
MiniMax MiniMax-H3 (T2VA, FL2VA, and Ref2VA)
## Performance Tips 1. **Start with defaults**: The default parameters work well for most models 2. **Use TaylorSeer**: It typically improves both speed and quality 3. **Tune R threshold**: Lower values = better quality, higher values = faster 4. **SCM for extra speed**: Use `medium` preset for good speed/quality balance 5. **Warmup matters**: Higher warmup = more stable caching decisions ## Limitations * **SGLang-native pipelines**: Distributed Cache-DiT paths exist for supported pipelines. Hybrid SP+TP configurations add communication and cache coordination overhead, so validate them on the target model and hardware before using them as production defaults. * **DiT layerwise offload**: Compatible. Skipped blocks are not streamed, and the first layer after a skip may sync-load. Still incompatible with `--use-fsdp-inference`. * **SCM minimum steps**: SCM requires >= 8 inference steps to be effective. Some pipelines report `steps - 1` NFEs (for example MiniMax-H3 `num_inference_steps=8` is 7 NFEs), which trips the upstream `steps_mask` assertion; use at least 9 requested steps or custom bins. * **Model support**: The model must be registered in Cache-DiT's `BlockAdapterRegister` or have an SGLang custom block adapter. ## Troubleshooting ### SCM disabled for low step count For models with \< 8 inference steps (e.g., DMD distilled models), SCM will be automatically disabled. DBCache acceleration still works. ### SVDQuant unavailable or load failure SVDQuant cases raise `svdq_is_available() = False` or `undefined symbol: ... materialize_cow_storage ...` when the installed cache-dit has no CUDA extension, or the prebuilt wheel was compiled against an incompatible torch. Fix: reinstall from the `cache-dit-cu13` wheel matching your torch version, or build cache-dit from source with `CACHE_DIT_BUILD_SVDQUANT=1` (see [Quantization](#quantization)). Quick self-check: ```bash Command theme={null} python -c "from cache_dit.quantization.svdquant import svdq_is_available, svdq_get_load_error as e; print(svdq_is_available(), e())" ``` ## References * [Cache-DiT](https://github.com/vipshop/cache-dit) * [SGLang Diffusion](./performance-optimization) # Caching Acceleration Source: https://docs.sglang.io/docs/sglang-diffusion/caching-acceleration Compare caching acceleration strategies for diffusion models. SGLang can reuse exact conditioning across requests and cache intermediate denoising results. Conditioning reuse preserves the encoded values; denoising caches such as Cache-DiT, TeaCache, and Spectrum use approximations. ## Conditioning cache Native pipelines enable a conditioning cache by default. Repeated inputs can reuse text embeddings (including negative prompts), image or vision-language encoder outputs, and VAE posteriors. The shared LRU holds at most **512 MiB per executor/rank** across CPU and device entries, with at most 128 entries. The common text stage keeps consumed negative conditioning on its original device to avoid repeated CPU-to-GPU transfers and postprocessing. Other entries live on the CPU. Oversized results are not admitted. Within a grouped or batched stage, the same conditioning boundary can reuse a device result before consulting the cross-request cache. The common text stage and the Qwen-Image 2.1, Ming-Image, and LingBot-Video text stages share read-only consumed tensors and give each request its own containers. Other cached boundaries return private copies, with device snapshots limited to 512 MiB and 128 entries per stage. These temporary references are released when the stage finishes or fails. Group reuse remains available when cross-request caching is disabled or its budget is too small. The standard text encoding stage stores the final embeddings, pooled outputs, masks, and sequence lengths. A hit skips encoder weight preparation, token uploads, and mask construction; CPU tokenization still runs to match the inputs. Qwen-Image 2.1 also caches its consumed prompt embeddings and image-token slots before processor execution, while retaining separate vision-feature reuse for changed edit instructions. Other custom stages and batch-DP encoding retain caching at the native encoder boundary. CPU hits restore outputs to their original devices; device hits return private device copies. Negative conditioning can therefore retain GPU memory between requests, within the shared cache budget. Measure memory as well as latency when using a tight VRAM budget. This is most useful when generating several variations from the same prompt or reference image, especially with a layerwise-offloaded encoder. A changed edit instruction recomputes the joint vision-language embedding, while the unchanged image can still reuse its vision features and VAE posterior. Separate CLI invocations start separate processes and cannot share this cache. Denoising and output decoding still run for every request, so end-to-end gains depend on how much time the original request spends encoding conditioning. To disable cross-request conditioning reuse: ```bash theme={null} sglang serve --model-path Qwen/Qwen-Image-2.1 --disable-conditioning-cache ``` To increase the shared cache budget: ```bash theme={null} sglang serve --model-path Qwen/Qwen-Image-2.1 --conditioning-cache-max-size-mb 1024 ``` The same settings are accepted by `DiffGenerator.from_pretrained()` as `disable_conditioning_cache` and `conditioning_cache_max_size_mb`. A capacity of `0` also disables both cross-request tiers. Neither setting disables conditioning reuse within a grouped stage or changes per-output seeds. ### Matching and correctness * Keys include the loaded encoder instance, actual input contents, shapes, dtypes, masks, and autocast precision. Image tensors and PIL images are matched by content, so replacing a file at the same path cannot reuse stale pixels. Alpha is part of the VAE input. * Positive and negative inputs use the same bounded cache. Different seeds can reuse conditioning; initial noise is still generated for each request. * VAE caching stores the posterior before sampling and normalization. Every request still calls `sample()` or `mode()` as its model requires, preserving the random-number sequence. * Cross-request hits return independent copies. Grouped consumed outputs may share read-only tensors; other boundaries isolate mutable tensors and posteriors. In-place normalization and downstream mutations cannot change a persistent entry. * Encoder TP and folding groups agree on hits before skipping collective work. Weight updates and LoRA changes invalidate the cache, including failed updates that may have modified weights. * Warmup refreshes cached results and executes each distinct encoding within a group once. This primes kernels while preserving multi-output reuse; subsequent requests can reuse conditioning such as the model-default negative prompt. Compare disabled, cold, and repeated-input latency separately when benchmarking. ### Coverage and limits The common encoder boundary covers native CLIP, T5/UMT5, Llama, Mistral 3, Gemma 2/3, Qwen 2.5-VL, Qwen 3/3-VL, Ideogram, and MiniMax-H3 encoders, including custom pipeline stages. Qwen-VL and Gemma 3 also cache vision features independently of text. Hunyuan3D image encoders and the LTX-2.3 video condition encoder participate. Native image, video, and audio VAE encode entry points cache deterministic tensors or supported posterior containers. Cosmos3 has no separate text encoder: its deterministic understanding pathway caches immutable per-layer K/V results, which are copied into fresh request state. SenseNova-U1 caches reference-image vision features; its noisy-image vision path still runs at each denoising step. VLA keeps its model-owned observation/prefix cache. FSDP disables cross-request caching and caching inside sharded modules. Consumed text-conditioning outputs still support group-local reuse, with all ranks agreeing before skipping the encoder. The common text stage also reuses consumed outputs from library fallback encoders within a group and keeps their negative conditioning device-hot across requests. Batch-DP can skip encoding and its output gather only when all participating ranks have the consumed result; this includes device-hot negatives. Otherwise each encoder copy retains its normal cache and gather behavior. Stateful autoregressive calls (`use_cache=True` or existing `past_key_values`), training/autograd, CUDA Graph capture, unsupported tensor or output types, and distributed VAE encoding bypass the shared caches. External SRT encoders and the Diffusers pipeline backend keep their own cache behavior. The global disable flag also disables existing realtime text and VLA prefix reuse; those model-specific caches otherwise retain their own memory settings. Joint vision-language embeddings depend on both the prompt and image. The cache does not reuse a previous turn's combined embedding when either changes, and does not carry mutable denoising KV state between requests. Dynamic batching still decides which requests can execute together. Conditioning reuse does not change encoder batch shapes, padding, or output order: a batched encoder call is matched as a whole, rather than split into individually cached rows. Model-specific grouped execution that also prepares materials, broadcasts results, or manages state retains its own execution rules. The common text stage also keeps group-local stage deduplication: equivalent requests execute the stage once and receive their own output containers. This avoids repeating tokenization and stage bookkeeping. When stage deduplication leaves only one computation, it also skips the temporary group cache and its hit synchronization; cross-request caching remains available. The conditioning cache shares encodings between otherwise different requests, including a common negative prompt. Custom stages that prepare request-specific metadata or realtime session state use encoder-result reuse while still preparing each request. ## Overview The following approaches reduce denoising computation:
Strategy Scope Mechanism Best For
Cache-DiT Block-level Skip individual transformer blocks dynamically Advanced, higher speedup
TeaCache Timestep-level Skip entire denoising steps based on L1 similarity Simple, built-in
Spectrum Timestep-level Forecast DiT features to skip selected denoising steps Experimental, model-validated tuning
## Cache-DiT [Cache-DiT](https://github.com/vipshop/cache-dit) provides block-level caching with advanced strategies like DBCache and TaylorSeer. It can achieve up to **1.69x speedup**. See [Cache-DiT](./cache_dit) for detailed configuration. Cache-DiT currently cannot be combined with `--use-fsdp-inference`. Keep FSDP disabled when enabling Cache-DiT. DiT layerwise offload is compatible: skipped blocks are not streamed, and the first layer after a skip may sync-load. ### Quick Start ```bash theme={null} SGLANG_CACHE_DIT_ENABLED=true \ sglang generate --model-path Qwen/Qwen-Image \ --prompt "A beautiful sunset over the mountains" ``` ### Key Features * **DBCache**: Dynamic block-level caching based on residual differences * **TaylorSeer**: Taylor expansion-based calibration for optimized caching * **SCM**: Step-level computation masking for additional speedup ## TeaCache TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely. See [TeaCache](./teacache) for detailed documentation. ### Quick Overview * Tracks L1 distance between modulated inputs across timesteps * When accumulated distance is below threshold, reuses cached residual * Uses separate positive/negative caches for supported CFG model families ### Supported Models * Wan2.1 * Z-Image * Wan2.2: coefficients are not calibrated yet; enabling TeaCache is accepted but currently no-ops * HunyuanVideo: not supported yet For Flux and Qwen models, TeaCache is automatically disabled when CFG is enabled. ## Spectrum Spectrum forecasts DiT features and skips selected denoising steps. It is approximate and currently applies only to selected native implementation paths. See [Spectrum Acceleration](./spectrum) for supported model families, constraints, and request controls. ## References * [Cache-DiT Repository](https://github.com/vipshop/cache-dit) * [TeaCache Paper](https://arxiv.org/abs/2411.14324) # CI Performance Baselines Source: https://docs.sglang.io/docs/sglang-diffusion/ci_perf Generate and update diffusion performance baselines used in CI. ## Sequential Request Checks Server cases can exercise request-to-request state without restarting the server: ```python theme={null} DiffusionTestCase( id="image_t2i", server_args=server_args, sampling_params=sampling_params, perf_repeat_requests=2, ) ``` Every request is checked, including the first one. Performance checks include load/runtime peak memory; consistency checks include audio when `expect_audio_output=True`. A failure in either request fails the case, even if the other succeeds. Reports retain the request index instead of overwriting the first result with the second. Keep workloads small, but regenerate baselines and GT whenever sampling parameters change. MiniMax-H3 T2VA and ref2va use different checkpoint partitions, so they remain separate cases, each issuing two requests to its own server. The T2VA case includes VAE layerwise offload to exercise weight restoration between requests; ref2va retains the resident VAE path. Use `SGLANG_GEN_BASELINE=1 pytest -s ... -k ` to collect baselines in the same sequential order, retaining the worst observed value of each metric. `SGLANG_GEN_GT=1` with `SGLANG_GT_OUTPUT_DIR` saves GT; repeated requests share one GT. ## Perf Baseline Generation Script `python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py` starts a local diffusion server, issues requests for selected test cases, aggregates stage/denoise-step/E2E timings from the perf log, and writes the results back to the `scenarios` section of `perf_baselines.json`. ### Usage Update a single case: ```bash theme={null} python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --case qwen_image_t2i ``` Select by regex: ```bash theme={null} python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --match 'qwen_image_.*' ``` Run all keys from the baseline file `scenarios`: ```bash theme={null} python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --all-from-baseline ``` Specify input/output paths and timeout: ```bash theme={null} python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --baseline python/sglang/multimodal_gen/test/server/perf_baselines.json --out /tmp/perf_baselines.json --timeout 600 ``` ## Nightly Comparison Telemetry The nightly comparison reports the median of three client-side measurements. SGLang requests also write server perf dumps for stage-level timing. An unreadable dump does not invalidate a completed client measurement, but the dashboard reports server sample coverage (for example, `2/3`) and computes the server total and stage medians only from readable dumps. Treat incomplete coverage as a telemetry warning rather than a performance regression by itself. # ComfyUI plugin Source: https://docs.sglang.io/docs/sglang-diffusion/comfyui Use SGLang Diffusion from ComfyUI in server mode or as a per-step DiT backend. The [ComfyUI SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion) has two modes. **Server mode** talks to a standalone `sglang serve` process over HTTP. ComfyUI sends prompts and receives images or video. This is the same path as the [OpenAI-compatible API](/docs/sglang-diffusion/api/openai_api). **Integrated mode** keeps ComfyUI's CLIP, VAE, and sampler loop. SGLang loads only the DiT and runs one forward per sampler step. The worker starts the native Flux / Qwen-Image / Z-Image / MiniMax-H3 pipeline under `--comfyui-mode`. A single-file ComfyUI `.safetensors` (or a GGUF overlay) is loaded through a checkpoint spec. Each `apply_model` call is translated by a per-model adapter. Integrated mode does not ship a separate `comfyui_*` pipeline class per model. ## Install the plugin 1. Install `sglang[diffusion]`. See [Installation](/docs/sglang-diffusion/installation). 2. Copy `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion` into ComfyUI's `custom_nodes/` directory. 3. Restart ComfyUI. Example workflows live next to the plugin under `workflows/`. ## Integrated mode 1. Load the DiT with `SGLDiffusion UNET Loader`. 2. Set `num_gpus`, `tp_size`, `model_type`, or compile flags with `SGLDiffusion Options`. 3. Connect the loaded model to a standard ComfyUI sampler. Supported integrated-mode families: Flux, Qwen-Image, Z-Image, MiniMax-H3. Qwen-Image edit is experimental. ## How a sampler step reaches the worker The plugin README has the [architecture diagram](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion#architecture). The hop is: 1. A model adapter packs ComfyUI tensors into an SGLang `Req`. 2. Local ZMQ replaces CUDA tensors with IPC handles so latents stay on GPU. 3. Rank 0 materializes the handles. Multi-rank `--comfyui-mode` then detaches CUDA tensors and broadcasts them over NCCL. The general SP / CFG / TP path is still the original `broadcast_pyobj`. 4. The worker pipeline keeps `transformer` plus a pass-through scheduler. After the first step, conditioning stays in a worker session; later steps send latents and the timestep. Integrated mode currently supports: | Family | ComfyUI `model_type` | Native pipeline | Example checkpoints | | - | - | - | - | | Flux | `flux` | `FluxPipeline` | `FLUX.1-dev` | | Z-Image | `lumina2` | `ZImagePipeline` | `Z-Image-Turbo` | | Qwen-Image | `qwen_image` | `QwenImagePipeline` | `Qwen-Image`, `Qwen-Image-2512` | | Qwen-Image edit | `qwen_image_edit` | `QwenImageEditPlusPipeline` | `Qwen-Image-Edit-2511` (experimental) | | MiniMax-H3 | `minimax_h3` | `MiniMaxH3Pipeline` | `minimax_h3_fl2va_*.safetensors`, `minimax_h3_ref2va_*.safetensors` | MiniMax-H3 example graphs: `minimax_h3_t2v_sgld.json` (T2V / I2VA / FL2VA on the FL2VA DiT), `minimax_h3_r2v_sgld.json` (reference-to-video; use a `ref2va` file), `minimax_h3_t2v_sgld_upscaler.json` (two-pass 3D ×2). Set `SGLDOptions.model_type` to `minimax_h3`. On 24 GB, keep the 32B CLIP on CPU and prefer pruned BF16, GGUF, or `kitchen_int8` with DiT layerwise offload. See the [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#comfyui-integrated-mode). To add a model, register a checkpoint spec under `runtime/loader/comfyui_checkpoints/` and a `ComfyUIModelAdapter`. Do not add another ComfyUI pipeline class. # Supported Models Source: https://docs.sglang.io/docs/sglang-diffusion/compatibility_matrix Browse model families and public checkpoints supported by SGLang Diffusion. ## Supported model inventory Use a listed checkpoint as `--model-path` with `sglang generate` or `sglang serve`. This registry-backed list contains known public entry points; family detection may also support compatible local directories. Open the linked Cookbook recipe for launch commands, optimizations, adapters, and model-specific notes. # Contributing to SGLang Diffusion Source: https://docs.sglang.io/docs/sglang-diffusion/contributing This guide outlines the requirements for contributing to the SGLang Diffusion module (`sglang.multimodal_gen`). ## Contributor Guides * [Support New Models](/docs/sglang-diffusion/support_new_models): implementation guide for adding new diffusion pipelines * [Helper ownership](/docs/sglang-diffusion/support_new_models#place-helpers-with-their-owners): where to put shared and model-specific utilities * [CI Performance](./ci_perf): update and regenerate perf baselines * [SGLang Plugin System](/docs/hardware-platforms/plugin): package a hardware platform outside the SGLang repository ## On AI-Assisted ("Vibe Coding") PRs Vibe-coded PRs are welcome — we judge code quality, not how it was produced. The bar is the same for all PRs: * **No over-commenting.** If the name says it all, skip the docstring. * **No over-catching.** Don't guard against errors that virtually never happen in practice. * **Test before submitting.** AI-generated code can be subtly wrong — verify correctness end-to-end. ## Commit Message Convention We follow a structured commit message format to maintain a clean history. **Format:** ```text theme={null} [diffusion] : ``` **Examples:** * `[diffusion] cli: add --perf-dump-path argument` * `[diffusion] scheduler: fix deadlock in batch processing` * `[diffusion] model: support Stable Diffusion 3.5` **Rules:** * **Prefix**: Always start with `[diffusion]`. * **Scope** (Optional): `cli`, `scheduler`, `model`, `pipeline`, `docs`, etc. * **Subject**: Imperative mood, short and clear (e.g., "add feature" not "added feature"). ## Performance Reporting For PRs that impact **latency**, **throughput**, or **memory usage**, you **should** provide a performance comparison report. ### How to Generate a Report 1. **Baseline**: run the benchmark (for a single generation task) ```bash theme={null} $ sglang generate --model-path --prompt "A benchmark prompt" --perf-dump-path baseline.json ``` 2. **New**: run the same benchmark, without modifying any server\_args or sampling\_params ```bash theme={null} $ sglang generate --model-path --prompt "A benchmark prompt" --perf-dump-path new.json ``` 3. **Compare**: run the compare script, which will print a Markdown table to the console ```bash theme={null} $ python python/sglang/multimodal_gen/benchmarks/compare_perf.py baseline.json new.json [new2.json ...] ### Performance Comparison Report ... ``` 4. **Paste**: paste the table into the PR description ## CI-Based Change Protection Consider adding tests to the `pr-test` or `nightly-test` suites to safeguard your changes, especially for PRs that: * support a new model * add a testcase for this new model to `testcase_configs.py` * support or fix important features * significantly improve performance Please run the according testcase, then update/add the baseline to `perf_baselines.json` by following the instruction in console if applicable. See [test](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/test) for examples ## Out-of-tree platform changes Use the `sglang.multimodal_gen.platforms` entry-point group when a hardware integration can live in a separately installed package. The [plugin guide](/docs/hardware-platforms/plugin#add-diffusion-support) describes the activation callback, diffusion platform contract, and selection behavior. When you change the platform interface or discovery implementation in SGLang: * Add focused CPU-only tests with a fake OOT platform. * Run the relevant SRT plugin tests when changing shared hook-registry behavior. * Document workload-specific kernels or optimized paths that still require vendor integration. # Deployment and Performance Modes Source: https://docs.sglang.io/docs/sglang-diffusion/deployment_cookbook Choose component residency, FSDP, CFG parallelism, SP, TP, and performance-mode presets in SGLang Diffusion. This page gives practical defaults for choosing `--performance-mode`, component residency, FSDP, CFG parallelism, SP, and TP. ## Quick Rule Use the simplest setting that fits your memory target:
Goal Recommended setting
Fastest single-GPU run when the model fits Use resident components and do not use FSDP.
Lower single-GPU memory usage Use component offload, then layerwise offload when a complete component still does not fit comfortably.
Faster multi-GPU Qwen/Wan CFG generation Use FSDP with CFG parallelism and keep the sharded component resident.
Sequence length or video-shape scaling Use SP/Ulysses/Ring when the model benefits from sequence parallelism.
TP compatibility or encoder-heavy paths Set TP explicitly; do not treat TP as the default latency optimization.
Base the decision on available memory on the selected GPU(s). * For multi-GPU deployment: the least-free selected GPU is the bottleneck. A busy 80GiB GPU can behave like a much smaller GPU. * For single-GPU deployment: FSDP shards weights across multiple GPUs. It is not useful for keeping a single-GPU deployment on one GPU; use component or layerwise offload instead. ## Health Probes Use `/liveness` to check that the HTTP process is alive and `/health` to check that the server is ready for inference. During server-based warmup, `/liveness` returns `200` while `/health` returns `503`. Configure the startup probe with a failure budget large enough for model loading and compilation: ```yaml theme={null} startupProbe: httpGet: path: /health port: 30010 periodSeconds: 10 failureThreshold: 180 readinessProbe: httpGet: path: /health port: 30010 livenessProbe: httpGet: path: /liveness port: 30010 ``` See [Health endpoints](/docs/sglang-diffusion/api/cli#health-endpoints) for the status-code contract and warmup-mode behavior. ## Stable Model Identity Use `--served-model-name` when the public model name must remain stable across replicas, hosts, or checkpoint mount paths: ```bash theme={null} sglang serve \ --model-path /mnt/checkpoints/Qwen-Image \ --model-id Qwen-Image \ --served-model-name image-production \ --port 30010 ``` The resolved public name follows `--served-model-name`, then `--model-id`, then `--model-path`. `--model-id` remains an internal model registry and configuration-resolution hint; it is not a replacement for a deployment alias. The resolved name is exposed through `/server_info` and `/v1/models` and is used by video and action responses when a request does not supply its own model. See [OpenAI API: Served model name](/docs/sglang-diffusion/api/openai_api#served-model-name) for discovery and retrieval examples. ## Performance Modes `--performance-mode` applies safe presets without overriding explicit offload, FSDP, or parallelism flags. `auto` is the default. Use `manual` when you need to keep performance-related server args under explicit user control. `--mode` is a short alias.
Mode Meaning
`manual` Keeps performance-related server args under explicit user control.
`auto` Default. Keeps legacy safe offload defaults and uses FSDP/CFG only on validated multi-GPU deployments where FSDP can replace DiT offload.
`speed` Favors GPU-resident execution for lower latency and higher throughput. Disables CPU offload when unset. `torch.compile` stays off unless the model has a validated default or it is enabled explicitly; may OOM.
`memory` Favors lower GPU memory. Uses component offload, or Wan/MOVA layerwise DiT offload when supported.
`auto` checks selected GPU memory before applying FSDP. In multi-GPU runs it uses the least available memory across selected GPUs, and only turns on FSDP automatically when doing so can replace DiT offload. For image workloads with at least 45 GiB available per selected GPU, it keeps the repeatedly reused DiT resident and uses layerwise offload for large auxiliary encoders; below that threshold it keeps the DiT offloaded. Model-specific components such as VAEs become resident only when their configured memory threshold is met. `memory` instead keeps the VAE in its default layerwise set to maximize memory headroom; use `--component-residency vae=resident` for a measured recipe with sufficient capacity. Video DiT residency remains model- and workload-specific because frame count and resolution change its peak memory substantially. When the model default uses CFG and the user did not set a parallelism policy, `auto` may also enable CFG parallelism. `speed` intentionally does not check memory; it is the mode for users who prefer latency/throughput and accept OOM risk. It keeps `torch.compile` disabled by default because its effect varies by model and workload. A model-specific deployment config may enable a validated compile path, and `--enable-torch-compile true` always opts in explicitly. The modes tune native pipeline components declared to the component residency manager. DiTs, text/image encoders, VAEs, vocoders, adapters, and upsamplers can use layerwise offload when their native module declares its executable layer structure. Explicitly selecting an unsupported component fails at startup instead of falling back to another residency mode. For direct control, assign one of `resident`, `component-offload`, `snapshot-offload`, or `layerwise-offload` with `--component-residency COMPONENT=MODE`: ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --component-residency dit=layerwise-offload text_encoder=component-offload vae=resident ``` Existing per-component CPU-offload and layerwise flags remain supported. Canonical selectors override matching legacy settings only; unmatched legacy settings and automatic defaults remain effective. See [Component Residency](/docs/sglang-diffusion/api/cli#component-residency) for the complete precedence rules. When `torch.compile` is enabled, `--offload-during-compile` stays on by default. During compile warmup it temporarily offloads the DiT and evicts resident non-DiT components so `max-autotune` fits on tighter-memory GPUs, then restores the configured serving residency before real traffic. Breakable CUDA graph is a separate manual opt-in for supported image pipelines. If you enable `--enable-breakable-cuda-graph`, declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures. The preset is intentionally coarse. A future continuous value such as `0.0` to `1.0` could express the speed-memory tradeoff more precisely, but it would need model-specific memory models and clearer user expectations. Until then, use the preset plus explicit flags for overrides. Examples: ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --num-gpus 2 \ --performance-mode auto ``` ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --performance-mode memory ``` Explicit flags win over the mode: ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --num-gpus 2 \ --performance-mode auto \ --use-fsdp-inference false ``` In this example, `auto` will not re-enable FSDP. The same applies to parallelism; for example, `--enable-cfg-parallel false` keeps CFG parallelism disabled. ## Interpreting The Levers **Resident** keeps the complete component on the accelerator. It is usually fastest when memory is sufficient. **Component offload** keeps a complete component on CPU between declared uses. It is simple and robust, but each use pays a whole-component transfer. **Layerwise offload** streams the declared layers of any supported native weighted component. It lowers peak accelerator memory further, but may increase latency and lower throughput. ### Which components to stream `--layerwise-offload-components` takes a list, and more is not better. Streaming pays one host-to-device transfer of a component's weights **every time that component runs**, overlapped with that run's compute. What matters is how often it runs per request, not how large it is. * The **DiT** runs once per denoising step. Its transfer amortizes over every step and hides behind attention, so it is the component layerwise offload is for. * A **video VAE** runs once per temporal chunk, not once: H3's decoder re-runs whole for each chunk, so streaming its blocks pays their transfer per chunk. * A **text encoder** runs once. Keep it resident if it fits; otherwise stream it with its blocks resident. There are three placements, not two. A component can be in the list and still hold its blocks — `--layerwise-offload-components dit,text_encoder,vae` together with `--layerwise-resident-layers video_vae=36` transfers them once instead of per pass, and gives the VRAM back when the component finishes rather than holding it through denoise. On MiniMax-H3 at 864x480 / 124 frames that is 13 s of decode against 150 s streamed. That hand-back is the default lifetime, `forward`, and it is what lets the DiT's resident layers and the VAE's share one budget: each set is on the GPU only while its component runs. When both sets fit at once, `--layerwise-residency-lifetime video_vae=permanent` (or `--dit-layerwise-residency-lifetime permanent` for the DiT) places them at load instead — no transfer per request, and no host copy pinned for a reload that never comes. The trade is peak memory: `permanent` adds every component's resident set together, `forward` only the largest. On a card where the sum does not fit, `permanent` fails at startup rather than slowly; keep `forward` there. On a consumer card the trade is usually bad, so measure before reaching for it. Qwen-Image-2.1 on one RTX 5090 (1024px, 40 steps, three interleaved rounds, five requests each), making the text encoder's resident set permanent: | `text_encoder` resident | p50 | vs `forward` | peak at load | of a 31.4 GiB card | | - | - | - | - | - | | none (default) | 13.53 s | — | 25168 MiB | 78% | | 0.3 permanent | 13.45 s | −0.6% | 29570 MiB | 92% | | 0.5 permanent | 13.41 s | −0.8% | 31480 MiB | 98% | | 0.8 permanent | 13.37 s | −1.2% | 32092 MiB | 100% | Every round reproduces to 0.1% and the ordering is consistent, so the win is real — and it is 0.6% to 1.2% in exchange for nearly all of the card's remaining headroom, on a machine that also has a desktop to run. The faster the card, the smaller the share the saved transfer represents: the same flag measured −2.8% on a slower box whose baseline was 14.14 s. Output stays within the usual floating-point reordering (0.5 is bit-identical to the default; 0.8 deviates by at most 1/255 with no pixel beyond one LSB). `permanent` earns its memory where one use makes many passes and the sum still fits — a datacenter card, or a component whose set is small next to the budget. So do not read "drop it from the list" as the fix for a one-shot component: that keeps it resident for the whole process. With no resident blocks on either side, moving H3's 10.4 GB video VAE out of the list took decode from 39.9 s to 5.3 s on one RTX 4090 — but it also added 5.8 GB of peak, which is the budget a 12 GB card does not have. ### How deep to prefetch `--dit-offload-prefetch-size` is not monotonic. Deeper prefetch hides more of the copy, but its staging buffers crowd out activations, so latency turns back up and peak memory keeps climbing. On the H3 configuration above, 1/2/3/4 layers measured 17.3 / 15.3 / 15.8 / 16.7 s of denoise, with the deepest setting reaching 96% of a 24 GB card. Sweep two or three values rather than trusting the default. ### When not to bother Before tuning residency or prefetch, measure whether the transfer is exposed — and measure it rather than inferring it from the checkpoint size. Bytes over bandwidth bounds what *could* be exposed, not what is, because prefetch exists to hide exactly that. A 1.3B video DiT moves about 2.6 GB a step, on the order of a tenth of a one-second step if none of it overlapped, and its residency settings measure flat: it overlaps, so there is nothing to recover. A 50-layer model at 1.4 GB a layer moves 66 GB a step, and there the same flags decide whether the model runs at all. Sweep two or three values on the target configuration and keep the measured winner. **FSDP** shards DiT weights across multiple GPUs and all-gathers weights during forward. It can reduce DiT CPU offload cost on multi-GPU deployments, especially for validated Wan I2V workloads. FSDP sharding granularity matters. SGLang Diffusion prefers sharding direct repeated transformer block entries such as `transformer_blocks.0` or `blocks.0`. Coarser sharding lowers wrapper count but can increase all-gather peak memory; finer sharding can reduce transient memory but adds communication and scheduling overhead. If a model does not define an explicit sharding rule, the loader falls back to repeated block class names and common direct numbered block paths. **CFG parallelism** splits positive and negative CFG branches across GPUs. For Qwen/Wan workloads with normal step counts, this is the most reliable multi-GPU speedup observed so far. **SP/Ulysses/Ring** splits sequence work. It can help video workloads, but validated Qwen/Wan runs showed CFG parallelism outperforming SP for latency. **TP** is supported for compatibility and some model structures, but current measurements do not make it the default latency path for Qwen/Wan. ## Current Benchmark Takeaways Observed regular-scale trends: * Z-Image: single-GPU no-offload was faster than FSDP/SP in the tested setting; keep FSDP off unless memory or parallelism requires it. * Qwen-Image: keep the default non-FSDP path unless a specific FSDP/SP/Ring setting has been benchmarked on the target hardware. * Wan: FSDP can replace DiT offload on validated multi-GPU workloads, while text/image encoders may still need component offload. Keep model-specific precision checks before making FSDP automatic for a path. * Component offload mainly reduced memory; it did not improve latency in the tested no-offload-vs-offload runs. Always benchmark with your actual resolution, frame count, step count, and GPU type before locking production defaults. # Disaggregated Diffusion Pipeline Source: https://docs.sglang.io/docs/sglang-diffusion/disaggregation Split a monolithic text-to-video/image pipeline into independent **Encoder**, **Denoiser**, and **Decoder** roles, each running on its own GPU(s). A central **DiffusionServer** routes requests through the pipeline. ## Quick Start Disaggregation is controlled by a single flag: `--disagg-role`. Each component is launched independently, just like LLM PD disaggregation.
--disagg-role What it runs
monolithic (Default) Standard single-server mode
encoder All stages with the default RoleType.ENCODER affinity: InputValidationStage, TextEncodingStage (plus ImageEncodingStage / ImageVAEEncodingStage for image-conditioned pipelines), LatentPreparationStage, TimestepPreparationStage, and any model-specific "before denoising" stage (e.g. QwenImageLayeredBeforeDenoisingStage, GlmImageBeforeDenoisingStage).
denoiser DenoisingStage (and its subclasses: CausalDMDDenoisingStage, DmdDenoisingStage, LTX2AVDenoisingStage, LTX2RefinementStage, Hunyuan3DShapeDenoisingStage, ...) — the DiT forward loop plus the scheduler stepping it drives.
decoder DecodingStage (VAE decode) and its subclasses (LTX2AVDecodingStage, HeliosDecodingStage, ...).
server DiffusionServer head node + HTTP server (no GPU)
> Each stage declares its role via the `role_affinity` property on `PipelineStage` (default `ENCODER`). When `--disagg-role` is not `monolithic`, the pipeline only instantiates stages whose affinity matches, so the above table is the source of truth for what actually runs in each process. ### Single-Machine Example (Verified) The following commands have been tested end-to-end on an 8×H200 machine with `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`. Each role runs on a separate GPU via `--base-gpu-id`; the `server` head node requires no GPU. ```bash theme={null} # Terminal 1: Encoder (GPU 0) sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --disagg-role encoder \ --disagg-server-addr tcp://127.0.0.1:19655 \ --scheduler-port 19000 \ --num-gpus 1 --base-gpu-id 0 # Terminal 2: Denoiser (GPU 1) sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --disagg-role denoiser \ --disagg-server-addr tcp://127.0.0.1:19655 \ --scheduler-port 19001 \ --num-gpus 1 --base-gpu-id 1 # Terminal 3: Decoder (GPU 2) sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --disagg-role decoder \ --disagg-server-addr tcp://127.0.0.1:19655 \ --scheduler-port 19002 \ --num-gpus 1 --base-gpu-id 2 # Terminal 4: DiffusionServer head (no GPU, receives HTTP requests) sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --disagg-role server \ --encoder-urls "tcp://127.0.0.1:19000" \ --denoiser-urls "tcp://127.0.0.1:19001" \ --decoder-urls "tcp://127.0.0.1:19002" \ --host 0.0.0.0 --port 22000 \ --scheduler-port 19655 # Send request (video generation) curl http://127.0.0.1:22000/v1/videos \ -H "Content-Type: application/json" \ -d '{"model": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers", "prompt": "A curious raccoon exploring a garden, cinematic", "size": "832x480"}' ``` > **Tested result (8×H200):** > Encoder 2.3 s (TextEncoding) → Denoiser 312.8 s (50 steps, layerwise offload) → Decoder 7.1 s (VAE decode). > Total \~322 s for 81-frame 1024×1024 video. > **Tip:** `--base-gpu-id` controls which physical GPU the role uses. > Encoder and Decoder can share a GPU (e.g. both `--base-gpu-id 0`) to save resources, > but make sure the combined GPU memory is sufficient. ### Multi-Machine Example The exact same CLI pattern — just replace `127.0.0.1` with actual IPs and add RDMA flags for direct transfer: ```bash theme={null} # Machine A (10.0.0.1): Encoder sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers \ --disagg-role encoder \ --disagg-server-addr tcp://10.0.0.4:19655 \ --scheduler-port 19000 \ --num-gpus 1 \ --disagg-p2p-hostname 10.0.0.1 --disagg-ib-device mlx5_0 # Machine B (10.0.0.2): Denoiser (4 GPUs with SP) sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers \ --disagg-role denoiser \ --disagg-server-addr tcp://10.0.0.4:19655 \ --scheduler-port 19001 \ --num-gpus 4 --denoiser-sp 4 --denoiser-ulysses 2 --denoiser-ring 2 \ --disagg-p2p-hostname 10.0.0.2 --disagg-ib-device mlx5_0 # Machine C (10.0.0.3): Decoder sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers \ --disagg-role decoder \ --disagg-server-addr tcp://10.0.0.4:19655 \ --scheduler-port 19002 \ --num-gpus 1 \ --disagg-p2p-hostname 10.0.0.3 --disagg-ib-device mlx5_0 # Machine D (10.0.0.4): DiffusionServer head sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers \ --disagg-role server \ --encoder-urls "tcp://10.0.0.1:19000" \ --denoiser-urls "tcp://10.0.0.2:19001" \ --decoder-urls "tcp://10.0.0.3:19002" \ --host 0.0.0.0 --port 30000 \ --scheduler-port 19655 \ --disagg-dispatch-policy max_free_slots ``` > ZMQ handles startup order gracefully — instances and head can start in any order. ## Multiple Instances per Role Use semicolons in `--*-urls` to register multiple instances: ```bash theme={null} # 2 encoders + 2 denoisers (4-GPU SP each) + 1 decoder sglang serve --model-path ... --disagg-role server \ --encoder-urls "tcp://10.0.0.1:35000;tcp://10.0.0.2:35000" \ --denoiser-urls "tcp://10.0.0.3:35000;tcp://10.0.0.4:35000" \ --decoder-urls "tcp://10.0.0.5:35000" ``` ### GLM-Image distributed mode GLM-Image can batch AR generation in the head and dispatch the resulting prior tokens to distributed denoiser workers. Each worker runs prompt/glyph preparation, DiT, and VAE decoding locally; no latent or embedding tensors are transferred. The following 16-device deployment uses devices 0-1 for the external AR server and devices 2-15 for 14 independent batch-1 Cache-DiT denoisers. ```bash theme={null} # Run 14 distributed denoisers DISAGG_SERVER="tcp://127.0.0.1:19655" MODEL_PATH="zai-org/GLM-Image" BASE_MASTER_PORT=29005 export SGLANG_CACHE_DIT_FN=2 export SGLANG_CACHE_DIT_BN=1 export SGLANG_CACHE_DIT_WARMUP=4 export SGLANG_CACHE_DIT_RDT=0.4 export SGLANG_CACHE_DIT_MC=4 export SGLANG_CACHE_DIT_TAYLORSEER=true export SGLANG_CACHE_DIT_TS_ORDER=2 export SGLANG_CACHE_DIT_ENABLED=true worker_pids=() cleanup() { trap - EXIT if ((${#worker_pids[@]})); then kill "${worker_pids[@]}" 2>/dev/null || true wait "${worker_pids[@]}" 2>/dev/null || true fi } trap cleanup EXIT trap 'exit 130' INT trap 'exit 143' TERM for i in $(seq 2 15); do scheduler_port=$((19001 + i)) master_port=$((BASE_MASTER_PORT + i)) sglang serve \ --model-path "$MODEL_PATH" \ --disagg-role denoiser \ --disagg-server-addr "$DISAGG_SERVER" \ --srt-encoder-url http://127.0.0.1:30020 \ --scheduler-port "$scheduler_port" \ --master-port "$master_port" \ --num-gpus 1 \ --base-gpu-id "$i" \ --denoiser-sp 1 \ --cfg-parallel-size 1 \ --batching-max-size 1 \ --dit-cpu-offload false \ --attention-backend fa & worker_pids+=("$!") done # Stop all denoisers if any worker exits or fails during startup. wait -n "${worker_pids[@]}" ``` Run the external AR server: ```bash theme={null} sglang serve \ --model-path ./zai-org/GLM-Image/vision_language_encoder/ \ --tokenizer-path ./zai-org/GLM-Image/processor/ \ --enable-multimodal \ --cuda-graph-max-bs-decode 28 \ --device npu \ --attention-backend ascend \ --disable-fast-image-processor \ --tp-size 2 \ --host 0.0.0.0 \ --port 30020 \ --mem-fraction-static 0.8 ``` Run the public head. `--encoder-urls` and `--decoder-urls` are intentionally omitted for this topology. ```bash theme={null} sglang serve \ --model-path zai-org/GLM-Image \ --disagg-role server \ --srt-encoder-url http://127.0.0.1:30020 \ --srt-encoder-timeout 300 \ --denoiser-urls "tcp://127.0.0.1:19003;tcp://127.0.0.1:19004;tcp://127.0.0.1:19005;tcp://127.0.0.1:19006;tcp://127.0.0.1:19007;tcp://127.0.0.1:19008;tcp://127.0.0.1:19009;tcp://127.0.0.1:19010;tcp://127.0.0.1:19011;tcp://127.0.0.1:19012;tcp://127.0.0.1:19013;tcp://127.0.0.1:19014;tcp://127.0.0.1:19015;tcp://127.0.0.1:19016" \ --batching-mode dynamic \ --batching-max-size 28 \ --batching-delay-ms 30 \ --enable-batching-metrics \ --host 0.0.0.0 \ --port 30052 \ --scheduler-port 19655 \ --output-path ./outputs ``` Workers return decoded pixels to the head, which saves and serves the final files from `--output-path`. The PR benchmark used [longtext-bench.zip](https://github.com/user-attachments/files/29779516/longtext-bench.zip): ```bash theme={null} python fetch_images.py \ --base-url http://localhost:30052/v1 \ --model GLM-Image-distributed-test \ --output-dir generated_images \ --max-concurrency 56 ``` ## Port Convention Result endpoints are derived deterministically from the head node's `--scheduler-port` (default: 5555):
Socket Port
DS frontend (ROUTER) scheduler\_port
Encoder result (PULL) scheduler\_port + 1
Denoiser result (PULL) scheduler\_port + 2
Decoder result (PULL) scheduler\_port + 3
Role instances derive their result endpoint automatically from `--disagg-server-addr`. No manual endpoint configuration needed. ## Transfer Mechanism Tensor data between roles (encoder→denoiser, denoiser→decoder) is transferred via a P2P transfer engine. The DiffusionServer only routes lightweight control messages (alloc/push/ready); actual tensor data flows directly between instances. **mooncake-transfer-engine** is required for disaggregated diffusion. It provides RDMA for direct GPU-to-GPU data movement. The GLM-Image distributed mode is an exception: it relays only prior token IDs and request metadata over ZMQ and does not require Mooncake. ```bash theme={null} pip install mooncake-transfer-engine ``` ### Transfer Flow 1. **Sender** (encoder/denoiser) stages tensors: async copy to transfer buffer (GPU or CPU pinned, depending on GPUDirect support), overlapped with metadata JSON serialization. 2. **Sender** sends `transfer_staged` control message to DiffusionServer (metadata only, no tensor data). 3. **DiffusionServer** sends `transfer_alloc` to receiver → receiver allocates buffer slot → replies `transfer_allocated`. 4. **DiffusionServer** sends `transfer_push` to receiver with sender's address info. 5. **Receiver** pulls data via transfer engine (Mooncake RDMA or mock), sends `transfer_ready`. 6. **Receiver** loads tensors async on a dedicated transfer stream, overlapped with the previous request's compute. Decoder results (final output) flow back through DiffusionServer as raw ZMQ frames to the HTTP client. ### RDMA Flags
Flag Default Description
--disagg-p2p-hostname 127.0.0.1 RDMA-reachable hostname/IP of this instance
--disagg-ib-device None InfiniBand device (e.g., mlx5\_0, mlx5\_roce0)
--disagg-transfer-pool-size 256 MiB Pinned memory pool per instance
Set `--disagg-p2p-hostname` to the actual IP on each machine. For multi-machine, `--disagg-ib-device` specifies the RDMA NIC. ## Per-Role Parallelism
Flag Description
--encoder-tp Encoder tensor parallelism
--denoiser-tp / --denoiser-sp / --denoiser-ulysses / --denoiser-ring Denoiser parallelism
--decoder-sp Decoder sequence parallelism
If not specified, parallelism is auto-derived from `--num-gpus`. ## Other Options
Flag Default Description
--disagg-timeout 600 Timeout (seconds) for pending requests
--disagg-dispatch-policy round\_robin round\_robin or max\_free\_slots
## Python API For programmatic single-machine deployment, `launch_pool_disagg_server()` is available. It spawns its workers, and each child re-executes this script's module scope before initializing its platform, so keep the diffusion imports inside the guard: ```python theme={null} if __name__ == "__main__": from sglang.multimodal_gen.runtime.launch_server import launch_pool_disagg_server from sglang.multimodal_gen.runtime.server_args import ServerArgs server_args = ServerArgs.from_kwargs( model_path="Wan-AI/Wan2.1-T2V-14B-Diffusers", denoiser_sp=4, denoiser_ulysses=2, denoiser_ring=2, disagg_ib_device="mlx5_0", ) launch_pool_disagg_server( server_args, encoder_gpus=[[0]], denoiser_gpus=[[1, 2, 3, 4], [5, 6, 7, 8]], decoder_gpus=[[0]], ) ``` ## Architecture ``` Client ─── HTTP (port 30000) ──► FastAPI Server │ ▼ DiffusionServer (ROUTER, scheduler_port) ┌───────┼───────┐ PUSH work │ │ │ PUSH work ▼ │ ▼ Encoder[0..N] │ Decoder[0..K] │ │ ▲ P2P tensor │ │ │ P2P tensor transfer ▼ │ │ transfer Denoiser[0..M] ─────┘ │ PULL results ◄────┘ (decoder → DS → client) ``` ### Request State Machine ``` PENDING → ENCODER_WAITING → ENCODER_RUNNING → ENCODER_DONE │ DENOISING_WAITING → DENOISING_RUNNING → DENOISING_DONE │ DECODER_WAITING → DECODER_RUNNING → DONE ``` Any state can transition to `FAILED` or `TIMED_OUT`. # Inference Batching Source: https://docs.sglang.io/docs/sglang-diffusion/dynamic_batching Batch compatible native SGLang-Diffusion requests during serving. Dynamic batching is an opt-in SGLang-Diffusion serving mode that merges compatible queued requests into one native pipeline batch. It is separate from LLM continuous batching and tokenizer batching. Use it for concurrent T2I or T2V traffic with the same model and sampling shape. Keep singleton serving for latency-sensitive or highly mixed traffic. ## Enable Dynamic batching is disabled by default with `--batching-max-size 1`. For GLM-Image T2I with an external AR server, compatible requests share one batched AR `/generate` call. DiT denoising and VAE decoding remain independent per-request executions; GLM-Image does not batch DiT work. ```bash Command theme={null} sglang serve \ --model-path black-forest-labs/FLUX.1-dev \ --port 30010 \ --batching-mode dynamic \ --batching-max-size 8 \ --batching-delay-ms 5 \ --enable-batching-metrics ``` For request formats, see the [OpenAI-Compatible API](./api/openai_api). Use `--batching-config /path/to/batching_config.json` to load JSON rules when a model or resolution needs a lower cap than `--batching-max-size`: ```json Config theme={null} { "schema_version": 1, "rules": [ { "model_contains": "Qwen-Image", "resolution": "1024x1024", "max_batch_size": 1 } ] } ``` ## Compatibility An initial implementation of dynamic batching for T2I and T2V models can be found in [#18764](https://github.com/sgl-project/sglang/pull/18764). The current compatibility grid is below and will be updated as more coverage is added. See [Supported Models](./compatibility_matrix) for common model IDs. `✅` means supported, `❌` means not currently supported, `?` means untested, and `-` means not applicable. ### Image
Model T2I I2I
FLUX.1-dev✅-
FLUX.2-dev✅❌
FLUX.2-dev-NVFP4??
FLUX.2-Klein-4B✅❌
FLUX.2-Klein-9B??
FLUX.2-Klein-Base-4B??
FLUX.2-Klein-Base-9B??
Z-Image?-
Z-Image-Turbo✅-
GLM-Image (external AR)✅-
Qwen Image✅-
Qwen Image 2512✅-
Qwen Image 2.1✅❌
Qwen Image Edit-❌
Qwen Image Edit 2509-?
Qwen Image Edit 2511-?
Qwen Image Layered??
SD3 Medium?-
SD3.5 Medium?-
SD3.5 Large?-
Hunyuan3D-2?-
SANA 1.5 1.6B✅-
SANA 1.5 4.8B✅-
SANA 1600M 1024px?-
SANA 600M 1024px?-
SANA 1600M 512px?-
SANA 600M 512px?-
FireRed-Image-Edit 1.0-?
FireRed-Image-Edit 1.1-?
ERNIE-Image?-
ERNIE-Image-Turbo?-
Qwen Image 2.1 supports merging compatible text-to-image requests. Image edits are not merged across requests; `n > 1` still produces multiple outputs within one edit request. See its [cookbook](/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1#batching) for deployment settings and batching tradeoffs. ### Video
Model Support
FastWan2.1 T2V 1.3B✅
FastWan2.2 TI2V 5B Full Attn❌
Wan2.2 TI2V 5B❌
Wan2.2 T2V A14B✅
Wan2.2 I2V A14B❌
HunyuanVideo❌
FastHunyuan❌
Wan2.1 T2V 1.3B✅
Wan2.1 T2V 14B✅
Wan2.1 I2V 480P?
Wan2.1 I2V 720P?
TurboWan2.1 T2V 1.3B✅
TurboWan2.1 T2V 14B✅
TurboWan2.1 T2V 14B 720P✅
TurboWan2.2 I2V A14B?
Wan2.1 Fun 1.3B InP?
Helios Base?
Helios Mid?
Helios Distilled?
LTX-2?
LTX-2.3?
## Notes * Requests batch only when model inputs, sampling parameters, output handling, and any configured rules are compatible. * There is no startup probing, runtime learning, OOM retry, or automatic fallback to singletons. If a merged batch fails or cannot be split, every request in that batch receives an error. * Batch shape can change kernels, so singleton and dynamic outputs are not expected to be bit-exact. * Use `--enable-batching-metrics` to inspect realized batches: ```text theme={null} Dynamic batch dispatch: size=2/8, user_max=8, queue_wait=5.12ms, stop_reason=delay Dynamic batch dispatch: size=1/8, user_max=8, queue_wait=0.04ms, stop_reason=config_cap:1 Dynamic batch stats (last 5 dispatches): avg_size=2.80, merged_rate=60.0%, full_rate=20.0%, utilization=35.0%, wait_avg=3.21ms, wait_p95=5.12ms, top_rejects=none ``` # Encoder Parallelism Source: https://docs.sglang.io/docs/sglang-diffusion/encoder_parallel While the DiT denoises, the text and image encoders are idle — and while they encode, the whole DiT replica is idle. `--encoder-parallel` decides how native encoders use those otherwise-unused GPUs for the encoding stage. Folding applies to native text and image encoders; within-replica batch DP currently requires an explicitly supported native text encoder. ```bash theme={null} --encoder-parallel {auto,fold,dp,replicate} ``` | Mode | What it does | Use when | | - | - | - | | `auto` | Picks folding or the existing layout for native text/image encoders, and batch DP for supported native text encoders | Default; you want the decision made per encoder | | `fold` | TP-shards the encoder weights across the idle DiT replica | One wide encoder dominates a single-request encode | | `dp` | Supported native text encoder copies split the prompt batch, then all-gather their outputs inside the replica | Throughput serving with `--batching-max-size > 1` | | `replicate` | Keeps the encoder on its DiT TP group and encodes redundantly across the other replica ranks | You want to disable folding and batch DP | The two accelerated modes are mutually exclusive per encoder: folding shards the weights for the lifetime of the loaded model, so a folded encoder cannot also be data-parallel. ## Which Mode Wins Measured on H100 across T5 (hidden 4096), Qwen3 (2560), and CLIP-L (768) at batch 1–8 and replica sizes 2 and 4: * **Folding** pays when the encoder is wide enough that sharding its GEMMs beats the per-layer all-reduce it adds. T5 gains; Qwen3 (+35%) and CLIP-L (+50%) get slower, so folding is gated at hidden ≥ 4096. Its benefit also saturates as the replica grows, since each rank's slice keeps shrinking. * **Data-parallel** pays only when the encode is compute-bound, which needs a wide encoder (hidden ≥ 1024 — CLIP-L is slower at every batch and replica measured) and more than one prompt in a single encode call. * **Replication** is the right answer whenever neither condition holds, which is most single-request latency work. `auto` encodes exactly these rules, so prefer it unless you are pinning a configuration you measured yourself. ## Numerics `replicate` matches single-GPU encoding bit-for-bit only when the encoder TP degree is one. Both the existing DiT TP layout and `fold` can reorder parallel reductions; they are mathematically equivalent but are not generally bitwise identical to a single-GPU kernel. `dp` is also not bitwise-identical: each encoder copy sees a smaller batch, so GEMM tiling can differ from the unsplit reference. It may compose with encoder TP: all ranks in one TP group receive the same batch slice, and corresponding TP ranks gather outputs across the orthogonal encoder-DP group. The result remains mathematically equivalent and deterministic for a fixed topology and batch shape, but long video sampling can amplify small floating-point differences. ## Recommended Commands Throughput serving. A single encode call must carry more than one prompt for DP to engage, so raise the batching ceiling too; an encoder flag deliberately does not change DiT batching for you: ```bash theme={null} sglang serve \ --model-path Wan-AI/Wan2.2-TI2V-5B-Diffusers \ --model-type diffusion \ --num-gpus 2 \ --tp-size 1 \ --ulysses-degree 2 \ --encoder-parallel dp \ --batching-max-size 2 ``` Single-request latency with one wide text encoder: ```bash theme={null} sglang serve \ --model-path Wan-AI/Wan2.2-TI2V-5B-Diffusers \ --model-type diffusion \ --num-gpus 4 \ --ulysses-degree 4 \ --encoder-parallel fold ``` Bit-exact reproducibility against a single-GPU reference: ```bash theme={null} sglang serve \ --model-path Qwen/Qwen-Image-2512 \ --model-type diffusion \ --num-gpus 2 \ --encoder-parallel replicate ``` ## Interaction With Other Flags * **Tensor parallel**: encoder DP composes with TP. A TP group jointly encodes one batch slice; the orthogonal ranks inside the same pipeline replica split and gather the batch. A pure-TP replica has one encoder copy, so there is no additional batch-DP degree. * **Data parallel**: encoder collectives never cross pipeline replicas. With `--dp-size > 1`, each replica independently uses its own TP/SP/CFG ranks. * **Dynamic batching**: `dp` only pays with a wide batch. Selecting it does not change `--batching-max-size`; configure that separately. See [Inference Batching](./dynamic_batching). * **Sequence parallelism**: SP splits the DiT latent sequence. During encoding, those ranks either hold encoder copies for batch DP or join a folded encoder. See [Sequence Parallelism](./ring_sp_performance). # Environment Variables Source: https://docs.sglang.io/docs/sglang-diffusion/environment_variables Configure SGLang diffusion behavior with environment variables. ## Runtime
Environment Variable Default Description
SGLANG\_EXTERNAL\_MODEL\_PACKAGE not set Installed package that registers out-of-tree diffusion pipelines and component models. The package is imported once in every process.
SGLANG\_DIFFUSION\_PLATFORM\_OVERRIDE not set Select cpu, cuda, rocm, mps, npu, musa, or an installed sglang.multimodal\_gen.platforms entry-point name. XPU remains automatic-only. When unset, SGLang Diffusion detects installed platform plugins before trying built-in platforms.
SGLANG\_PLUGINS not set Comma-separated allowlist for installed sglang.multimodal\_gen.plugins hooks. When unset, all discovered diffusion hooks load.
SGLANG\_DIFFUSION\_TARGET\_DEVICE cuda Target device for inference (cuda, rocm, xpu, npu, musa, mps, cpu)
SGLANG\_DIFFUSION\_ATTENTION\_BACKEND not set Override attention backend via env var (e.g. fa, torch\_sdpa, sage\_attn)
SGLANG\_DIFFUSION\_ATTENTION\_CONFIG not set Path to attention backend configuration file (JSON/YAML)
SGLANG\_DIFFUSION\_STAGE\_LOGGING false Enable per-stage timing logs
SGLANG\_DIFFUSION\_SERVER\_DEV\_MODE false Enable dev-only HTTP endpoints for debugging
SGLANG\_DIFFUSION\_TORCH\_PROFILER\_DIR not set Directory for torch profiler traces (absolute path). Enables profiling when set
SGLANG\_DIFFUSION\_CACHE\_ROOT \~/.cache/sgl\_diffusion Root directory for cache files
SGLANG\_DIFFUSION\_CONFIG\_ROOT \~/.config/sgl\_diffusion Root directory for configuration files
SGLANG\_DIFFUSION\_LOGGING\_LEVEL INFO Default logging level
SGLANG\_DIFFUSION\_IPC\_A2A true Enable CUDA-IPC all-to-all for eligible same-host, peer-accessible TP1 + two-rank Ulysses groups. Unsupported topology falls back to NCCL; set 0 to force NCCL.
SGLANG\_DIFFUSION\_IPC\_A2A\_TIMEOUT\_MS 10000 Peer-wait timeout in milliseconds. This is a hang backstop, not a per-step budget; raise it only for known long stalls such as layerwise offload.
SGLANG\_DIFFUSION\_IPC\_A2A\_MAX\_BUFFERS 16 Maximum cached IPC staging-buffer shape pairs. Cap this for multi-resolution serving to bound permanent staging memory.
SGLANG\_DIFFUSION\_ULYSSES\_PIPELINE\_GROUPS -1 Head groups for pipelining the Ulysses exchange against dense attention over the copy engine. -1 picks the largest count that divides the heads per rank while each group still fills the GPU; 0 keeps the sequential exchange; N forces N groups. Results are bit-identical either way.
SGLANG\_USE\_RUNAI\_MODEL\_STREAMER true Use Run:AI model streamer for model loading
SGLANG\_DIFFUSION\_CONVROT\_INT8\_BACKEND auto Kernel backend for convrot\_int8: auto takes SGLang's JIT-compiled ConvRot ops where they build and run (CC 9.0, 10.0, 12.0, 12.1 with nvcc) and comfy\_kitchen otherwise; jit or comfy\_kitchen forces one.
SGLANG\_KITCHEN\_INT8\_MAX\_ROWS 8192 Max activation rows per convrot\_int8 fused GEMM on the comfy\_kitchen backend. Set 0 to disable row splitting.
SGLANG\_KITCHEN\_INT8\_MIN\_SPLIT\_N 8192 Minimum output features before convrot\_int8 (comfy\_kitchen backend) splits large-M GEMMs. Narrow outputs stay as a single call.
## Platform-Specific ### Apple MPS
Environment Variable Default Description
SGLANG\_USE\_MLX not set SRT only: enables the MLX serving backend. It has no effect on SGLang Diffusion, which uses PyTorch MPS.
### ROCm (AMD GPUs)
Environment Variable Default Description
SGLANG\_USE\_ROCM\_VAE false Use AITer GroupNorm in VAE for improved performance on ROCm
SGLANG\_USE\_ROCM\_CUDNN\_BENCHMARK false Enable MIOpen auto-tuning for VAE conv layers on ROCm
### Quantization
Environment Variable Default Description
SGLANG\_DIFFUSION\_FLASHINFER\_FP4\_GEMM\_BACKEND not set Optional FlashInfer FP4 GEMM backend override for diffusion NVFP4. When unset, SGLang defaults to flashinfer\_trtllm.
SGLANG\_DIFFUSION\_ENABLE\_W8A8\_FP8\_GEMM false Experimental opt-in for fused W8A8 FP8 GEMM in diffusion weight-only FP8 linears. When disabled, FP8 weights are dequantized to the compute dtype before matmul. Enabling this dynamically quantizes activations to FP8 and may change output quality.
SGLANG\_DIFFUSION\_ENABLE\_MXFP8\_ATTENTION false Enable Ascend MXFP8 FA for supported non-causal self-attention layers. Applies to online MXFP8Config and offline ModelSlim W8A8\_MXFP8 checkpoints. Unsupported calls continue to use the regular attention path.
SGLANG\_DIFFUSION\_MXFP8\_FA\_HEAD\_CHUNK\_SIZE 4 Maximum number of attention heads processed by each Ascend MXFP8 FA call. Smaller chunks may improve performance for large workloads but add kernel launches; the optimal value depends on the model and input shape. Set to 0 to disable head splitting.
## Caching Acceleration These variables configure caching acceleration for Diffusion Transformer (DiT) models. SGLang supports multiple caching strategies - see [caching documentation](./caching-acceleration) for an overview. ### Cache-DiT Configuration See [cache-dit documentation](./cache_dit) for detailed configuration.
Environment Variable Default Description
`SGLANG_CACHE_DIT_ENABLED` false Enable Cache-DiT acceleration
`SGLANG_CACHE_DIT_FN` 1 First N blocks to always compute
`SGLANG_CACHE_DIT_BN` 0 Last N blocks to always compute
`SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching
`SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
`SGLANG_CACHE_DIT_MC` 3 Max continuous cached steps
`SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
`SGLANG_CACHE_DIT_TS_ORDER` 1 TaylorSeer order (1 or 2)
`SGLANG_CACHE_DIT_DMD` false Enable the DMD (Dynamic Mode Decomposition) calibrator (mutually exclusive with TaylorSeer)
`SGLANG_CACHE_DIT_DMD_HISTORY` 6 DMD snapshot window length (5-6 typical; needs >= 4 uniformly spaced snapshots, otherwise DMD falls back to TaylorSeer)
`SGLANG_CACHE_DIT_DMD_RANK` 0 DMD SVD truncation rank (0 = automatic)
`SGLANG_CACHE_DIT_DMD_RIDGE` 1e-8 DMD Tikhonov regularization added to the inverted singular values
`SGLANG_CACHE_DIT_DMD_SVD_PRECISION` medium DMD SVD precision (low/medium/high)
`SGLANG_CACHE_DIT_SCM_PRESET` none SCM preset (none/slow/medium/fast/ultra)
`SGLANG_CACHE_DIT_SCM_POLICY` dynamic SCM caching policy
`SGLANG_CACHE_DIT_SCM_COMPUTE_BINS` not set Custom SCM compute bins
`SGLANG_CACHE_DIT_SCM_CACHE_BINS` not set Custom SCM cache bins
### Cache-DiT Secondary Transformer For dual-transformer models (e.g., Wan2.2 with high/low-noise experts), these variables configure caching for the secondary transformer. Each falls back to its primary counterpart if not set.
Environment Variable Default Description
SGLANG\_CACHE\_DIT\_SECONDARY\_FN (from primary) First N blocks to always compute
SGLANG\_CACHE\_DIT\_SECONDARY\_BN (from primary) Last N blocks to always compute
SGLANG\_CACHE\_DIT\_SECONDARY\_WARMUP (from primary) Warmup steps before caching
SGLANG\_CACHE\_DIT\_SECONDARY\_RDT (from primary) Residual difference threshold
SGLANG\_CACHE\_DIT\_SECONDARY\_MC (from primary) Max continuous cached steps
SGLANG\_CACHE\_DIT\_SECONDARY\_TAYLORSEER (from primary) Enable TaylorSeer calibrator
SGLANG\_CACHE\_DIT\_SECONDARY\_TS\_ORDER (from primary) TaylorSeer order (1 or 2)
SGLANG\_CACHE\_DIT\_SECONDARY\_DMD (from primary) Enable the DMD calibrator (mutually exclusive with TaylorSeer)
SGLANG\_CACHE\_DIT\_SECONDARY\_DMD\_HISTORY (from primary) DMD snapshot window length
SGLANG\_CACHE\_DIT\_SECONDARY\_DMD\_RANK (from primary) DMD SVD truncation rank (0 = automatic)
SGLANG\_CACHE\_DIT\_SECONDARY\_DMD\_RIDGE (from primary) DMD Tikhonov regularization term
SGLANG\_CACHE\_DIT\_SECONDARY\_DMD\_SVD\_PRECISION (from primary) DMD SVD precision (low/medium/high)
## Cloud Storage These variables configure S3-compatible cloud storage for automatically uploading generated images and videos.
Environment Variable Default Description
`SGLANG_CLOUD_STORAGE_TYPE` not set Set to `s3` to enable cloud storage
`SGLANG_S3_BUCKET_NAME` not set The name of the S3 bucket
`SGLANG_S3_ENDPOINT_URL` not set Custom endpoint URL (for MinIO, OSS, etc.)
`SGLANG_S3_REGION_NAME` us-east-1 AWS region name
`SGLANG_S3_ACCESS_KEY_ID` not set AWS Access Key ID
`SGLANG_S3_SECRET_ACCESS_KEY` not set AWS Secret Access Key
## CUDA Crash Debugging These variables enable kernel API logging and optional input/output dumps around diffusion CUDA kernel call boundaries. They are useful when tracking down CUDA crashes such as illegal memory access, device-side assert, or shape mismatches in custom kernels.
Environment Variable Default Description
SGLANG\_KERNEL\_API\_LOGLEVEL 0 Controls crash-debug kernel API logging. 1 logs API names, 3 logs tensor metadata, 5 adds tensor statistics, and 10 also writes dump snapshots.
SGLANG\_KERNEL\_API\_LOGDEST stdout Destination for crash-debug kernel API logs. Use stdout, stderr, or a file path. %i is replaced with the process PID.
SGLANG\_KERNEL\_API\_DUMP\_DIR sglang\_kernel\_api\_dumps Output directory for level-10 kernel API dumps. %i is replaced with the process PID.
SGLANG\_KERNEL\_API\_DUMP\_INCLUDE not set Comma-separated wildcard patterns for kernel API names to include in level-10 dumps.
SGLANG\_KERNEL\_API\_DUMP\_EXCLUDE not set Comma-separated wildcard patterns for kernel API names to exclude from level-10 dumps.
# Fused Kernels Source: https://docs.sglang.io/docs/sglang-diffusion/fused_kernels The fused CUDA/Triton kernels SGLang Diffusion ships, what each one replaces, and which are on by default. Diffusion transformers and VAEs spend a large share of their non-GEMM time on short elementwise and selection chains — adaLN modulate, residual gating, QK-norm, RoPE, norm epilogues, and MoE routing — each of which can require several kernel launches and HBM round trips in eager PyTorch. SGLang Diffusion replaces these chains with fused kernels under [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion). This page is an inventory: what each kernel fuses, what its numerical contract is, and which models use it. It is not a lever you tune — most of these kernels are on by default and require no flag. The one switch is `--quality`, described below. ## Numerical contracts and quality tiers Multi-step denoising amplifies a per-step rounding difference into visible quality loss, so "close enough" and "bit-exact" are different products here. The `quality` switch distinguishes unconditional bit-exact replacements from non-bit-exact eager-chain fusions: **Bit-exact — mounted unconditionally.** The kernel reproduces every rounding boundary of the eager chain, so `torch.equal` holds against the reference. Some go quite far to get there: the fused LayerNorm+modulate kernel replicates PyTorch's `vectorized_layer_norm_kernel` down to its Welford update order, guarded reciprocal, and warp-fold tree; the fused RMSNorm+scale/shift kernel replicates FlashInfer's CuTe-DSL `RMSNormKernel` fragment order and `shfl.bfly` fold. Because the dispatch they replicate can change underneath them, each one still verifies itself against the live eager chain on first sight and falls back permanently on any mismatch. **Same math, different rounding — request-gated at `lossless`.** These keep the reference function and the precision of every operand and accumulator; only the place or the order of the rounding moves, so the result is within rounding error of the reference rather than equal to it. They are mounted for `quality="lossless"` (the default) and `quality="high"` requests, at batch boundaries, all-or-nothing per transformer. `quality="exact"` runs the unmodified reference chain instead. **Approximate — request-gated at `high`.** A fusion that quantizes, lowers an operand or accumulator precision, or sits upstream of a quantizer or threshold is no longer a rounding difference, so it needs model-level quality evaluation and is mounted only for `quality="high"`. Today that is the FLUX.2 NVFP4 FC1+SwiGLU+quant path (it quantizes FC2's input before the reference BF16 intermediate exists), Ideogram's BF16-native gate RMSNorm (the reference keeps the norm statistics in fp32), and SANA-Video's BF16-input linear attention (the reference promotes those inputs to FP32). **Model/checkpoint-native.** Generic close-contract kernels, sparse operators, and FP8/NVFP4 producers can be part of a model implementation or a separately selected deployment path. They are documented in the inventory, but `quality` does not select or undo those choices. **Selection-equivalent routing — enabled unconditionally.** LingBot Video's fused group-limited top-k returns the same selected expert-id set as its guarded CUDA `torch.topk(..., sorted=False)` reference chain. The order of those ids is not part of either path's contract. Because the selected experts are unchanged, this path does not depend on the request `quality` tier. A plain fp32 single-pass norm fusion looks harmless and is not. On ERNIE-Image it moved the 50-step trajectory to 18.83 dB PSNR, which is what motivated the bit-exact rewrite of that path. The quality levels are cumulative: | `quality` | Included optimization set | | - | - | | `exact` | The selected deployment's reference path plus every unconditional bit-exact replacement | | `lossless` | Everything in `exact`, plus request-gated DiT and VAE fusions that keep the reference math and every operand's precision; this level does not itself enable quantization, sparsity, caching, or another approximate path | | `high` | Everything in `lossless`, plus any model-owned approximate optimization, such as an audited Cache-DiT policy, lower-precision VAE decode, or the approximate fusions listed above | A fusion claiming `lossless` has to pass the admission checks in `sglang/multimodal_gen/test/quality_tier_admission.py`: scored against an fp64 evaluation of the same math, its RMS and max error may not exceed 1.5x the path the model runs today, its rounding may not be biased, and NaN, Inf and non-contiguous inputs have to behave as before. Lowering an accumulation below fp32, or quantizing, makes it `high` by construction — no measurement admits it. If a model has no eligible request-gated fusion, `lossless` can execute the same path as `exact`. Likewise, `high` adds only the model-specific high-only paths that the active pipeline implements. `quality` is not a master precision switch. A quantized checkpoint, an explicitly selected approximate attention backend, or an independently enabled cache remains active at every quality tier. ## Enabling the request-gated set ```bash theme={null} sglang generate --model-path MODEL_PATH --prompt "..." --quality lossless ``` The server default is `lossless`, so this is the set a request gets unless it asks otherwise; the OpenAI-compatible endpoints carry the level per request. Images: ```bash theme={null} curl -X POST http://${HOST}:${PORT}/v1/images/generations \ -H 'Content-Type: application/json' \ -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "lossless"}' ``` Video, same field: ```bash theme={null} curl -X POST http://${HOST}:${PORT}/v1/videos \ -H 'Content-Type: application/json' \ -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "lossless"}' ``` The `quality` field in a **video response** body is unrelated. It is Sora-compatible response metadata and is always reported as `"standard"`; it does not reflect the sampling quality the request ran with. `quality` participates in the dynamic-batch signature, so mixed-quality traffic is batched separately and the transition happens safely at a batch boundary. Mounting is all-or-nothing: if any marked site on a transformer fails its static guards, no site on that transformer is fused. Do not combine request-gated DiT fusions with `--enable-breakable-cuda-graph`. BCG warmup captures the lossless module branches before an `lossless` or `high` request mounts its DiT fusions, so replay would bypass the requested kernels. SGLang rejects this combination for models with eligible DiT quality sites. Models whose request-gated path changes only VAE decode remain allowed because BCG captures the DiT only. These fusion families mount under both `quality="lossless"` and `quality="high"`: | Fusion | What it folds | | - | - | | Linear + tanh-GELU | Bias-add and GELU into the GEMM epilogue (cublasLt), removing the `[tokens, 4*dim]` intermediate round trip | | Wan NVFP4 linear + GELU | Bias-add and GELU fused into Wan's NVFP4 FFN projection output | | Qwen-Image added-QKV | Added Q/K/V projections fused into joint-buffer production | | LayerNorm + modulate | `layer_norm(x, weight=(1 + scale), bias=shift)` in place of affine-free LN plus a separate modulate | | LTX-2 RMSNorm + modulate | `rms_norm(x) * (1 + scale) + shift` in one launch | | Gate RMSNorm (BF16-native) | `RMSNorm + tanh + mul + add` in one pass | | HunyuanVideo strided QK RMSNorm | Per-head QK RMSNorm over the packed QKV layout | | LingBot Video fused RMSNorm | Replaces the handwritten cast, square, mean, rsqrt, and multiply chain with existing Triton RMSNorm kernels | | LingBot Video per-token gated residual + RMSNorm modulate | Folds `residual + gate * update` (per-token `[B, S, 1]` gate) and the `rmsnorm(x) * (1 + scale) + shift` adaLN chain (strided `[B, S, 6D]` chunk views) into single kernels | | SANA-Video BF16-input linear attention | Keeps the first linear-attention GEMM's inputs in BF16 with FP32 accumulation/output; the second GEMM remains FP32 | | Dynamic LoRA delta | An unmerged adapter's scaled update accumulated onto the base output by its second GEMM (`addmm`, one rounding), and a row-parallel layer's bias added in the base GEMM's epilogue, instead of materializing the `[tokens, out]` delta and scaling and adding it in separate passes | | FLUX-family VAE fast paths | Channels-last decode, GroupNorm(+SiLU), upsample, and attention replacements for FLUX.2 and AutoencoderKL-based FLUX.1, Z-Image, and SD3 pipelines | | Wan VAE RMSNorm + SiLU | Replaces the channel-first RMSNorm/SiLU chain while keeping the decode in `channels_last_3d` | | MiniMax-H3 VAE decoder fast path | Per-head fp32 RMSNorm, rounded to the activation dtype, and NeoX RoPE in one launch over the strided Q/K views of the ViT decoder's QKV buffer; cuDNN SDPA for the block's attention | ## Kernel inventory 46 operators are registered in the kernel registry across 52 implementations (some operators carry several backends). Backends are named by provenance, not device: `KDA` identifies Kernel Design Agents implementations, `JIT` compiles under nvcc *and* hipcc, `TRITON` identifies Triton sources, `CUTE_DSL` needs CUTLASS, `FLYDSL` is ROCm gfx950 only, and `AOT` comes from the `sgl_kernel` wheel. Per-operator capability metadata determines which devices can load each implementation. ### Normalization | Operator | Backend | Contract | Replaces | | - | - | - | - | | `rmsnorm_scale_shift` | Triton | bit-exact | RMSNorm + `* (1 + scale) + shift` (4 kernels) | | `scale_residual_norm_scale_shift` | KDA / Triton / CuTe-DSL / FlyDSL | bit-exact (Triton) or backend-specific close contract | the above plus the preceding `residual + gate * update` | | `scale_residual_norm_scale_shift_nvfp4` | JIT CUDA | matches the selected NVFP4 producer contract | Qwen residual LayerNorm/modulation + FC1 NVFP4 quantization | | `layernorm_modulate` | Triton | bit-exact | affine-free LayerNorm + adaLN modulate | | `try_fused_fp32_layernorm_bf16` | KDA / Triton | bit-exact against the verified native dispatch | LingBot World causal FP32 LayerNorm on Hopper with FP32 affine or adaLN post-ops, including BF16 input/output conversions; other devices retain native LayerNorm | | `qk_head_layernorm` | Triton | bit-exact | per-head LayerNorm on q/k | | `qk_rmsnorm_native` | Triton | bit-exact | Z-Image per-head QK RMSNorm | | `norm_scale_shift` | KDA / CuTe-DSL / FlyDSL | backend-specific close contract | LN-or-RMS + scale/shift, many broadcast modes | | `rmsnorm_scale`, `rmsnorm_tanh_residual` | Triton | bf16-native statistics | `RMSNorm(x) * scale`, `x + tanh(gate) * RMSNorm(y)` | | `apply_group_norm_silu` | Triton | close | `GroupNorm + SiLU`, NCHW-contiguous | | `group_norm_silu_4d`, `group_norm_silu_rows` | Triton | close | channels-last GroupNorm(+SiLU); what lets a VAE decoder run channels\_last end to end with no `nchwToNhwc` transposes | | `wan_rmsnorm_silu` | Triton | close | Wan VAE `channels_last_3d` RMSNorm + SiLU | | `wan_norm_silu_post` | JIT CUDA | bit-exact, first-sight verified | Wan VAE FP32 divide, scale, affine, and SiLU after the unchanged native L2 reduction; dense NCDHW and `channels_last_3d` | ### adaLN modulation and gating | Operator | Backend | Contract | Replaces | | - | - | - | - | | `modulate_scale_shift` | JIT CUDA | bit-exact | `x * (1 + scale) + shift` | | `residual_gate_add` | KDA (JIT CUDA) | bit-exact | `residual + gate * update` | | `try_fused_scaled_residual_bf16` | Triton | bit-exact against the verified native dispatch | LingBot World causal BF16 residual with an FP32 per-frame gate; skips the unused normalized output before camera conditioning | | `timestep_embedding` | JIT CUDA | close | sinusoidal timestep embedding | | `temb_table_slices` | Triton | bit-exact | see note below | | `ltx2_ada_values` | Triton | bit-exact | LTX-2 nine-way adaLN value split, slices come out contiguous | LingBot World camera conditioning also uses `modulate_scale_shift` with one affine row per token. The causal normalization and residual paths verify each new signature outside CUDA graph capture and fall back to their native chains on mismatch or unsupported layouts. These paths preserve `lossless` arithmetic; kernel replay support does not enable BCG for a model configuration that disables it. `temb_table_slices` is worth knowing about. The eager `(scale_shift_table + temb.float()).chunk(6, dim=2)` materializes roughly 8 GB of fp32 at 704p/121f *and* hands six **strided** slices downstream, whose `.contiguous()` calls then copy each one again. The fused kernel produces the six slices in one pass, each naturally contiguous, so the downstream copies become no-ops. ### RoPE and QK-norm | Operator | Backend | Contract | Replaces | | - | - | - | - | | `fused_inplace_qknorm_rope` | JIT CUDA | one bf16 rounding step vs the split baseline; exact with `round_norm_before_rope=True` | separate QK-norm kernel + RoPE | | `flux2_qkv_epilogue` | KDA (JIT CUDA) | bit-exact against its selected BF16 reference chain | FLUX.2 QK RMSNorm + RoPE + joint text/image QKV packing | | `flux2_strided_qknorm_rope` | Triton | bit-exact against native tiled RMSNorm followed by FlashInfer interleaved RoPE; verified per signature | FLUX.2 Klein single-stream packed Q/K copies + QK RMSNorm + RoPE on Hopper, BF16, 24 heads of dimension 128 | | `qwen_qkv_epilogue` | JIT CUDA | bit-exact against its selected BF16 reference chain | Qwen-Image QK RMSNorm + RoPE + joint QKV writes on SM100+ | | `rope_rotate_half` | Triton | bit-exact | `chunk` → `cat(-x2, x1)` → two muls + add → `cat(tail)`, about 7 kernels per projection | | `interleaved_rope_fp64` | JIT CUDA | bit-exact | paired SANA-Video Q/K RoPE with fp64 tables, about 14 eager kernels | | `helios_qk_rope` | JIT CUDA | bit-exact | paired in-place Helios Q/K RoPE with transposed frequency layout | | `ltx2_qknorm_split_rope` | KDA (JIT CUDA) | close (validated on B200) | LTX-2 QK-norm + split RoPE | | `ltx25_decoder_rope` | JIT CUDA | bit-exact | paired LTX-2.5 decoder 3D RoPE from cached compact axis tables | | `hunyuan_qkv_rope_pack` | Triton | bit-exact | QKV pack and RoPE in one pass | The Klein strided path preserves the BF16 rounding between RMSNorm and RoPE, including signed zeros. It reads packed Q/K views without changing the adjacent V or MLP values. Unsupported layouts, devices, or RMSNorm dispatches use the original helper. A new signature is verified outside CUDA Graph capture; an unverified capture falls back. `SGLANG_ENABLE_FUSED_QKNORM_ROPE=0` also disables this path. This kernel does not enable model-level BCG support for FLUX.2. ### Activation | Operator | Backend | Contract | Replaces | | - | - | - | - | | `silu_mul` | Triton | bit-exact | `F.silu(a) * b` for split-projection SwiGLU, where the concatenated `silu_and_mul` kernels do not apply without an extra full-width `cat` | | `bias_silu`, `bias_glu` | Triton | bit-exact | Sana GLUMB conv bias + SiLU / GLU post-processing | | `linear_gelu_tanh` | AOT (cublasLt) | not bit-exact, request-gated | bias-add and tanh-GELU folded into the GEMM epilogue | ### Attention | Operator | Backend | Notes | | - | - | - | | `sparse_linear_attn_fwd` | Triton | block-map, compression, and forward for sparse linear attention | | `bigdn` | Triton | Sana-WM bidirectional gated delta-net; the chunkwise form splits phase A along the KV and Z streams so two blocks stay resident per SM, and stores `(I - P)` so phase B's MMA folds the identity-add in | | `vsa_block_sparse_sm100` | JIT CUDA | FastVideo's warp-specialized tcgen05 block-sparse VSA forward for 64-token tiles (sm\_100a / sm\_103a); a CTA owns two adjacent query tiles with their own key lists. VSA-H3 uses it on Blackwell and keeps the Triton tile-64 kernel elsewhere | ### MoE routing | Operator | Backend | Contract | Replaces | | - | - | - | - | | `group_limited_topk` | Triton | selected expert-id set matches the guarded CUDA reference; output order is unspecified | LingBot Video's per-group top-2 reduction, group top-k, mask construction, masked expert scores, and final expert top-k | ### Data movement Every kernel here only moves values (plus zero fill, plus at most one same-order add), so each is bitwise identical to the aten chain it replaces. | Operator | Backend | Replaces | | - | - | - | | `usp_merge_heads` | JIT CUDA | USP all-to-all output head merge (`permute` + `contiguous`) | | `pack_qkv_destination_major` | Triton | Ulysses destination-major QKV pack | | `varlen_pack_qkv`, `varlen_scatter_to_padded` | Triton | varlen gather/scatter around the masked attention path | | `varlen_pack_segmented_qkv` | Triton | varlen gather from a virtual prefix/main Q/K/V sequence | | `joint_qkv_cat` | Triton | Joy Image Edit's three image/text Q/K/V concatenations, including strided packed-V inputs | | `causal_conv3d_cat_pad` | KDA (JIT CUDA) / Triton | causal Conv3d `cat` + `pad` | | `cat_pad_channels_last_3d` | Triton | Wan causal VAE `cat + F.pad + contiguous` (three passes plus cache bookkeeping) in one pass | | `dup_up3d_add` | Triton | `repeat_interleave + permute().contiguous() + add` | Joy Image Edit uses the joint-copy path for eligible CUDA FP16/BF16 image tensors of at least 32 MiB. It preserves image-first token order, copies values without arithmetic, and verifies each new shape/stride signature against the native concatenations before enabling it. Small inputs, unsupported layouts, gradient-bearing inputs, and unverified signatures during graph capture use the native path. This does not enable model-level BCG support. For large Hopper BF16 image Q/K with 32 heads of width 128, Joy also uses the existing out-of-place QK-Norm + RoPE kernel to read the packed projection directly. This removes two input copies while retaining the original CUDA arithmetic and contiguous outputs. Each new signature is checked bitwise; inputs remain intact if the operation fails. Other shapes and platforms, compilation, unverified capture, and `SGLANG_ENABLE_FUSED_QKNORM_ROPE=0` retain the original helper. ### Quantized layout producers These kernels preserve the quantized checkpoint path's selected reference operation. They are not a claim that FP8 or NVFP4 is equivalent to an unquantized BF16 checkpoint. | Operator | Backend | Replaces | | - | - | - | | `flux2_layernorm_modulate_fp8_quant` | KDA (Triton) | FLUX.2 LayerNorm plus adaLN modulation directly into static FP8 output | | `flux2_token_cat_fp8` | KDA (Triton) | FLUX.2 single-block attention/MLP concatenation plus static FP8 quantization | | `flux2_token_cat_nvfp4` | JIT CUDA | FLUX.2 single-block attention/MLP concatenation plus NVFP4 quantization | ## Coverage by model Kernels are written against a specific eager chain in a specific model, so coverage is per-model rather than universal. | Model | Fused paths | | - | - | | FLUX.1 | LN+modulate, modulate, residual-gate add, linear+GELU | | FLUX.2 | LN+modulate, fused LN+modulate-to-FP8, packed SwiGLU, gated residual/norm, residual-gate add, QK RMSNorm+RoPE+joint QKV packing, FP8/NVFP4 token-cat producers | | Qwen-Image | linear+GELU, select-0/1 LN modulation, added-QKV fusion, QK RMSNorm+RoPE+joint QKV writes, residual norm/modulate+NVFP4 producer | | GLM-Image | LN+modulate, per-head qk LN, residual-gate add, linear+GELU | | ERNIE-Image | RMSNorm+scale/shift, residual-gated variant, rotate-half RoPE, residual-gate add | | Joy Image Edit | Bitwise image/text QKV concatenation; strided image QK-Norm + RoPE on Hopper | | Z-Image | BF16-native RMSNorm scale / tanh-residual, per-head QK RMSNorm | | Ideogram 4 | gate RMSNorm, SwiGLU, rotate-half RoPE, modulate, residual-gate add | | LTX-2 | QK-norm + split RoPE, ada-values split, RMSNorm+modulate, modulate, residual-gate add, linear+GELU | | LTX-2.5 decoder | paired 3D RoPE with shared axis-table cache | | HunyuanVideo / Helios | QKV+RoPE pack, strided QK RMSNorm, linear+GELU; Helios also has paired in-place Q/K RoPE | | LingBot Video MoE | Default-on group-limited top-k expert selection; fused RMSNorm, per-token gated residual, and fused RMSNorm+modulate at `quality=lossless` or `quality=high` | | Sana | LN+modulate, GLUMB bias+SiLU / bias+GLU, residual-gate add | | SANA-Video | Packed QKV/KV; paired fp64 interleaved RoPE; GLUMB bias+SiLU / bias+GLU in eager and BCG; LN+modulate and residual-gate add during BCG; BF16-input linear attention at `quality=lossless` or `quality=high` | | Sana-WM | bidirectional gated delta-net, fused QK inverse-RMS | | Wan / LongLive 2 VAE | temb table slices (Wan); VAE cat+pad and DupUp3D add; lossless FP32 normalization post-ops; full `channels_last_3d` RMSNorm+SiLU at `quality=lossless` or `quality=high` | | Cosmos3 | QK-norm + RoPE + KV packing; rounded T2I fusion also supports Hopper TP2/SP1 Super-Text2Image with the 5120-wide SwiGLU architecture | | Krea2 / MiniMax-H3 | QK-norm + RoPE (Krea2 also CuTe-DSL norm+scale/shift; MiniMax-H3 also indexed modulation) | | FLUX.2 VAE / HunyuanVAE / latent upsampler | GroupNorm + SiLU (channels-last two-pass for FLUX.2) | ## Inspecting what is registered Every kernel is described by a `KernelSpec` in the process-wide registry, so the inventory is queryable without importing any backend: ```python theme={null} from sglang.kernels.registry import registry diffusion_ops = [op for op in registry.ops() if op.startswith("diffusion.")] for spec in registry.get("diffusion.scale_residual_norm_scale_shift"): print(spec.backend, spec.target, spec.capabilities) ``` Registration is metadata only — it imports neither torch nor a backend and triggers no JIT build. To pick a specific implementation of an operator that has several: ```python theme={null} from sglang.kernels import select_kernel, KernelBackend fn = select_kernel( "diffusion.scale_residual_norm_scale_shift", backend=KernelBackend.CUTE_DSL ).load() ``` ## Importing the kernels Runtime code imports from the package, never from a submodule: ```python theme={null} from sglang.kernels.ops.diffusion import fused_rmsnorm_scale_shift_bitexact ``` Resolution is lazy: the backends have disjoint, heavy dependencies (Triton, CUTLASS/CuTe-DSL, and FlyDSL on ROCm), so an eager re-export would make every one of them an import-time requirement on every platform. Each public kernel is a predicate-plus-kernel pair — call `can_use_(...)` first and fall back to the reference chain when it returns `False`; the kernel raises on an unsupported input rather than silently returning `None`. The package `README.md` carries a selection matrix for the cases where several kernels look interchangeable and are not. The normalization domain alone holds more than a dozen implementations that differ by numerical contract, activation layout, and backend rather than by speed. ## References * [Performance Optimization](/docs/sglang-diffusion/performance-optimization) * [Attention Backends](/docs/sglang-diffusion/attention_backends) * [Quantization](/docs/sglang-diffusion/quantization) * [Profiling](/docs/sglang-diffusion/profiling) * [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion) — source and selection matrix * [RFC #29630](https://github.com/sgl-project/sglang/issues/29630) — the unified `sglang.kernels` namespace # SGLang Diffusion Source: https://docs.sglang.io/docs/sglang-diffusion/index Accelerated image and video generation with diffusion models. SGLang Diffusion is SGLang's built-in inference engine for image and video generation. It ships in the same [repository](https://github.com/sgl-project/sglang) and `sglang` Python package, using the shared `sglang generate` and `sglang serve` commands. It provides native SGLang pipelines, diffusers backend support, an OpenAI-compatible server, and an optimized kernel stack built on both precompiled `sgl-kernel` operators and JIT kernels for key inference paths. ## Key Features * Broad model support across Wan, Hunyuan, Qwen-Image, FLUX, Z-Image, GLM-Image, and more * Fast inference with `sgl-kernel`, JIT kernels, scheduler improvements, and caching acceleration * Multiple interfaces: `sglang generate`, `sglang serve`, and an OpenAI-compatible API * Multi-platform support for NVIDIA, AMD, Intel XPU, Ascend, Apple Silicon, and Moore Threads ## Quick Start Start with the [recommended Docker setup](/docs/sglang-diffusion/installation) for Linux GPU deployments. The installation guide also covers pip/uv, source installation, and other platforms. Run the following commands inside the container or your activated Python environment. Generate an image: ```bash theme={null} sglang generate --model-path Qwen/Qwen-Image \ --prompt "A beautiful sunset over the mountains" \ --save-output ``` Or start an HTTP server: ```bash theme={null} sglang serve --model-path Qwen/Qwen-Image --host 0.0.0.0 --port 30000 ``` ## Start Here * [Installation](/docs/sglang-diffusion/installation): install SGLang Diffusion and platform dependencies * [Supported Models](/docs/sglang-diffusion/compatibility_matrix): browse supported model families, tasks, and public checkpoints * [CLI](/docs/sglang-diffusion/api/cli): run one-off generation jobs or launch a persistent server * [OpenAI-Compatible API](/docs/sglang-diffusion/api/openai_api): send image and video requests to the HTTP server * [ComfyUI plugin](/docs/sglang-diffusion/comfyui): use SGLang from ComfyUI in server mode or as a per-step DiT backend * [Performance Overview](/docs/sglang-diffusion/performance-optimization): choose speed, memory, parallelism, caching, and quality-tradeoff levers * [Caching Acceleration](/docs/sglang-diffusion/caching-acceleration): use Cache-DiT, TeaCache, or Spectrum to reduce denoising cost * [Quantization](/docs/sglang-diffusion/quantization): configure component checkpoint and causal KV-cache quantization * [Realtime and Causal Video Models](/docs/sglang-diffusion/realtime_models): understand session state, causal caches, and realtime-only controls * [Contributing](/docs/sglang-diffusion/contributing): contribution workflow, adding new models, and CI perf baselines ## Additional Documentation * [Post-Processing](/docs/sglang-diffusion/api/post_processing): frame interpolation and upscaling * [Models with AR Stage](/docs/sglang-diffusion/models_with_ar): run hybrid diffusion pipelines like GLM-Image with a separate AR encoder server * [Models with Prompt Enhancement](/docs/sglang-diffusion/models_with_pe): run ERNIE-Image with either built-in PE or a separate PE server * [Deployment and Performance Modes](/docs/sglang-diffusion/deployment_cookbook): choose `--performance-mode`, offload, FSDP, CFG parallelism, SP, and TP * [Attention Backends](/docs/sglang-diffusion/attention_backends): choose the best backend for your model and hardware * [Sequence Parallelism](/docs/sglang-diffusion/ring_sp_performance): configure SP, Ulysses, and ring-based splitting for long sequences * [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel): fold, data-parallel, or replicate the text/image encoders across idle GPUs * [Inference Batching](/docs/sglang-diffusion/dynamic_batching): batch compatible native diffusion requests during serving * [Progressive Resolution Generation](/docs/sglang-diffusion/progressive_resolution): run early denoising steps at lower latent resolution for selected pipelines * [Environment Variables](/docs/sglang-diffusion/environment_variables): platform, caching, storage, and debugging configuration ## Developer Documentation * [Support New Models](/docs/sglang-diffusion/support_new_models): implementation guide for new diffusion pipelines * [CI Performance Baselines](/docs/sglang-diffusion/ci_perf): generate and update performance baselines used in CI ## References * [SGLang GitHub](https://github.com/sgl-project/sglang) * [Cache-DiT](https://github.com/vipshop/cache-dit) * [FastVideo](https://github.com/hao-ai-lab/FastVideo) * [xDiT](https://github.com/xdit-project/xDiT) * [Diffusers](https://github.com/huggingface/diffusers) # Install SGLang Diffusion Source: https://docs.sglang.io/docs/sglang-diffusion/installation Run SGLang's built-in diffusion engine with Docker, or install the diffusion extra with pip, uv, or from source. SGLang Diffusion is SGLang's built-in image and video generation engine. It ships in the same [repository](https://github.com/sgl-project/sglang) and **`sglang` Python package**, using `sglang generate` and `sglang serve`. Docker is the recommended setup for Linux GPU deployments. The official NVIDIA images include SGLang, its diffusion dependencies, and the optimized kernel stack; no separate Python installation or `pip install` inside the container is needed for the standard diffusion runtime. Optional backends and model-specific dependencies are documented in the [cookbook](/cookbook/diffusion/intro). For Python installation, use `sglang[diffusion]`: `[diffusion]` adds optional dependencies to the same package, not a separate engine. ## Install on NVIDIA GPUs Use Linux with a supported NVIDIA GPU, a driver compatible with the image's CUDA version, [Docker Engine](https://docs.docker.com/engine/install/), and [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). Configure Docker's GPU runtime before continuing. See the [SGLang installation guide](/docs/get-started/install) for current CUDA requirements. The example uses the latest stable release. For reproducible deployments, replace `latest` with a release tag or image digest from [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang/tags). For models or features not yet released, choose a [nightly image](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly) containing the required changes, or install from source. ```bash theme={null} docker pull lmsysorg/sglang:latest ``` Start a container with persistent model and output directories. The working directory is `/outputs`, so generated files remain in the host's `outputs` directory after the container exits. The server port is published only on the host's loopback interface. ```bash theme={null} mkdir -p "$HOME/.cache/huggingface" "$PWD/outputs" docker run --rm -it --gpus all \ --ipc=host \ -p 127.0.0.1:30000:30000 \ -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \ -v "$PWD/outputs:/outputs" \ -w /outputs \ lmsysorg/sglang:latest bash ``` For gated or private models, export your [Hugging Face token](https://huggingface.co/docs/hub/en/security-tokens) as `HF_TOKEN` on the host and add `--env HF_TOKEN` before the image name. Mount any local input media or checkpoints into the container and use their container paths in requests. Run the commands in [Generate or serve](#generate-or-serve) inside this container. For development tools and source changes, see the [development environment guide](/docs/developer_guide/development_guide_using_docker#setup-docker-container). Use this option for an existing Python environment. Follow the [Python and CUDA prerequisites](/docs/get-started/install), then install the diffusion extra in a virtual environment: ```bash theme={null} pip install uv uv venv --python 3.12 source .venv/bin/activate uv pip install "sglang[diffusion]" --prerelease=allow ``` With pip, use `pip install "sglang[diffusion]" --pre` in your activated environment. Use source installation to develop SGLang or try changes that are not in a published image. Follow the [Python and CUDA prerequisites](/docs/get-started/install). The commands below install the current `main`; check out a release tag or commit before installation when you need a fixed revision. ```bash theme={null} git clone https://github.com/sgl-project/sglang.git cd sglang pip install uv uv venv --python 3.12 source .venv/bin/activate uv pip install -e "python[diffusion]" --prerelease=allow ``` The diffusion implementation lives in [`python/sglang/multimodal_gen`](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen). ## Generate or serve Run these commands inside the container or your activated Python environment. For a one-off image: ```bash theme={null} sglang generate --model-path Qwen/Qwen-Image \ --prompt "A beautiful sunset over the mountains" \ --save-output ``` Alternatively, start an HTTP server. Binding to `0.0.0.0` makes the published container port reachable from the host: ```bash theme={null} sglang serve --model-path Qwen/Qwen-Image --host 0.0.0.0 --port 30000 ``` Choose a model and GPU configuration from the [cookbook](/cookbook/diffusion/intro). See the [OpenAI-compatible API](/docs/sglang-diffusion/api/openai_api) to send requests. ## AMD GPUs (ROCm) Docker is also recommended for AMD GPUs. Use a ROCm image matched to your GPU and host driver, not the NVIDIA `latest` image. Follow [AMD GPUs](/docs/hardware-platforms/amd_gpu) for the current image tags, device passthrough flags, and source installation options. For **Strix Halo / gfx1151**, use `rocm/sgl-dev:*-gfx1151*` (image sets `SGLANG_USE_AITER=0`) and pass `--attention-backend torch_sdpa --num-gpus 1`. Do not use AITER on wave32 gfx115x. See the Strix Halo section in [AMD GPUs](../hardware-platforms/amd_gpu). ## Platform-Specific: MUSA (Moore Threads GPUs) ## Moore Threads GPUs (MUSA) For Moore Threads GPUs (MTGPU) with the MUSA software stack, follow the platform guide first. If the source tree still requires the alternate platform `pyproject` fallback, keep a backup of the default file before switching: ```bash Command theme={null} # Clone the repository git clone https://github.com/sgl-project/sglang.git cd sglang # Install the Python packages pip install --upgrade pip mv python/pyproject.toml python/pyproject.toml.bak cp python/pyproject_other.toml python/pyproject.toml pip install -e "python[all_musa]" ``` ## Intel XPU For Intel Data Center GPU Max or Arc GPUs, follow the Docker instructions in the [XPU installation guide](/docs/hardware-platforms/xpu). The Dockerfile already includes diffusion dependencies. ## Ascend NPU For Ascend NPU, please follow the [NPU installation guide](/docs/hardware-platforms/ascend-npus/getting-started/installation). Quick test: ```bash Command theme={null} sglang generate --model-path black-forest-labs/FLUX.1-dev \ --prompt "A logo With Bold Large text: SGL Diffusion" \ --save-output ``` ## Platform-Specific: Apple MPS For Apple MPS, install natively on macOS using the source instructions below; the Linux Docker image is not an MPS runtime. If the source tree still requires the alternate platform `pyproject` fallback, keep a backup of the default file before switching: ```bash Command theme={null} # Install ffmpeg brew install ffmpeg # Install uv brew install uv # Clone the repository git clone https://github.com/sgl-project/sglang.git cd sglang # Create and activate a virtual environment uv venv -p 3.12 sglang-diffusion source sglang-diffusion/bin/activate # Install the Python packages uv pip install --upgrade pip mv python/pyproject.toml python/pyproject.toml.bak cp python/pyproject_other.toml python/pyproject.toml uv pip install -e "python[all_mps]" ``` SGLang Diffusion uses PyTorch MPS. The `all_mps` extra also installs the SRT MLX backend dependencies; `SGLANG_USE_MLX` applies only to SRT serving. # Diffusion models with autoregressive stages Source: https://docs.sglang.io/docs/sglang-diffusion/models_with_ar Run diffusion pipelines with in-process or separately deployed autoregressive encoders. SGLang Diffusion supports two AR execution paths. Qwen Image Layered and LongCat-Image use the native Qwen2.5-VL component in process. GLM-Image can use its bundled Transformers implementation or delegate AR inference to a separate SGLang server. ## GLM-Image Quick Start Run GLM-Image with its bundled Transformers implementation (default): ```bash theme={null} # Terminal 1 : launch server sglang serve --model-path zai-org/GLM-Image --port ${PORT} ``` ```bash theme={null} # Terminal 2 : launch client curl http://${HOST}:${PORT}/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "prompt": "prompt", "n": 1, "size": "widthxheight" }' ``` Run GLM-Image with a separate SGLang server for its AR stage: ```bash theme={null} # Terminal 1 : launch server with AR model sglang serve --model-path /path/to/zai-org/GLM-Image/vision_language_encoder/ \ --tokenizer-path /path/to/zai-org/GLM-Image/processor/ --enable-multimodal --port ${AR_PORT} ``` ```bash theme={null} # Terminal 2 : launch server with Diffusion model sglang serve --model-path /path/to/zai-org/GLM-Image/ \ --srt-encoder-url "http://${HOST}:${AR_PORT}" \ --port ${PORT} ``` ```bash theme={null} # Terminal 3 : launch client curl http://${HOST}:${PORT}/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "prompt": "prompt", "n": 1, "size": "widthxheight" }' ``` ## Support matrix
Model Transformers backend SGLang backend
GLM-Image T2I, I2I, V2I T2I
Qwen Image Layered Not used I2I, in-process native Qwen2.5-VL
LongCat-Image Not used T2I, in-process native Qwen2.5-VL
## Deployment Assumptions & Limitations :::warning **Network Latency & Timeouts:** In SGLang backend mode, the Diffusion server sends an HTTP request to `--srt-encoder-url` for **every auto-regressive (AR) step**. * To prevent requests from breaking during long model generations, increase `--srt-encoder-timeout` (e.g., set to 100 seconds). * To protect the system against temporary network delays or brief drops in connection, use `--srt-encoder-connection-timeout`. ::: * **Recommended Setup:** Run both servers on the same machine or inside the same fast local network. * **Cross-Region Warning:** Running the Diffusion server and the AR server in different geographic regions will slow down token generation and heavily reduce performance. * **Startup Connection Check:** SGLang automatically checks the connection to `--srt-encoder-url` when starting up. The server will stop immediately if the remote AR host is offline. ## Ascend NPU ENV To run 2 servers on same group of NPU you need to specify env variables [https://www.hiascend.com/document/detail/zh/canncommercial/850/maintenref/envvar/envref\_07\_0144.html](https://www.hiascend.com/document/detail/zh/canncommercial/850/maintenref/envvar/envref_07_0144.html) Example: ```bash theme={null} # Terminal 1 : server with AR model export HCCL_IF_BASE_PORT=23000 export HCCL_HOST_SOCKET_PORT_RANGE="23000-23199" export HCCL_NPU_SOCKET_PORT_RANGE="23200-23399" ``` ```bash theme={null} # Terminal 2 : server with diffusion model export HCCL_IF_BASE_PORT=24000 export HCCL_HOST_SOCKET_PORT_RANGE="24000-24199" export HCCL_NPU_SOCKET_PORT_RANGE="24200-24399" ``` ## Best practices GLM-Image example for Ascend A3 Series, 2 cards (4 devices) ```bash theme={null} # Terminal 1 : server with AR model export HCCL_IF_BASE_PORT=23000 export HCCL_HOST_SOCKET_PORT_RANGE="23000-23199" export HCCL_NPU_SOCKET_PORT_RANGE="23200-23399" sglang serve --model-path /path/to/zai-org/GLM-Image/vision_language_encoder/ \ --tokenizer-path /path/to/zai-org/GLM-Image/processor/ --enable-multimodal \ --cuda-graph-bs-decode 1 --device npu --attention-backend ascend --image-processor-backend pil \ --tp-size 4 --port ${PORT} --mem-fraction-static 0.4 ``` Second terminal with diffusion server: ```bash theme={null} # Terminal 2 : run SGL-Diffusion generate command export HCCL_IF_BASE_PORT=24000 export HCCL_HOST_SOCKET_PORT_RANGE="24000-24199" export HCCL_NPU_SOCKET_PORT_RANGE="24200-24399" SGLANG_CACHE_DIT_FN=2 SGLANG_CACHE_DIT_BN=1 SGLANG_CACHE_DIT_WARMUP=4 SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 SGLANG_CACHE_DIT_TAYLORSEER=true SGLANG_CACHE_DIT_TS_ORDER=2 \ SGLANG_CACHE_DIT_ENABLED=true sglang generate --model-path /path/to/zai-org/GLM-Image/ \ --prompt "A curious raccoon" --height 1920 --width 1088 --num-inference-steps 50 --num-gpus 4 \ --sp-degree 4 --srt-encoder-url "http://${HOST}:${PORT}" --warmup-mode request ``` Result: ```bash theme={null} Warmed-up request processed in 33.82 seconds (with warmup excluded) ``` # Prompt enhancement Source: https://docs.sglang.io/docs/sglang-diffusion/models_with_pe Connect an independently deployed SGLang LLM or VLM to diffusion image and video APIs. Run an SRT server for prompt expansion and a diffusion server for generation. The diffusion HTTP frontend calls SRT's `/v1/chat/completions` once before constructing the generation request. The enhancer does not replace the diffusion model's text encoder or run inside its GPU workers. ```text theme={null} Application -> diffusion HTTP server -> SRT enhancer server <- rewritten prompt <- image/video <- diffusion pipeline ``` Prompt enhancement is opt-in per request. It changes conditioning and can change intent or quality; it is not a lossless acceleration feature. Evaluate the chosen enhancer and template on your workload before enabling it. ## Deploy two services This example uses Linux with NVIDIA CUDA and two GPUs, one for each server. Install [SGLang](/docs/get-started/install) for SRT and [SGLang Diffusion](/docs/sglang-diffusion/installation) for diffusion. Separate environments and hosts are supported; the diffusion host needs HTTP access to SRT. Start a text-only enhancer: ```bash theme={null} CUDA_VISIBLE_DEVICES=0 sglang serve \ --model-path Qwen/Qwen3-4B-Instruct-2507 \ --port 31000 ``` Create `enhancer.json` on the diffusion host: ```json theme={null} { "base_url": "http://127.0.0.1:31000/v1", "model": "Qwen/Qwen3-4B-Instruct-2507" } ``` Start diffusion in another terminal: ```bash theme={null} CUDA_VISIBLE_DEVICES=1 sglang serve \ --model-path Tongyi-MAI/Z-Image-Turbo \ --prompt-enhancer-config enhancer.json \ --port 30000 ``` Keep the SRT port separate from diffusion's HTTP and internal ports; diffusion reserves `--port + 1` for its broker. For separate hosts, bind SRT to its service interface with `--host` and update `base_url`. Restrict network access and configure SRT authentication. Do not expose an unauthenticated enhancer publicly. A shared enhancer can serve multiple diffusion servers; size its concurrency and memory independently. Send a generation request: ```bash theme={null} curl --fail-with-body http://127.0.0.1:30000/v1/images/generations \ -H 'Content-Type: application/json' \ -d '{ "prompt": "A red teapot on a windowsill, morning light", "enhance_prompt": true, "size": "512x512", "seed": 42, "response_format": "b64_json" }' ``` Omit `enhance_prompt` or set it to `false` to use the original prompt without calling SRT. With the OpenAI Python SDK, pass `extra_body={"enhance_prompt": True}` to `images.generate()`. The same switch works with `/v1/images/edits` and `/v1/videos`, including multipart requests (`-F 'enhance_prompt=true'`). Image responses expose the rewritten text as `data[].revised_prompt`; video responses use `revised_prompt`. Video creation waits for enhancement before returning the queued job. All `n` outputs share one rewritten prompt. Include enhancement in client-observed end-to-end latency. The response's `inference_time_s` measures diffusion execution, not the preceding SRT call. ## Configure the enhancer The configuration file is read at server startup. Its supported fields are: | Field | Meaning | | - | - | | `base_url` | SRT OpenAI API base URL, including `/v1` | | `model` | SRT model ID or its configured served model name | | `system_prompt` | Optional replacement for the built-in, task-aware rewriting instruction | | `include_images` | Forward reference images to a VLM; defaults to `false` | | `timeout` | HTTP operation timeout in seconds; defaults to `60` | | `api_key_env` | Name of an environment variable on the diffusion server containing the SRT API key | | `generation_kwargs` | SRT chat-completion options, such as `max_tokens`, `temperature`, `chat_template_kwargs`, or `response_format` | Default generation options are `max_tokens=512` and `temperature=0`. `generation_kwargs` cannot replace the model, messages, single-output or non-streaming settings, or enable tools. Choose a non-thinking model or configure its supported chat template to suppress reasoning; only message `content` is used as the diffusion prompt. The enhancer receives a JSON text message containing `task` (`image`, `image_edit`, or `video`) and `prompt`. It returns the replacement prompt as text. Seed, negative prompt, references and diffusion sampling options are not rewritten. The diffusion frontend reuses HTTP connections and makes no enhancer calls during synthetic warmup. Timeouts return HTTP 504. Upstream failures, empty responses and truncated completions return HTTP 502 before diffusion is queued. There is no automatic retry or fallback to the original prompt. An unconfigured server rejects `enhance_prompt=true` with HTTP 400. ## Choose a model combination There is no enhancer or diffusion model allowlist in this integration. Protocol compatibility does not establish quality: use a template appropriate for the target checkpoint and verify prompt adherence. | Diffusion input | Enhancer requirements | | - | - | | Natural-language image/video prompts, such as Z-Image, FLUX, Qwen-Image, Wan, LTX or MiniMax-H3 | An SRT-served instruction model with chat completions; the default template is a starting point | | Image editing or image-to-video | Text-only instruction rewriting, or an SRT VLM with `include_images=true` for image-aware rewriting | | Structured captions, such as Ideogram 4 or LingBot-Video-MoE | A model-specific `system_prompt` and preferably schema-constrained `response_format`; the default prose template is unsuitable | | ERNIE-Image or LongCat with built-in rewriting | Disable the built-in rewriter for the request when using this generic enhancer, to avoid rewriting twice | For a VLM, deploy an image-capable SRT model and set `include_images=true` in its configuration. Uploaded images are sent as data URLs, so the SRT server does not need access to the diffusion host's filesystem. HTTP image URLs must be reachable from SRT. Text-only models should leave this option disabled. Audio and video references are not forwarded to the enhancer; it cannot inspect their content. Image edits forward the supplied images; video creation forwards the primary image reference, not model-specific reference bundles. For [Ideogram 4](/cookbook/diffusion/Ideogram/Ideogram4), instruct the enhancer to produce the checkpoint's JSON caption schema, including `high_level_description` and `compositional_deconstruction`. Set the matching JSON schema through `generation_kwargs.response_format`. JSON text is passed through unchanged, not converted to prose. Valid JSON alone does not guarantee the correct caption schema or equivalent output to a hosted service's expander. This integration applies to the three HTTP image/video endpoints above, including their normal parallelism, caching and disaggregated diffusion paths. It does not add enhancement to offline `sglang generate`, realtime streaming, mesh or action endpoints. For those workflows, call SRT in your application first and pass its returned text as the diffusion prompt. For reproducibility, store the rewritten prompt and replay it with `enhance_prompt=false`; a fixed diffusion seed does not fix the enhancer's output. ## ERNIE-Image's checkpoint-provided enhancer ERNIE-Image also has a separate, existing PE integration. By default, diffusion loads its native Ministral3 enhancer in-process: ```bash theme={null} sglang serve --model-path baidu/ERNIE-Image --port 30000 ``` Use `use_pe=false` to skip it for a request. For a memory-constrained deployment, `--layerwise-offload-components pe` streams the native PE decoder from CPU. To serve the checkpoint-provided PE through SRT, use a local checkpoint's PE directory with its tokenizer and retain the ERNIE-specific protocol: ```bash theme={null} CUDA_VISIBLE_DEVICES=0 sglang serve \ --model-path /models/ERNIE-Image/pe --port 31000 CUDA_VISIBLE_DEVICES=1 sglang serve \ --model-path /models/ERNIE-Image \ --pe-server-url http://127.0.0.1:31000 --port 30000 ``` `--pe-server-url` uses the checkpoint's PE tokenizer/template and SRT's `/generate`; it is not the generic chat-completion integration. Keep using it when you need ERNIE's trained PE behavior. If you instead use `--prompt-enhancer-config`, send both `enhance_prompt=true` and `use_pe=false`. For LongCat, use `enable_prompt_rewrite=false` with the generic enhancer. These request flags skip native rewriting, not loading its model components. For Ascend setup, see [models with AR stages](/docs/sglang-diffusion/models_with_ar#ascend-npu-env). # Parallelism Overview Source: https://docs.sglang.io/docs/sglang-diffusion/parallelism SGLang Diffusion ships several parallelism strategies. Each one splits a different dimension of the DiT forward pass, which is exactly why they can be combined: the total GPU count is the product of the degrees, ```text theme={null} num_gpus = cfg_parallel_degree × tp_size × sp_degree sp_degree = ulysses_degree × ring_degree ``` This page is the map — what each axis does, which combinations are legal, and how to pick one. Per-axis depth lives in [Sequence Parallelism](./ring_sp_performance), [Encoder Parallelism](./encoder_parallel) (text/image encoders are a separate axis with their own knob), and the [CLI reference](./api/cli). ## The axes | Strategy | Splits | Communication | Flag | | - | - | - | - | | CFG parallel | guidance branches | one combine per denoise step | `--cfg-parallel-size` | | Tensor parallel (TP) | weights and attention heads | all-reduce per transformer block | `--tp-size` | | Ulysses SP | sequence outside attention ↔ heads inside it | two all-to-alls per attention | `--ulysses-degree` | | Ring SP | sequence rows inside attention | neighbor-only K/V rotation, overlapped with compute | `--ring-degree` | | K/V-gather CP | sequence rows inside attention | one K/V all-gather per attention | `--kv-gather-degree` | | Data parallel | requests | none between replicas | `--dp-size` | Two strategies compose when they split different dimensions. TP and Ulysses both touch heads but compose serially — TP splits the projection weights, then Ulysses splits the activations of the TP-local heads. Ring and Ulysses compose because ring splits rows while Ulysses splits heads. Ring has no composition with a K/V-all-gather style of attention parallelism: both answer the same question (how a rank's query rows see remote K/V), so they are alternatives for one slot, not complements. ## What happens to the shapes For `tp_size = T`, `ulysses_degree = U`, `ring_degree = R`, one attention runs: ```text theme={null} [B, S/(U·R), H/T, D] sequence-sharded activations │ Ulysses input all-to-all (inside each Ulysses group) ▼ [B, S/R, H/(T·U), D] full sequence of this ring block, few heads │ ring attention (R−1 neighbor hops; Q never moves, K/V rotate) ▼ [B, S/R, H/(T·U), D] │ Ulysses output all-to-all (inverse) ▼ [B, S/(U·R), H/T, D] ``` The ring merge (online softmax) requires every rank in a ring group to hold the *same heads* over *different rows*; the group construction guarantees this. A K/V-gather (CP-style) variant fills the same slot differently: instead of R−1 overlapped hops it all-gathers K/V once and computes the local Q rows against the full sequence in one shot — fewer, larger transfers, paid for by holding the whole K/V per rank. Like ring it splits rows, so it adds no head constraint. When no SP degree is set explicitly, `sp_degree=2` defaults to `kv_gather_degree=2` — its measured-win zone — and higher degrees default to Ulysses. Ulysses groups are laid out on contiguous ranks and ring groups on strided ranks, so with a node-major rank mapping, Ulysses traffic stays on intra-node NVLink (all-to-all needs full-bisection bandwidth) while ring hops cross the slower interconnect where neighbor-only transfers overlap with compute. A mis-mapped layout stays numerically correct and silently loses the performance — worth checking when a sharded run is unexpectedly slow. ## Constraints * `num_attention_heads % tp_size == 0` — TP splits heads at the projections. * `(num_attention_heads / tp_size) % ulysses_degree == 0` — Ulysses splits the **TP-local** heads. `H % U == 0` alone is not sufficient: 56 heads pass with `tp=2, ulysses=4` (28 % 4) and fail with `tp=4, ulysses=4` (14 % 4). * Ring adds no head constraint (it splits rows), but the sequence — including any model-specific packing alignment — must divide by `ulysses × ring`, since ring adds an outer row split on top of Ulysses's inner one. * Ring requires an attention backend that declares `supports_ring_rotation()` — the per-hop merge needs the kernel's softmax LSE. `fa` and `sage_attn` declare it; the launcher auto-selects `fa` when unset. * `USPAttention`'s masked/tail-padded text path and its replicated-prefix, -suffix, and -kv-prefix paths all support ring: the sharded K/V rotates through the ring while the tail-pad or replicated portion is attended locally once and combined into the ring result with the same online-softmax merge. This covers the joint text+image attention most models use (flux, flux\_2, qwen\_image, zimage, glm\_image, ernie\_image, and others). * What still raises `NotImplementedError` under ring: `USPAttention`'s generic varlen path (multiple packed segments per row, as used by HunyuanVideo — no ring-aware rotation for it yet), and the legacy stacked-QKV `UlyssesAttention` layer, now down to one user (Wan's VSA sparse-attention variant) that has no softmax LSE to merge and can't gain ring support without a different kernel. * The launcher validates `num_gpus` against the product of the degrees and fails fast on any mismatch. ## The Ulysses transport The all-to-alls normally run over NCCL. On exactly 2 GPUs with peer-to-peer access, a CUDA-IPC transport replaces them by default: each rank writes its half directly into the peer's mapped staging buffer, with GPU-side sequence counters instead of a NCCL rendezvous. An all-to-all is a permutation, never a reduction, so the transport cannot change results — outputs are bitwise identical to the NCCL path. | Environment variable | Default | Meaning | | - | - | - | | `SGLANG_DIFFUSION_IPC_A2A` | `1` | set `0` to force NCCL | | `SGLANG_DIFFUSION_IPC_A2A_TIMEOUT_MS` | `10000` | deadlock backstop for the peer wait; on expiry the transport retires on every rank and the request fails rather than returning incomplete data | | `SGLANG_DIFFUSION_IPC_A2A_MAX_BUFFERS` | `16` | staging pairs kept alive; raise for many-resolution serving | | `SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS` | `-1` | head groups for the pipelined exchange below; `0` keeps the sequential exchange, `N >= 2` forces `N` groups | On a single host with peer-to-peer access, Ulysses attention also pipelines the exchange against attention, one head group at a time: while group g attends, group g + 1's q/k/v moves on the copy engine, and finished groups send their output back while later groups still attend. Every Ulysses path of `USPAttention` goes through it under FlashAttention or cuDNN SDPA, at any batch size and with GQA: dense attention, key masks and tail padding, and text tokens replicated on every rank (a prefix, a suffix, or a K/V-only prefix), which stay out of the exchange. MiniMax-H3's attention uses the same transport. Ring attention, uneven sequence shards (`seq_lens`) and the K/V-gather mode keep the sequential exchange. The automatic count is the largest of 7, 5, 4, 3 and 2 that divides the heads per rank while each group's call still fills four waves of 128-row tiles, so short sequences such as a 1024² image keep the sequential exchange. Its buffers are one pool per dtype that every shape shares, grown to the largest (about 4 × rows per rank × heads × head\_dim); a shape that would not fit on every rank keeps the sequential exchange. The first call of every shape runs both exchanges and keeps the pipeline only if they give the same bytes on every rank, so results never change; that check runs the sequential exchange on per-call buffers, so no staging for it stays allocated. A custom attention module reuses it through `ulysses_pipelined_attention(q, k, v, attend, groups, replicated=..., sequential=...)`. Independently of the transport, the default path already packs the three Q/K/V input exchanges into one destination-major collective. A handful of models (e.g. LTX-2) instead opt into `enable_packed_qkv_input_a2a`, which pipelines three separate exchanges over a dedicated stream rather than merging them into one payload — a different trade-off, not a strict upgrade over the default. ## Which axes tolerate crossing nodes Each axis has a fixed communication pattern, and the pattern — volume per step, how often it fires, and whether it can hide behind compute — decides whether the axis survives the drop from NVLink to the inter-node fabric. Ordered from most to least cross-node friendly: | Axis | Pattern | Traffic per denoise step | Cross-node verdict | | - | - | - | - | | Data parallel | none between replicas | zero on the request path | **Best.** Replicas only share startup init and control-op fan-out; no fast interconnect needed at all. | | CFG parallel | one branch-combine | one latent-sized exchange, once | **Good candidate.** Once per step, small payload, naturally deadline-tolerant. Unmeasured across nodes so far. | | K/V-gather CP | one K/V all-gather per attention | (R−1)/R of K/V, one unoverlapped burst | **Between Ulysses and ring:** only K/V moves (queries never do), but the burst cannot hide behind compute — prefer ring across nodes, gather at small degrees within one. | | Ring SP | neighbor-only K/V rotation | K/V ÷ ring\_degree, (R−1) hops, overlapped with attention tiles | **Designed for it.** The mechanism now covers most models' joint attention (see Constraints). Actually crossing nodes end-to-end is validated for MiniMax H3 (Ulysses intra-node × ring across, net positive and growing with sequence length); other models' ring support is same-node-validated so far. | | Ulysses SP | all-to-all | full q/k/v activations, twice per attention, every layer | **Keep intra-node.** All-to-all needs full-bisection bandwidth; across nodes it becomes R² flows with receiver incast. | | Tensor parallel | all-reduce | hidden-sized reduction per transformer block (×60 blocks for qwen-class DiTs) | **Worst.** Highest frequency, no overlap, already \~70% of sharded kernel time on NVLink. | Two caveats keep this a map rather than a promise. Cross-node launch (`--nnodes`/`--node-rank`/`--dist-init-addr`) is merged, and per-model ring support is broad now (see the constraints above) — but those two facts together still don't add up to "any model, any node count." The only configuration actually run end-to-end across nodes is the H3 recipe; other models' ring support is same-node-validated so far, which means crossing nodes with them is untested, not disallowed. And the data-parallel row describes the design: the current implementation binds each replica's ingress on the local host, so replicas spanning hosts additionally need per-replica host addressing before `--dp-size` can place one replica per node. ## Choosing a configuration Measured guidance rather than rules — the right combination depends on the model's communication profile and the hardware topology, and legal does not mean profitable: * **Multi-branch (true-CFG) models**: CFG parallelism first. Branches run the whole DiT independently and combine once per step, avoiding per-layer communication entirely. * **Single-branch image models on 2 GPUs**: Ulysses and TP trade places by model. Communication-heavy DiTs measured faster with Ulysses; smaller DiTs with TP. Measure both; do not copy a winner across models. * **Long video / packed sequences**: Ulysses up to the head-divisibility limit, then ring for the remaining factor — sequence length scales past the head count where Ulysses alone cannot. * **When Ulysses's head divisibility blocks the degree you need** (`H/T` not divisible by the target `U`): the row-splitting slot sidesteps it — ring today, or `--kv-gather-degree` (it splits rows, so it adds no head constraint either). * **TP beyond 2 ranks** rarely improves image-DiT latency: the per-block all-reduce grows with rank count faster than the GEMM savings. ## Data parallelism `--dp-size N` runs N full engine replicas on `num_gpus / N` GPUs each, every replica with its own ingress. Generation requests round-robin across replicas, realtime sessions stick to the replica holding their state, and control operations (weights, LoRA, memory occupation, shutdown) apply to every replica; replicas exchange nothing on the request path. Monolithic serving only, and the ingress currently binds on the local host — one replica per node needs per-replica host addressing first. # Performance Optimization Source: https://docs.sglang.io/docs/sglang-diffusion/performance-optimization Choose performance levers for SGLang Diffusion by latency, throughput, memory, and quality tradeoffs. Use this page as the starting point for SGLang Diffusion performance work. It separates performance levers into two decision classes: * **Output-preserving / lossless-style:** system settings that should preserve model behavior while changing residency, parallelism, kernels, or scheduling. * **Quality-tradeoff / lossy or approximate:** techniques that can change the denoising path, numerical representation, or generated output. The docs use "output-preserving" instead of promising bit-exact "lossless" because different kernels, GPU types, or precision paths can still introduce small numerical differences. The decision boundary is whether the optimization intentionally trades quality or output equivalence for speed. ## Start Here 1. Pick a serving or generation mode from [Deployment and Performance Modes](/docs/sglang-diffusion/deployment_cookbook). `--performance-mode auto` is the default; use `speed` when the model fits in GPU memory and latency matters most, `memory` when GPU memory is the bottleneck, and `manual` when every performance flag should be explicit. 2. Choose the right attention backend from [Attention Backends](/docs/sglang-diffusion/attention_backends). 3. Use [Sequence Parallelism](/docs/sglang-diffusion/ring_sp_performance) only when the model and video shape benefit from sequence splitting. 4. Use [Inference Batching](/docs/sglang-diffusion/dynamic_batching) for concurrent compatible requests during serving. 5. Use [Profiling](/docs/sglang-diffusion/profiling) before changing several levers at once. ## Choose a request quality tier `--quality` is cumulative: a broader tier never drops an optimization from a stricter tier. | Tier | Optimization boundary | | - | - | | `exact` | The selected deployment's reference execution plus all unconditional bit-exact replacements. Output is bit-identical to the reference path in the same environment | | `lossless` (default) | Everything in `exact`, plus request-gated DiT/VAE fusions that keep the reference math and the precision of every operand and accumulator, changing only where or in which order the rounding happens. Output matches the reference to rounding error, not bit for bit | | `high` | Everything in `lossless`, plus model-owned paths that change what is computed -- lower precision, sparsity, or reuse across steps, such as an audited Cache-DiT policy or lower-precision VAE decode | Bit-identical describes reproducibility, not quality: a `lossless` fast path is the same mathematical function, and its rounding is no further from the exact value than the reference's own. On FLUX.1-dev the two tiers differ by 34.5 dB PSNR, while the same tier differs by 32.5 dB between one GPU and two -- the tier moves the image less than the parallel layout does, which is why it is the default. Ask for `exact` when a run has to be byte-for-byte reproducible, or when bisecting a numerical bug. `extra-high` is accepted as the former name of `lossless`. Note that `lossless` itself was the former name of `exact`, and that it is now the default: a request that passed `quality="lossless"` for bit-identical output has to pass `quality="exact"`, and a deployment that never set `quality` moves from the bit-exact path to this one. A tier may be a no-op when the active model has no eligible path. Separately configured quantization, attention, or caching options still apply. See [Fused Kernels](/docs/sglang-diffusion/fused_kernels) for the current request-gated families and their numerical contracts. ### Library numerics below the tiers PyTorch enables two approximations by default, underneath the whole tier system: `torch.backends.cudnn.allow_tf32` truncates fp32 convolution inputs to 10 mantissa bits, and `torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction` lets a bf16 GEMM accumulate split-K partials below fp32. Both change what is computed, so by the definitions above they belong to `high`. They are process-global and a server mixes tiers across concurrent batches, so they cannot follow a request. The runtime sets both explicitly per worker and logs the result, instead of inheriting whatever the installed PyTorch defaults to: | Flag | Default | Effect of turning it off | | - | - | - | | `--allow-cudnn-tf32` | `true` (PyTorch's own default) | fp32 convolutions run on fp32 CUDA cores instead of TF32 tensor cores. Affects the VAE decoders that default to `vae_precision="fp32"` | | `--allow-bf16-reduced-precision-reduction` | `true` (PyTorch's own default) | bf16 GEMMs accumulate split-K partials at fp32 | The defaults keep PyTorch's values so that `exact` is no slower than it was before these flags existed. A deployment that needs the strict contract — fp32 and bf16 math at reference precision throughout — passes both as `false` and accepts the slower convolutions. ## Output-Preserving / Lossless-Style Levers These settings should preserve model behavior while changing residency, parallelism, kernels, or scheduling. They are the first choices for production tuning.
Lever Use when Docs
--performance-mode You want a safe preset for speed or memory without overriding explicit flags. Deployment and Performance Modes
Breakable CUDA graph A supported pipeline serves a fixed set of shapes and eager execution is launch-bound. CLI reference
Offload, FSDP, CFG parallelism GPU memory, multi-GPU residency, or CFG branch splitting is the main bottleneck. Deployment and Performance Modes
Sequence parallelism Long image/video sequences need sequence-level parallelism. Sequence Parallelism
--encoder-parallel Text/image encoding is a visible share of the request and the DiT replica sits idle during it. Encoder Parallelism
Attention backend Kernel choice dominates DiT latency or memory. Attention Backends
Fused kernels You want to know which elementwise chains are already fused, or to opt into the request-gated set. Fused Kernels
Dynamic batching Serving many compatible requests concurrently. Inference Batching
--async-output-save Image/video encoding and disk writes leave the GPU idle between queued requests. Asynchronous output saving
## Asynchronous output saving Enable `--async-output-save` to overlap output saving with the next request's GPU work during serving. Frame materialization, image/video encoding, and disk writes run on a background thread. The flag is disabled by default and does not change the model's generation settings or the requested output format. Use it when you have concurrent requests and saving takes a noticeable share of each request, such as PNG images or large videos from a fast model. A client that waits for each response before sending the next request has no work to overlap. ### Enable The feature requires NVIDIA CUDA and the native SGLang backend in monolithic serving mode. After following [Installation](/docs/sglang-diffusion/installation), you can start this example on one GPU with the public Hugging Face model `Tongyi-MAI/Z-Image-Turbo`: ```bash Command theme={null} sglang serve \ --model-path Tongyi-MAI/Z-Image-Turbo \ --backend sglang \ --num-gpus 1 \ --batching-max-size 1 \ --async-output-save ``` Send concurrent requests using the [OpenAI-compatible API](/docs/sglang-diffusion/api/openai_api). `--batching-max-size 1` keeps each forward separate while allowing requests to queue. Grouped and dynamically batched forwards currently keep synchronous output saving. ### Scope and tradeoffs * Only the output rank performs asynchronous saving, and only for requests that save files and return their paths (`save_output=true` and `return_file_paths_only=true` in the sampling parameters). * Warmup requests, raw-frame responses, requests with `save_output=false`, and requests with `perf_dump_path` set or `SGLANG_DIFFUSION_STAGE_LOGGING=1` keep the synchronous path. This includes the frame-return path used by `/rollout/generate`. * Each reply is sent from the saving thread as soon as its output is saved, even while the next request's forward is running. Measure both throughput and response latency against the flag disabled. * Up to two pending finalizations retain request outputs in GPU memory. The scheduler waits for capacity before starting another forward; leave memory headroom for the pending outputs as well as inference. * Per-request peak memory metrics can include allocations from the next request. Asynchronous finalization skips the usual per-request allocator cache release. Compare the flag on and off with the same model, output shape, format, and client concurrency. Keep it disabled when the reply latency or extra memory outweighs the throughput gain. ## Encoding during VAE decode Native Wan, H3, and LTX video VAEs can hand completed frames to the shared MP4 encoder within the current request. Unlike `--async-output-save`, this does not require a second request to obtain a benefit. Eligible single-video, CUDA, file-path-only requests use this path automatically; raw frames, compilation, interpolation, upscaling, and custom post-decoding hooks keep the normal path. LTX's diffusion decoder is not included. Ordinary VAEs without frame callbacks keep the existing GPU-direct saving path. Handing over a complete clip cannot overlap encoding with decode, and is not automatically enabled: it can add overhead to short requests. The shared `DecodingStage.decode(..., on_frames=...)` supports an explicit complete-clip hand-off for custom integrations. Custom decoding stages retain their original path unless they explicitly integrate the encoder, preserving output transforms and audio muxing. Image, audio, and mesh outputs are not MP4 inputs. LTX without temporal tiling hands over the complete clip after decode. This reduces CPU float-frame materialization but does not overlap VAE decoding with encoding. Existing temporal tiling can emit finalized, blended prefixes while later tiles decode. This optimization does **not** enable temporal tiling: changing the tile schedule can change the video. Keep the same decode, output compression, and x264 preset when measuring. Include time until the MP4 is fully written, not just the decoding-stage timer. The encoder bounds queued frames and frame-conversion intermediates, and a failed encode falls back to normal saving. Model authors can follow the [frame hand-off contract](/docs/sglang-diffusion/support_new_models#video-encode-hand-off). ## Quality-Tradeoff / Lossy Or Approximate Levers These techniques can change the denoising path, numerical representation, or generated output. They are useful after you have a baseline and an acceptance criterion for quality.
Lever Tradeoff Docs
Cache-DiT Skips selected DiT block or step computation based on cache decisions. Cache-DiT
TeaCache Reuses residuals when consecutive denoising steps are similar enough. TeaCache
Progressive resolution Runs early denoising at lower latent resolution for supported pipelines. Progressive Resolution Generation
Quantization Uses lower-precision transformer weights or activations. Quantization
## Practical Order 1. Establish a baseline with the target model, resolution, frame count, step count, and GPU type. 2. Select `--performance-mode` and explicit residency or parallelism flags. 3. Compare `quality=lossless` with `quality=lossless` to isolate the request-gated fusion set. 4. Compare breakable CUDA graph against eager execution for supported fixed-shape pipelines. Pass every served resolution to `--warmup-resolutions` and confirm capture in the server log. Models with request-gated DiT fusions cannot combine those fusions with a graph captured from the lossless branches. 5. Tune attention backend and batching for the deployment pattern. 6. Profile if the bottleneck is unclear. 7. Add `quality=high`, caching, progressive resolution, or quantization only after comparing output quality against your acceptance target. ## Per-model tuning starting points Warmup and breakable CUDA graph (BCG) solve different problems. `--warmup-mode request` runs a warmup copy derived from the first request to prime one-time compilation and caches; it does not remove recurring Python launches from each denoising step. BCG captures supported DiT segments and can reduce that recurring launch overhead for captured shapes. Use the table below as a first experiment, then keep a lever only when profiling confirms that it addresses the active bottleneck. | Observed bottleneck or constraint | Example models | First experiment | What to verify | | - | - | - | - | | The first execution of a shape pays substantial compilation or cache setup | ERNIE-Image-Turbo, GLM-Image, FastWan / TurboWan family, FLUX.2-klein-4B, LingBot-World, LongLive, Cosmos3, SANA, LongCat-Image-Edit-Turbo | Compare `--warmup-mode request` with `off` | Separate warmup time, first-real-request latency, and warmed steady-state latency. Request-based warmup is primarily a benchmark aid and still consumes time before the first real request completes. | | A supported fixed-shape pipeline shows recurring host-launch gaps between GPU kernels | GLM-Image, Z-Image-Turbo, Qwen-Image-2512, Ideogram-4, LongCat-Image, LTX-2 / LTX-2.3, SANA-1.5 1.6B, JoyEcho | Compare eager execution with `--enable-breakable-cuda-graph` | Confirm graph capture and replay in the server log, then compare steady-state latency and GPU memory. Pass additional production shapes through `--warmup-resolutions`; without it, BCG captures only the model's default warmup resolution. | | The model does not fit while both DiT stages are resident | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, LingBot-Video-MoE | Start with `--dit-layerwise-offload` | Verify peak GPU memory first, then measure the transfer overhead. Add warmup separately if first-run compilation is also material. | | Dynamic masks, metadata, or image conditions make graph replay ineffective | Qwen-Image-Edit family, FireRed-Image-Edit | Keep eager execution as the baseline | Profile before attempting BCG. Dynamic per-step host work can outweigh graph replay savings, and unsupported pipelines fall back to eager execution. | | Kernels or collectives already dominate a many-step workload | Qwen-Image, FLUX.1-dev, FLUX.2-dev, LTX-2, Wan2.1 14B, HunyuanVideo, MOVA | Warm the baseline, then profile before enabling another lever | A small host gap limits the benefit available from BCG. Prioritize kernels, attention, parallelism, or residency according to the trace. | | A one-shot or very few-step workload cannot amortize warmup | FastVideo-FastH3 (8-step) | Start with `--warmup-mode off` | Compare end-to-end latency including warmup. Use request warmup only when intentionally separating cold-start setup from the measured request. | The example assignments come from single-GPU NVIDIA B300/GB300 profiles and are directional rather than an exhaustive compatibility list. A model can match more than one row as resolution, frame count, step count, parallelism, or GPU type changes; the measured bottleneck takes precedence over the model name. BCG is enabled only for model and pipeline configurations accepted by the runtime support check. It captures the default warmup shape automatically. Use `--warmup-resolutions` for additional served resolutions; for video and variable prompt lengths, set `--warmup-num-frames` and `--bcg-text-buckets` to cover the intended workload. Requests that do not match a captured signature fall back to eager execution. Warmup and BCG do not intentionally trade output quality for speed, but that is not a guarantee of byte-identical output. Different execution paths can introduce numerical differences, and some pipelines customize the scheduler or schedule used by a synthetic warmup request. Before production rollout, compare output hashes when bitwise stability is required, otherwise run the project's quality acceptance check. Performance results are configuration-dependent. Record the exact checkpoint revision, GPU, precision, resolution, frame count, step count, parallelism, command line, and whether the measurement includes warmup. Re-profile with [Profiling](/docs/sglang-diffusion/profiling) on the target workload rather than transferring a percentage from another model or shape. ## Diagnostics [Profiling](/docs/sglang-diffusion/profiling) is not an optimization technique by itself. It belongs in the performance workflow because it tells you which stage, kernel, or denoising step is worth optimizing before you change multiple levers. ## References * [Deployment and Performance Modes](/docs/sglang-diffusion/deployment_cookbook) * [Attention Backends](/docs/sglang-diffusion/attention_backends) * [Fused Kernels](/docs/sglang-diffusion/fused_kernels) * [Sequence Parallelism](/docs/sglang-diffusion/ring_sp_performance) * [Caching Strategies](/docs/sglang-diffusion/caching-acceleration) * [Exact conditioning reuse across requests](/docs/sglang-diffusion/caching-acceleration#conditioning-cache) * [Profiling](/docs/sglang-diffusion/profiling) # Production Metrics Source: https://docs.sglang.io/docs/sglang-diffusion/production_metrics Monitor SGLang Diffusion requests, queues, stages and LoRA state with Prometheus. ## Enable metrics Enable metrics on an NVIDIA CUDA deployment, for example: ```bash theme={null} sglang serve --model-path black-forest-labs/FLUX.2-klein-4B \ --num-gpus 1 --enable-metrics --port 30000 curl http://localhost:30000/metrics ``` Metrics are opt-in. Disabled metrics do not scan queues or collect LoRA status. Enabled metrics add host-side bookkeeping, not GPU synchronization or collectives. ## Metric reference All diffusion metrics carry `role` and `replica` labels. `replica` is the scheduler endpoint; only each DP replica's leader publishes, so TP/SP ranks do not multiply request counts. The table lists additional labels. | Metric | Type | Labels | Description | | - | - | - | - | | `sglang:diffusion_num_queue_reqs` | Gauge | none | Original generation requests waiting for their first dispatch. | | `sglang:diffusion_num_running_reqs` | Gauge | none | Number of diffusion generation requests dispatched by the scheduler and not yet finished. | | `sglang:diffusion_requests_total` | Counter | `status`, `is_warmup` | Completed diffusion generation requests. Dynamic batches are counted per original scheduler request after the merged output is split. | | `sglang:diffusion_request_latency_seconds` | Histogram | `status`, `is_warmup` | Scheduler acceptance to completion, excluding HTTP preprocessing, media encoding and response delivery. | | `sglang:diffusion_queue_time_seconds` | Histogram | `is_warmup` | Time spent waiting in the diffusion scheduler queue. | | `sglang:diffusion_generation_batch_size` | Histogram | `stop_reason` | Generation batch size selected by the diffusion scheduler at dispatch time. | | `sglang:diffusion_stage_host_latency_seconds` | Histogram | `stage` | Host wall time around a stage, not GPU kernel execution time. Step labels are normalized to `DenoisingStep`. | | `sglang:diffusion_lora_loaded_adapters` | Gauge | none | Number of loaded diffusion LoRA adapters. | | `sglang:diffusion_lora_active_modules` | Gauge | none | Number of diffusion modules with active LoRA adapters. | | `sglang:diffusion_lora_active_adapters` | Gauge | none | Number of unique active diffusion LoRA adapters. | | `sglang:diffusion_lora_module_active` | Gauge | `module` | Whether a diffusion module currently has an active LoRA adapter. | Request counts refer to original scheduler requests, not generated images, denoising steps or distributed shards. `status` is `success` or `error`; `is_warmup` is `true` or `false`. Queue and running gauges include warmup. Stage observations include warmup and count stage invocations, not requests; asynchronous GPU work can complete in a later stage. For synchronized diagnostic timings, use `SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1` separately, accepting its synchronization overhead. LoRA gauges update at startup and after LoRA control operations. ## Disaggregated serving In disaggregated serving, the head (`role="server"`) records the original request lifecycle, including role handoffs and errors. Queue time ends at the first encoder dispatch; intermediate waits remain part of request latency. Workers report their own stage and LoRA metrics, not duplicate completed requests. The generation-batch histogram currently describes monolithic scheduling only. Single-host pool mode exposes all child metrics through the head's `/metrics`. A scrape aggregates only processes sharing that host's metrics directory. For standalone remote roles, pass `--enable-metrics` to each process and scrape its `--host`/`--port` as well as the head. Each role serves a metrics-only HTTP endpoint. Use a separate, empty `PROMETHEUS_MULTIPROC_DIR` per server launch if you set it yourself; otherwise SGLang creates and owns a temporary directory. Do not share this directory between independent servers or reuse stale files. ## Query throughput Successful, non-warmup request throughput: ```promql theme={null} sum(rate(sglang:diffusion_requests_total{status="success",is_warmup="false"}[5m])) ``` # Profiling Source: https://docs.sglang.io/docs/sglang-diffusion/profiling Profile SGLang diffusion workloads with PyTorch Profiler and Nsight Systems. This guide covers profiling techniques for multimodal generation pipelines in SGLang. ## PyTorch Profiler PyTorch Profiler provides detailed kernel execution time, call stack, and GPU utilization metrics. ### Denoising Stage Profiling Profile the denoising stage with sampled timesteps (default: 5 steps after 1 warmup step): ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --seed 0 \ --profile ``` **Parameters:** * `--profile`: Enable profiling for the denoising stage * `--num-profiled-timesteps N`: Number of timesteps to profile after warmup (default: 5) * Smaller values reduce trace file size * Example: `--num-profiled-timesteps 10` profiles 10 steps after 1 warmup step ### Full Pipeline Profiling Profile all pipeline stages (text encoding, denoising, VAE decoding, etc.): ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --seed 0 \ --profile \ --profile-all-stages ``` **Parameters:** * `--profile-all-stages`: Used with `--profile`, profile all pipeline stages instead of just denoising ### Output Location By default, trace files are saved in the ./logs/ directory. The exact output file path will be shown in the console output, for example: ```bash theme={null} [mm-dd hh:mm:ss] Saved profiler traces to: /sgl-workspace/sglang/logs/mocked_fake_id_for_offline_generate-5_steps-global-rank0.trace.json.gz ``` ### View Traces Load and visualize trace files at: * [https://ui.perfetto.dev/](https://ui.perfetto.dev/) (recommended) * chrome://tracing (Chrome only) For large trace files, reduce `--num-profiled-timesteps` or avoid using `--profile-all-stages`. ### `--perf-dump-path` (Stage/Step Timing Dump) Besides profiler traces, you can also dump a lightweight JSON report that contains: * stage-level timing breakdown for the full pipeline * step-level timing breakdown for the denoising stage (per diffusion step) This is useful to quickly identify which stage dominates end-to-end latency, and whether denoising steps have uniform runtimes (and if not, which step has an abnormal spike). The dumped JSON contains a `denoise_steps_ms` field formatted as an array of objects, each with a `step` key (the step index) and a `duration_ms` key. Example: ```bash theme={null} sglang generate \ --model-path \ --prompt "" \ --perf-dump-path perf.json ``` ## Nsight Systems Nsight Systems provides low-level CUDA profiling with kernel details, register usage, and memory access patterns. ### Installation See the [SGLang profiling guide](../developer_guide/benchmark_and_profiling#profile-with-nsight) for installation instructions. ### Basic Profiling Profile the entire pipeline execution: ```bash theme={null} nsys profile \ --trace-fork-before-exec=true \ --cuda-graph-trace=node \ --force-overwrite=true \ -o QwenImage \ sglang generate \ --model-path Qwen/Qwen-Image \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --seed 0 ``` ### Targeted Stage Profiling Use `--delay` and `--duration` to capture specific stages and reduce file size: ```bash theme={null} nsys profile \ --trace-fork-before-exec=true \ --cuda-graph-trace=node \ --force-overwrite=true \ --delay 10 \ --duration 30 \ -o QwenImage_denoising \ sglang generate \ --model-path Qwen/Qwen-Image \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --seed 0 ``` **Parameters:** * `--delay N`: Wait N seconds before starting capture (skip initialization overhead) * `--duration N`: Capture for N seconds (focus on specific stages) * `--force-overwrite`: Overwrite existing output files ## Notes * **Reduce trace size**: Use `--num-profiled-timesteps` with smaller values or `--delay`/`--duration` with Nsight Systems * **Stage-specific analysis**: Use `--profile` alone for denoising stage, add `--profile-all-stages` for full pipeline * **Multiple runs**: Profile with different prompts and resolutions to identify bottlenecks across workloads ## FAQ * If you are profiling `sglang generate` with Nsight Systems and find that the generated profiler file did not capture any CUDA kernels, you can resolve this issue by increasing the model's inference steps to extend the execution time. # Progressive Resolution Generation Source: https://docs.sglang.io/docs/sglang-diffusion/progressive_resolution Experimental spectral progressive resolution growing for selected SGLang Diffusion pipelines. Progressive resolution growing is an experimental feature for selected SGLang Diffusion pipelines. It runs early denoising steps at a coarser latent resolution and spectrally upsamples the latent before the full-resolution steps. On the benchmark setup below, this reduces the quadratic attention cost of the DiT transformer and yields up to **1.63× speedup on FLUX.1**, **1.93× speedup on FLUX.2**, **2.33× speedup on Z-Image**, **2.78× speedup on Wan 2.1 T2V**, **1.69× speedup on Qwen-Image**, and **1.56× speedup on Ideogram 4**. Based on [Spectral Progressive Diffusion (arXiv 2605.18736)](https://arxiv.org/abs/2605.18736). ## Overview DiT attention is O(n²) in sequence length. Running the first N denoising steps at half the spatial resolution cuts the attention cost to \~6% for those steps. The transition point — how many steps to run at each resolution — is computed from the **Bayes-optimal frequency-activation criterion**: frequencies that cannot be resolved at the coarse scale are not denoised there. The method is designed to preserve quality under this criterion, but generated outputs can still differ from the full-resolution baseline. | Model | Full-res tokens | Half-res tokens | Token-step ratio | | - | - | - | - | | FLUX.1 1024×1024 | 4,096 | 1,024 | 4.0× | | FLUX.2 1024×1024 | 4,096 | 1,024 | 4.0× | | Z-Image 1024×1024 | 4,096 | 1,024 | 4.0× | | Wan 2.1 T2V 480×832 (81 frames) | 6,240 | 1,560 | 4.0× | | Ideogram 4 1024×1024 | 4,096 | 1,024 | 4.0× | ## Parameters | Parameter | CLI flag | Default | Description | | - | - | - | - | | `progressive_mode` | `--progressive-mode` | `"fullres"` | `"fullres"` disables (identical to standard generation). `"dct_rewind"` enables spectral upsample with scheduler rewind (recommended). `"dct"` enables upsample without rewind. | | `progressive_levels` | `--progressive-levels` | `1` | Number of resolution halvings. `1` = one coarse stage (64×64 latent → 128×128). `2` = two coarse stages (32×32 → 64×64 → 128×128). | | `progressive_delta` | `--progressive-delta` | `0.01` | Noise-dominated tolerance δ. Controls how many steps run at coarse resolution. Higher δ = more coarse steps = more speedup. | > **Tip:** Add `--dit-cpu-offload false` to keep the transformer GPU-resident. With CPU offload each step pays a fixed PCIe transfer cost regardless of sequence length, which dilutes the speedup. *** ## FLUX.1 ### Usage ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.1-dev \ --prompt "A serene mountain lake at golden hour, photorealistic" \ --num-inference-steps 50 \ --dit-cpu-offload false \ --progressive-mode dct_rewind \ --progressive-levels 1 \ --progressive-delta 0.05 ``` ### Choosing delta | δ | Coarse steps (50 total) | Denoising speedup | | - | - | - | | `0.01` | 18 @ 64² + 32 @ 128² | **1.32×** | | `0.05` | 28 @ 64² + 22 @ 128² | **1.63×** | For most prompts `0.05` is recommended — it gives the largest speedup with no visible degradation. ### Benchmark Hardware: RTX A6000 48 GB, `--dit-cpu-offload false`. Timing = denoising loop only. | Config | Stage split | Denoise | Speedup | | - | - | - | - | | Fullres (baseline) | 50 @ 128² latent | 36.65 s | 1.00× | | dct\_rewind L1 δ=0.01 | 18\@64² + 32\@128² | 27.67 s | **1.32×** | | dct\_rewind L1 δ=0.05 | 28\@64² + 22\@128² | 22.58 s | **1.62×** | | dct\_rewind L2 δ=0.01 | 10\@32² + 8\@64² + 32\@128² | 26.48 s | **1.38×** | ### Python API ```python theme={null} from sglang.multimodal_gen import DiffGenerator gen = DiffGenerator.from_pretrained( model_path="black-forest-labs/FLUX.1-dev", dit_cpu_offload=False, ) result = gen.generate(sampling_params_kwargs={ "prompt": "A serene mountain lake at golden hour, photorealistic", "num_inference_steps": 50, "height": 1024, "width": 1024, "progressive_mode": "dct_rewind", "progressive_levels": 1, "progressive_delta": 0.05, }) ``` *** ## FLUX.2 Supports `FLUX.2-dev`, `FLUX.2-klein-4B`, and `FLUX.2-klein-9B`. ### Usage ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.2-klein-4B \ --prompt "A serene mountain lake at golden hour, photorealistic" \ --num-inference-steps 30 \ --dit-cpu-offload false \ --progressive-mode dct_rewind \ --progressive-levels 1 \ --progressive-delta 0.10 ``` ### Benchmark Hardware: RTX A6000 48 GB, `--dit-cpu-offload false`. Model: FLUX.2-klein-4B, 30 steps, 1024×1024. Timing = denoising loop only, averaged across 10 diverse prompts. | Config | Stage split | Denoise | Speedup | | - | - | - | - | | Fullres (baseline) | 30 @ 64² latent | 9.72 s | 1.00× | | dct\_rewind L1 δ=0.05 | 18\@32² + 12\@64² | 5.50 s | **1.77×** | | dct\_rewind L1 δ=0.10 | 20\@32² + 10\@64² | 5.03 s | **1.93×** | ### Python API ```python theme={null} from sglang.multimodal_gen import DiffGenerator gen = DiffGenerator.from_pretrained( model_path="black-forest-labs/FLUX.2-klein-4B", dit_cpu_offload=False, ) result = gen.generate(sampling_params_kwargs={ "prompt": "A serene mountain lake at golden hour, photorealistic", "num_inference_steps": 30, "progressive_mode": "dct_rewind", "progressive_levels": 1, "progressive_delta": 0.10, }) ``` *** ## Wan 2.1 T2V Supports `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` and `Wan-AI/Wan2.1-T2V-14B-Diffusers`. > **Note:** Progressive generation grows only the **spatial** H×W dimensions. The temporal dimension T (number of latent frames) is kept fixed across all stages. ### Usage ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --prompt "A cheetah sprinting across the Serengeti at sunset, slow motion, photorealistic" \ --num-inference-steps 50 \ --num-frames 81 \ --height 480 \ --width 832 \ --guidance-scale 5.0 \ --flow-shift 5.0 \ --dit-cpu-offload false \ --progressive-mode dct_rewind \ --progressive-levels 1 \ --progressive-delta 0.05 ``` ### Choosing delta | δ | Coarse steps (50 total) | Denoising speedup | | - | - | - | | `0.01` | 23 @ 30×52 + 27 @ 60×104 | **1.65×** | | `0.02` | 27 @ 30×52 + 23 @ 60×104 | **1.86×** | | `0.05` | 33 @ 30×52 + 17 @ 60×104 | **2.32×** | | `0.10` | 37 @ 30×52 + 13 @ 60×104 | **2.78×** | For most prompts `0.05` is recommended. `0.10` provides maximum speedup but should be validated on motion-heavy scenes. ### Python API ```python theme={null} from sglang.multimodal_gen import DiffGenerator gen = DiffGenerator.from_pretrained( model_path="Wan-AI/Wan2.1-T2V-1.3B-Diffusers", dit_cpu_offload=False, flow_shift=5.0, ) result = gen.generate(sampling_params_kwargs={ "prompt": "A cheetah sprinting across the Serengeti at sunset, slow motion, photorealistic", "num_inference_steps": 50, "num_frames": 81, "height": 480, "width": 832, "guidance_scale": 5.0, "progressive_mode": "dct_rewind", "progressive_levels": 1, "progressive_delta": 0.05, }) ``` *** ## Z-Image Supports `Tongyi-MAI/Z-Image`. Z-Image uses the same VAE as FLUX.1 (`FluxVAEConfig`), so the power-law spectrum constants are identical. The progressive stage handles Z-Image's 5-D latent format `[B, C, 1, H, W]` with squeeze/unsqueeze hooks and recomputes caption+image RoPE positional embeddings on each stage transition. > **Note:** Always specify `--height 1024 --width 1024` (or another resolution where H\_lat and W\_lat are both divisible by 2). Z-Image's default resolution (360×640) produces a 45×80 latent where H=45 is not divisible by the patch size. ### Usage ```bash theme={null} # Standard fullres — unchanged behavior sglang generate --model-path Tongyi-MAI/Z-Image \ --prompt "A serene mountain lake at golden hour, photorealistic" \ --height 1024 --width 1024 # Progressive dct_rewind L1 δ=0.10 → 2.33× denoising speedup sglang generate --model-path Tongyi-MAI/Z-Image \ --prompt "A serene mountain lake at golden hour, photorealistic" \ --height 1024 --width 1024 \ --num-inference-steps 50 \ --dit-cpu-offload false \ --progressive-mode dct_rewind \ --progressive-levels 1 \ --progressive-delta 0.10 ``` ### Choosing delta | δ | Coarse steps (50 total) | Denoising speedup | | - | - | - | | `0.01` | 26 @ 64² + 24 @ 128² | **1.53×** | | `0.05` | 35 @ 64² + 15 @ 128² | **2.03×** | | `0.10` | 42 @ 64² + 8 @ 128² | **2.33×** | Z-Image achieves higher progressive speedups than FLUX.1 at the same δ because it uses dual CFG (two forward passes per step), doubling the absolute attention savings at coarse resolution. `0.10` is the recommended tradeoff. ### Python API ```python theme={null} from sglang.multimodal_gen import DiffGenerator gen = DiffGenerator.from_pretrained( model_path="Tongyi-MAI/Z-Image", dit_cpu_offload=False, ) result = gen.generate(sampling_params_kwargs={ "prompt": "A serene mountain lake at golden hour, photorealistic", "num_inference_steps": 50, "height": 1024, "width": 1024, "progressive_mode": "dct_rewind", "progressive_levels": 1, "progressive_delta": 0.10, }) ``` *** ## Qwen-Image Qwen-Image uses the same 2×2 patchify convention as FLUX.1 (in\_channels=64, C=16), so the same progressive stage wires in with model-specific hooks for RoPE (`freqs_cis`) and spatial metadata (`img_shapes`). ```bash theme={null} # Standard fullres — unchanged behavior sglang generate --model-path Qwen/Qwen-Image \ --prompt "A serene mountain lake at golden hour" # Progressive dct_rewind L1 δ=0.20 → 1.69× denoising speedup sglang generate --model-path Qwen/Qwen-Image \ --prompt "A serene mountain lake at golden hour" \ --progressive-mode dct_rewind --progressive-levels 1 --progressive-delta 0.20 \ --num-inference-steps 30 --dit-cpu-offload false ``` Hardware: RTX A6000 48 GB, `--dit-cpu-offload false`. Timing = denoising loop only. | Config | Stage split | Denoise | Speedup | | - | - | - | - | | Fullres (baseline) | 30 @ 128² | 43.00 s | 1.00× | | dct\_rewind L1 δ=0.05 | 13\@64² + 17\@128² | 33.25 s | **1.29×** | | dct\_rewind L1 δ=0.10 | 16\@64² + 14\@128² | 33.86 s | **1.27×** | | dct\_rewind L1 δ=0.20 | 19\@64² + 11\@128² | 25.40 s | **1.69×** | ## Ideogram 4 Supports `ideogram-ai/ideogram-4`. Ideogram 4 uses a **dual-transformer architecture**: a conditional transformer (text + image tokens) and a separately-weighted unconditional transformer (image tokens only, zero LLM features). Both transformers shrink at coarse resolution, providing the same token-ratio benefit as single-transformer models. > **Note:** Ideogram 4's logit-normal noise schedule (`std=1.75`, `mu=0`) concentrates steps near the mid-sigma range. Fewer steps fall in the high-sigma coarse-eligible region compared to FLUX, which limits the achievable speedup at a given δ. ### Usage **20-step (V4\_DEFAULT\_20 preset)** ```bash theme={null} sglang generate \ --model-path ideogram-ai/ideogram-4 \ --prompt "A serene mountain lake at golden hour, photorealistic" \ --height 1024 --width 1024 \ --num-inference-steps 20 \ --dit-cpu-offload false \ --progressive-mode dct_rewind \ --progressive-levels 1 \ --progressive-delta 0.05 ``` **48-step (V4\_QUALITY\_48 preset)** ```bash theme={null} sglang generate \ --model-path ideogram-ai/ideogram-4 \ --prompt "A serene mountain lake at golden hour, photorealistic" \ --height 1024 --width 1024 \ --num-inference-steps 48 \ --dit-cpu-offload false \ --progressive-mode dct_rewind \ --progressive-levels 1 \ --progressive-delta 0.05 ``` ### Benchmark Hardware: RTX A6000 48 GB, `torch_sdpa`, `--dit-cpu-offload false`. Timing = denoising loop only. **20-step (V4\_DEFAULT\_20)** | Config | Stage split | Denoise | Speedup | | - | - | - | - | | Fullres (baseline) | 20 @ 64² | 53.99 s | 1.00× | | dct\_rewind L1 δ=0.01 | 6 @ 32² + 14 @ 64² | 43.47 s | **1.24×** | | dct\_rewind L1 δ=0.05 | 9 @ 32² + 11 @ 64² | 38.14 s | **1.42×** | | dct\_rewind L1 δ=0.10 | 11 @ 32² + 9 @ 64² | 34.60 s | **1.56×** | **48-step (V4\_QUALITY\_48)** | Config | Stage split | Denoise | Speedup | | - | - | - | - | | Fullres (baseline) | 48 @ 64² | 130.92 s | 1.00× | | dct\_rewind L1 δ=0.01 | 12 @ 32² + 36 @ 64² | 109.79 s | **1.19×** | | dct\_rewind L1 δ=0.05 | 21 @ 32² + 27 @ 64² | 93.83 s | **1.40×** | | dct\_rewind L1 δ=0.10 | 26 @ 32² + 22 @ 64² | 84.94 s | **1.54×** | ### Python API ```python theme={null} from sglang.multimodal_gen import DiffGenerator gen = DiffGenerator.from_pretrained( model_path="ideogram-ai/ideogram-4", dit_cpu_offload=False, ) result = gen.generate(sampling_params_kwargs={ "prompt": "A serene mountain lake at golden hour, photorealistic", "num_inference_steps": 48, "height": 1024, "width": 1024, "progressive_mode": "dct_rewind", "progressive_levels": 1, "progressive_delta": 0.05, }) ``` *** ## Limitations * **Sequence parallelism incompatible.** Cannot be combined with `--ulysses-degree` or `--ring-degree`. The stage raises a `RuntimeError` if SP is enabled. * **torch.compile incompatible.** Compiled kernels have a fixed sequence length; the resolution transition causes a recompile or error. Use progressive without `--enable-torch-compile`. * **Cache-DiT interaction is experimental.** The stage refreshes Cache-DiT context at resolution transitions, but quality and speedup should be benchmarked before relying on this combination. ## References * [Spectral Progressive Diffusion (arXiv 2605.18736)](https://arxiv.org/abs/2605.18736) * [SGLang Diffusion Performance Optimization](./performance-optimization) # Quantization Source: https://docs.sglang.io/docs/sglang-diffusion/quantization SGLang-Diffusion treats component path selection and quantized checkpoint materialization as separate capabilities. Every loaded component can use an independent checkpoint path, but that checkpoint is quantized only when its selected loader supports the serialized format. Transformer, encoder, and VAE precision are resolved independently. ## Quick Reference Choose the option from the source you have, not from the precision name: | Source or operation | CLI | Result | | - | - | - | | Complete component repo/directory | `--component-paths. ` | Replaces that component's config and weights. | | Weight file/directory for an existing component | `--component-weights-paths. ` | Keeps the base component config and replaces only its weights. | | Online quantization of unquantized weights | `--component-quantizations. ` | Constructs a supported quantized implementation during load; use `--component-quantization-ignored-layers.` for exclusions. | | Primary DiT convenience spellings | `--transformer-path`, `--transformer-weights-path`, `--quantization` | Aliases for pipelines with one primary DiT; they remain supported but are not generic component names. | | Causal KV-cache compression | `--kv-cache-quant` | Quantizes completed runtime cache chunks, not checkpoint weights. | `--model-path` always selects the base model. The component key must be a real key from its `model_index.json` or native pipeline registry; examples include `transformer`, `transformer_2`, `text_encoder`, `video_vae`, and `audio_vae`, but no one name is universal. Recommended example for pre-quantized checkpoints: ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.2-dev \ --component-weights-paths.transformer black-forest-labs/FLUX.2-dev-NVFP4 \ --prompt "a curious pikachu" ``` For quantized transformers-style transformer component folders: ```bash theme={null} sglang generate \ --model-path /path/to/base-model \ --component-paths.transformer /path/to/quantized-transformer \ --prompt "A Logo With Bold Large Text: SGL Diffusion" ``` NOTE: Some model-specific integrations also accept a quantized repo or local directory directly as `--model-path`, but that is a compatibility path. If a repo contains multiple candidate checkpoints, select an exact component or weight source explicitly. Pre-quantized files are self-describing. Pass one selected file by local path, `owner/repo/path/file.safetensors`, or direct Hugging Face file URL and do not add an online quantization option. The [MiniMax-H3 compatibility table](/cookbook/diffusion/MiniMax/MiniMax-H3#checkpoint-and-adapter-formats) is the canonical list of H3 component formats and published example sources; this page defines the backend families and their shared constraints. ## Quantized Component Repositories Every loaded component can point to an independent repository, but path routing does not imply that every loader can materialize every quantization format. SGLang resolves quantized component checkpoints through one of three explicit paths: * load through an SGLang quantization implementation; * delegate a standard component to Transformers or Diffusers; * fail closed in a native plain-state loader when the format cannot be restored. | Component path | Quantized checkpoint behavior | | - | - | | `transformer`, `transformer_2`, `unconditional_transformer`, `audio_dit`, `video_dit` | Uses the SGLang transformer quantization adapters documented below. | | `text_encoder*`, `image_encoder*` | Prefers a compatible registered native encoder. Otherwise, a standard top-level `quantization_config` recognized by the installed Transformers version is delegated to its `from_pretrained` path; unsupported formats and metadata locations fail closed. | | `vae`, `video_vae`, `audio_vae` | A standard top-level Diffusers `quantization_config` is delegated to `AutoModel.from_pretrained`. Native-only VAEs and nested/compression metadata fail closed. | | Library-managed Transformers or Diffusers components | Delegates to the upstream `from_pretrained` path and inherits its format support and validation behavior. The local PE model uses this path; compatible formats remain model-specific. | | Native auxiliary components that load raw state dicts | Quantized checkpoints are rejected before model construction until that component has a quantized materialization implementation. This includes connectors, duration heads, bridges, diffusion decoders, sound tokenizers, spatial upsamplers, and vocoders. | `--quantization` is the explicit override for the primary transformer loader; `--component-quantizations.` expresses the same intent for one supported component. Pair the latter with `--component-quantization-ignored-layers.` to keep matching layers unquantized. Pre-quantized component repositories instead select their format through their own metadata and the capability of the selected loader. A generic string override without a matching materialization backend would advertise support that the component does not have, while a quantized weight file without matching config metadata cannot be identified or restored generically. ## Quant Families Here, `quant_family` means a checkpoint and loading family with shared CLI usage and loader behavior. It is not just the numeric precision or a kernel backend.
quant\_family checkpoint form checkpoint selector supported models extra dependency platform / notes
fp8 / mxfp4 (online quantization) Unquantized checkpoint (offline via AMD Quark coming soon) --component-quantizations.\ \{fp8,mxfp4} Z-Image-Turbo (validated), others likely work. More support coming soon. MXFP4: aiter on ROCm MXFP4 requires ROCm and MI350+ (gfx95x). Weights quantized at load time, activations quantized to fp8 / mxfp4 dynamically.
convrot\_int8 (online quantization; kitchen\_int8 is a deprecated alias) Unquantized BF16/FP16 checkpoint --component-quantizations.\ convrot\_int8 MiniMax-H3, Qwen-Image, Qwen-Image-Edit (quality-validated); other BF16 DiTs load but are not quality-validated SGLang JIT ops (bundled, need nvcc) or comfy-kitchen Data-free INT8 ConvRot W8A8 at load time, per layer on the fused JIT-compiled ops (CC 9.0, 10.0, 12.0, 12.1) or on comfy\_kitchen.int8\_linear (Turing+); SGLANG\_DIFFUSION\_CONVROT\_INT8\_BACKEND forces one. Layers whose input dim is not divisible by the group size stay in BF16. TP, LoRA, FSDP and memory notes under ConvRot INT8 below.
fp8 (offline quantization) Quantized transformer component folder, or safetensors with quantization\_config metadata --component-paths.\ or --component-weights-paths.\ Native DiTs whose linear layers support the selected FP8 method; validate quality per model None Component-folder and single-file flows are both supported
modelopt-fp8 Converted ModelOpt FP8 transformer directory or repo with config.json --component-paths.\ FLUX.1, FLUX.2, Wan2.2, HunyuanVideo, Qwen Image, Qwen Image Edit None Serialized config stays quant\_method=modelopt with quant\_algo=FP8; dit\_layerwise\_offload is supported and dit\_cpu\_offload stays disabled
auto-round W4A16 Transformer component repo with a self-describing quantization\_config and auto\_round:auto\_gptq packing --component-paths.\ Native dense DiTs with compatible component parameter mappings; MiniMax-H3 Diffusers components are supported None Auto-detected; reuses the SRT GPTQ/Marlin backend. No --quantization flag is needed; use TP/sequence parallelism rather than FSDP.
modelopt-nvfp4 Mixed transformer directory/repo with config.json, raw or Comfy layer-marked NVFP4 safetensors, or full ModelOpt Diffusers repo --component-paths.\ for component repos; --component-weights-paths.\ for raw files; --model-path for full repos FLUX.1, FLUX.2, Wan2.2, Qwen Image, Qwen Image 2512, Qwen Image Edit, Qwen Image Edit 2511, MiniMax-H3 None Mixed override repos keep the base model separate; full Qwen Image exports can be loaded directly as --model-path; raw exports such as black-forest-labs/FLUX.2-dev-NVFP4 use the weights-path flow. Comfy markers select NVFP4 plus INT8 or FP8 companion linears automatically; omit --quantization.
gguf One selected GGUF DiT or native encoder file --component-weights-paths.\ MiniMax-H3 original or pruned FL2VA / Ref2VA DiTs and its Qwen3-VL text encoder None CUDA only and auto-detected; supports standard and K-quant GGML types. TP requires aligned GGML blocks; component/layerwise offload is supported and FSDP is not.
comfy-fp8 One selected Comfy safetensors file with per-layer comfy\_quant metadata --component-weights-paths.\ MiniMax-H3 pruned FL2VA / Ref2VA DiTs None CUDA; auto-detected; TP, sequence parallelism, and component/layerwise offload are supported, while FSDP is not. Checkpoint-marked fc2 layers retain FP8 storage and use compute-dtype matmul.
comfy-int8-convrot One selected safetensors file with per-layer int8\_tensorwise and ConvRot metadata --component-weights-paths.\ Native DiTs and encoders whose parameter mappings preserve each marked linear; MiniMax-H3 DiT and Qwen3-VL encoder checkpoints have validated tensor contracts SGLang JIT ops (bundled, need nvcc) or comfy-kitchen CUDA; auto-detected; each layer runs on the JIT-compiled ConvRot ops where they apply (CC 9.0/10.0/12.0/12.1, group size 64 or 256) and on comfy\_kitchen.int8\_linear otherwise; weight/scale layout is validated before model construction. TP requires every row-parallel input shard to preserve the checkpoint's ConvRot group boundary. The H3 256-group DiT supports TP1/2/4, not TP8; its Qwen3-VL encoder keeps TP8 by replicating only incompatible row projections. Offload is supported; FSDP is not.
mxfp8 Self-describing serialized weights, or BF16/FP16 weights for online quantization --component-weights-paths.\, or --component-quantizations.\ mxfp8 online Native diffusion linears; mixed MiniMax-H3 checkpoints are selected per layer SRT's platform MXFP8 backend Serialized metadata is auto-detected. NVIDIA and ROCm reuse SRT's dense MXFP8 kernels; Ascend keeps its native online path. Mixed per-layer checkpoints do not support FSDP.
comfy-w4a8-convrot Safetensors with serialized asym\_w4a8\_int8 layer metadata and packed weights --component-weights-paths.\ MiniMax-H3 FL2VA / Ref2VA DiTs and its native Qwen3-VL encoder comfy-kitchen>=0.2.27 Auto-detected; omit --quantization. Requires SM80+ and validates packed weights, group/channel scales, and optional codebooks before model construction. Mixed encoder files may keep their embedding tensorwise INT8. TP must preserve ConvRot group boundaries; offload is supported and FSDP is not.
comfy-w4a4-convrot Safetensors with serialized convrot\_w4a4 metadata, optionally mixed with int8\_tensorwise layers --component-weights-paths.\ Native DiTs and encoders with matching parameter mappings; MiniMax-H3 FL2VA / Ref2VA DiTs and Qwen3-VL encoder layouts are recognized comfy-kitchen Auto-detected; omit --quantization. Each layer dispatches to its serialized W4A4 or INT8 ConvRot kernel. CUDA requires SM75+; TP must preserve each format's quantization and ConvRot group boundaries. Offload is supported and FSDP is not.
comfy-nvfp4 Safetensors with serialized nvfp4 and optional scalar/row-wise int8\_tensorwise layer metadata --component-weights-paths.text\_encoder MiniMax-H3 native Qwen3-VL encoder comfy-kitchen for NVFP4 matmul Auto-detected; omit explicit quantization. Preserves high-nibble-first weights, swizzled block scales, and optional AWQ input pre-scales. Layers declaring full\_precision\_matrix\_mult use BF16/FP16 compute with packed storage. Other NVFP4 layers use dynamic activation quantization and NVFP4 matmul on NVIDIA SM100+. Companion INT8 embeddings support scalar or per-row scales without expanding the full table.
quanto-int8 One native encoder safetensors file with an embedded Quanto quantization map --component-weights-paths.\ Native encoders whose mapped linear layers consume every declared qint8 entry; MiniMax-H3's Qwen3-VL encoder is supported None Auto-detected weight-only qint8 storage. Each active matrix is dequantized to the compute dtype for the ordinary linear operation, so this reduces stored/resident weight memory rather than promising INT8 GEMM speed. TP and offload are supported; FSDP is not.
qvg-kv Unquantized model with runtime causal KV-cache compression --kv-cache-quant \{int4,int2} LingBot World realtime causal path quant-videogen CUDA only; compresses completed cache chunks rather than model weights; lossy and disabled by default
nunchaku-svdq Pre-quantized Nunchaku transformer weights, usually named svdq-\{int4|fp4}\_r\{rank}-... --transformer-weights-path Model-specific support such as Qwen-Image, FLUX, and Z-Image nunchaku SGLang can infer precision and rank from the filename and supports both int4 and nvfp4
msmodelslim Pre-quantized msmodelslim transformer weights --model-path Wan2.2 family None Currently only compatible with the Ascend NPU family and supports mxfp8, mxfp4, w8a8, and w4a4
## Causal KV-Cache Quantization Quant-VideoGen KV-cache quantization targets long-running autoregressive video sessions, where the causal self-attention cache can become comparable to the model weights. It does not change or quantize the checkpoint weights. See [Realtime and Causal Video Models](./realtime_models) for the session lifecycle, supported pipelines, and the distinction between realtime and request-based causal generation. Install the optional dependency without allowing its stale Torch requirement to replace SGLang's pinned Torch version, then enable int4 compression when serving a supported LingBot World realtime pipeline: ```bash theme={null} pip install "sglang[diffusion,diffusion-qvg]" pip install --no-deps quant-videogen==0.1.0 sglang serve \ --model-path robbyant/lingbot-world-fast-diffusers \ --pipeline-class-name LingBotWorldCausalDMDPipeline \ --num-gpus 4 \ --ulysses-degree 4 \ --kv-cache-quant int4 \ --dit-cpu-offload false \ --text-encoder-cpu-offload false ``` ### Storage Policy The current chunk is rewritten at every denoising step, so it remains in BF16. The newest `--kv-cache-quant-keep-recent` completed chunks also remain in BF16. Older completed chunks are stable and are packed once with Progressive Residual Quantization (PRQ); their dense BF16 tensors are then released. When a transformer layer runs attention, its packed visible chunks are dequantized and concatenated with the recent BF16 chunks. This creates one layer's dense attention view at a time instead of keeping dense windows resident for every transformer layer. ### How PRQ Works For each K or V vector, PRQ uses k-means to select a centroid, then quantizes the remaining error: ```text theme={null} x = centroid_1 + residual_1 residual_1 = centroid_2 + residual_2 ... x_hat = centroid_1 + centroid_2 + ... + dequantize(low_bit_residual) ``` Each additional stage applies another centroid lookup to the previous stage's residual. SGLang's default uses one stage, 128 centroids, and an int4 or int2 block-quantized residual. More stages or centroids can reduce reconstruction error but add codebook storage and packing work. PRQ is the compression algorithm; selecting older completed chunks is the runtime storage policy that makes it practical. Stable chunks are compressed once, while mutable and recent chunks avoid repeated packing and retain higher precision. ### Quality And Performance KV-cache quantization is lossy. Disabling it uses the original dense BF16 cache and is bit-exact with the unmodified path. Enabling int4 or int2 reconstructs an approximation of K and V, so fixed-seed generated frames are not expected to be pixel-identical to BF16. In the initial LingBot measurements, int4 used about 47% of the dense resident KV-cache memory for a 24-frame window and added about 18% per-chunk latency. Int2 used about 37% of the dense resident KV-cache memory but introduces more quantization error. These measurements are configuration-specific; benchmark memory, latency, temporal consistency, identity stability, and motion quality on the intended session length. Start with int4 unless capacity requires int2. The current implementation is limited to the LingBot realtime sliding-window-and-sink path, including Ulysses sequence sharding. It does not support LongLive2 pinned sinks, global sinks, or dynamically growing caches. ### Tuning Options | Option | Default | Effect | | - | -: | - | | `--kv-cache-quant {off,int4,int2}` | `off` | Enables QVG KV-cache compression and selects residual precision. | | `--kv-cache-quant-stages` | `1` | Number of progressive centroid-residual stages. | | `--kv-cache-quant-centroids` | `128` | Number of k-means centroids per stage. | | `--kv-cache-quant-block-size` | `64` | Block size used to quantize the final residual. | | `--kv-cache-quant-iters` | `2` | K-means iterations used while packing a chunk. | | `--kv-cache-quant-asymmetric` | disabled | Uses asymmetric residual quantization. | | `--kv-cache-quant-keep-recent` | `1` | Number of newest completed chunks retained in BF16. | | `--kv-cache-quant-sink {0,1}` | `1` | Whether to quantize completed sink chunks. | | `--kv-cache-quant-sink-keep` | `0` | Number of leading sink chunks retained in BF16. | ## Online Quantization This section describes the online methods currently implemented by the transformer loader. They are useful when a pre-quantized transformer checkpoint is not available. Encoders, VAEs, and auxiliary components are independent: their repositories may still carry serialized quantized weights, which are restored when the selected component loader supports that format. ### FP8 Online Quantization Apply FP8 quantization to a supported unquantized DiT checkpoint: ```bash theme={null} sglang generate \ --model-path Tongyi-MAI/Z-Image-Turbo \ --component-quantizations.transformer fp8 \ --prompt "a beautiful sunset" \ --save-output ``` For a pipeline whose primary DiT is named `transformer`, the shorter `--quantization fp8` spelling is equivalent. MiniMax-H3 supports this path while preserving its required FP32 patch, timestep, and output projections. See the [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#9-feature-contracts-and-advanced-recipes) for its distributed serving recipe. ### MXFP4 Online Quantization MXFP4 provides aggressive 4-bit compression with online quantization. **Note: Requires ROCm and MI350+ (gfx95x) GPU.** ```bash theme={null} sglang generate \ --model-path Tongyi-MAI/Z-Image-Turbo \ --component-quantizations.transformer mxfp4 \ --prompt "a beautiful sunset" \ --save-output ``` **Note:** Requires `aiter` package with MXFP4 kernel support ### ConvRot INT8 `convrot_int8` (formerly `kitchen_int8`, still accepted as a deprecated alias) is online W8A8 INT8 for DiTs and compatible native encoders: every linear whose input dim is divisible by the Hadamard group size (256) is rotated and quantized per output channel after loading a stock BF16 checkpoint (ConvRot, arXiv:2512.03673; no calibration data). Each forward call runs one fused op per linear: group-wise Hadamard rotation, dynamic per-row INT8 activation quantization, INT8 GEMM, dequantization and bias add. Two kernel backends implement the same quantization and consume the same INT8 weights and per-row scales, so the choice is made per layer at load time: * `jit`: the `convrot_int8` ops in `sglang.kernels.ops.gemm.convrot_int8`, compiled by SGLang's JIT for the local GPU on first use (about a minute with `nvcc` on the machine; cached under `SGLANG_JIT_CACHE_DIR`, default `~/.cache/sglang/jit`, for later runs). CC 9.0 (Hopper), 10.0 (B200, GB200) and 12.0 / 12.1 (RTX PRO 6000 Blackwell, RTX 50 series, DGX Spark); group size 64, 128, 256 or 512; output width a multiple of 8. CC 10.3 (B300, GB300) is refused on purpose: Blackwell Ultra cuts INT8 tensor-core throughput to a fraction of its BF16 rate, so use FP8 or NVFP4 there. Linears sharing one input (a block's Q/K/V projections) rotate and quantize it once, and the Qwen-Image FFN down-projection folds its GELU(tanh) into the rotate kernel; both are bitwise identical to the per-linear ops. * `comfy_kitchen`: `comfy_kitchen.int8_linear` from the optional `comfy-kitchen` package (`pip install comfy-kitchen`). Turing and later (SM75); group size 16, 64 or 256. The default `auto` takes `jit` wherever its ops build and run (no CUDA toolchain, or a failed build, counts as unavailable and is logged) and `comfy_kitchen` otherwise; loading stops with an install hint if `comfy-kitchen` is then missing. A layer whose output width is not a multiple of 8 goes to `comfy_kitchen` when it is installed and stays BF16 otherwise. `SGLANG_DIFFUSION_CONVROT_INT8_BACKEND=jit` or `=comfy_kitchen` forces one backend. LoRA adapters run in dynamic mode (they cannot be merged into rotated INT8 weights); under FSDP `auto` resolves to `comfy_kitchen` and an explicit `jit` is refused. Serialized Comfy ConvRot INT8 DiTs and native encoders use `--component-weights-paths.`; they are auto-detected from per-layer markers, load INT8 weights and row scales directly (omit online quantization), and follow the same per-layer backend choice. `--transformer-weights-path` remains the primary-DiT convenience spelling. ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --quantization convrot_int8 \ --prompt "a beautiful sunset" \ --save-output ``` ```bash theme={null} sglang generate \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --component-quantizations.transformer convrot_int8 \ --attention-backend fa \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder \ --dit-offload-prefetch-size 1 \ --dit-layerwise-resident-layers 0 \ --enable-torch-compile false \ --prompt "A cat walking on a sunny beach, gentle waves." \ --save-output ``` Quantization runs after the model's weight loaders, so MiniMax-H3's grouped `qkv` reorder is already applied. Layers whose input dim is not divisible by the group size stay in BF16 instead of failing the load (H3's AdaLN projections take that path); under TP every row-parallel input shard must also divide it, or the load stops naming the layer, which `--quantization-ignored-layers` keeps in BF16. Without offload, loading stages the BF16 weights on the GPU before quantizing them, so it needs the BF16 checkpoint's worth of free VRAM (38 GiB for Qwen-Image); under layerwise offload the weights stay on the host and are quantized one layer at a time. `--quantization-ignored-layers img_mod txt_mod txt_mlp.net.2` keeps the few-row Qwen-Image GEMMs (AdaLN modulation, text-stream FFN down-projection) in BF16 at the cost of memory (45.5 GB peak) for no consistent speed gain, so the default quantizes everything. Measured against BF16 on GB200 in eager execution with every eligible layer quantized: | Model | Backend | Speedup vs BF16 | Peak GPU memory | | - | - | -: | - | | Qwen-Image 1024x1024, 50 steps, CFG 4, 1 GPU | `jit` | 1.19x (1.13x on 2 GPUs, CFG parallel) | 56 -> 37 GB | | Qwen-Image-Edit 1024x1024, 1 GPU | `jit` | 1.32x | | | MiniMax-H3 1344x768, 5 s, 50 steps, 1 GPU | `jit` | 1.07x (1.15x on 4 GPUs, Ulysses 4) | 125 -> 107 GiB | | MiniMax-H3, same | `comfy_kitchen` | 0.86x | 125 -> 107 GiB | On MiniMax-H3 both backends quantize the same 209 linears and sit in the same quality class against BF16 over six prompt/seed pairs (video PSNR 23.4 vs 23.8 dB, SSIM 0.80 vs 0.81, audio spectral similarity 0.96 vs 0.97; the same-seed distance between the two backends equals each one's distance to BF16). Loading an online-quantized H3 without offload peaks at 115 GiB resident; the serialized Comfy INT8 checkpoint loads at 96 GiB resident and runs on `jit` at the same speed. `convrot_int8` is approximate: same-seed outputs are not bit-exact with BF16 or between the two backends (both stay within INT8 quantization noise of BF16), so it is not a consistency ground-truth mode. Quality was validated against BF16 on MiniMax-H3, Qwen-Image and Qwen-Image-Edit; other DiTs load but are not quality-validated. Without `convrot_int8` neither backend is loaded and the BF16 path is untouched. See the [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#9-feature-contracts-and-advanced-recipes) for the 24 GB offload recipe, including why `vae` must stay out of `--layerwise-offload-components`. With the `comfy_kitchen` backend, large-M GEMMs (`rows > 8192` and `out_features >= 8192`) are split by rows so the fused kernel stays on the data-parallel CUTLASS config. Override the thresholds with `SGLANG_KITCHEN_INT8_MAX_ROWS` and `SGLANG_KITCHEN_INT8_MIN_SPLIT_N`. ### Skipping Layers By default, transformer online quantization quantizes every supported linear layer in that component. However, `--quantization-ignored-layers` can keep specific transformer layers in their original precision: ```bash theme={null} sglang generate \ --model-path Tongyi-MAI/Z-Image-Turbo \ --quantization fp8 \ --quantization-ignored-layers attention.to_ \ --prompt "a beautiful sunset" \ --save-output sglang generate \ --model-path Tongyi-MAI/Z-Image-Turbo \ --quantization mxfp4 \ --quantization-ignored-layers attention.to_ \ --prompt "a beautiful sunset" \ --save-output ``` Each pattern is matched against the full layer prefix (e.g. `layers.0.attention.to_q`). A layer is skipped and left unquantized if its prefix contains any of the given patterns. ## Transformers-managed Quantized Components Model components that already have a native Transformers loading path delegate self-describing checkpoints whose standard top-level `quantization_config` is recognized by the installed Transformers version. Successful loading still depends on that backend's optional dependencies, platform, and model/checkpoint compatibility. An available native SGLang implementation remains preferred when it can restore the same format. For example, replace FLUX's T5 component with the official BitsAndBytes checkpoint: ```bash Command theme={null} sglang serve \ --model-path black-forest-labs/FLUX.1-dev \ --component-paths.text_encoder_2 \ diffusers/FLUX.1-dev-bnb-4bit/text_encoder_2 ``` Transformers owns format validation and places the quantized component directly on its resident device. SGLang rejects component/layerwise offload, FSDP, nonstandard metadata locations, unsupported upstream formats, and native-only fallbacks instead of silently changing the checkpoint contract. Diffusion DiT components declared under the Diffusers library use the separate quantization backends documented above. ## Validated ModelOpt Checkpoints This section is the canonical support matrix for the thirteen published diffusion ModelOpt checkpoints currently wired up in SGLang docs and validation coverage. Published checkpoints keep the serialized quantization config as `quant_method=modelopt`; the FP8 vs NVFP4 split below is a documentation label derived from `quant_algo`. Twelve of the thirteen repos live under `lmsys/*`. The FLUX.2 NVFP4 entry keeps the official `black-forest-labs/FLUX.2-dev-NVFP4` repo.
Quant Algo Base Model Preferred CLI HF Repo Current Scope Notes
FP8 black-forest-labs/FLUX.1-dev --transformer-path lmsys/flux1-dev-modelopt-fp8-sglang-transformer single-transformer override, deterministic latent/image comparison, H100 benchmark, torch-profiler trace SGLang converter keeps a validated BF16 fallback set for modulation and FF projection layers; use --model-id FLUX.1-dev for local mirrors
FP8 black-forest-labs/FLUX.2-dev --transformer-path lmsys/flux2-dev-modelopt-fp8-sglang-transformer single-transformer override load and generation path published SGLang-ready transformer override
FP8 Wan-AI/Wan2.2-T2V-A14B-Diffusers --transformer-path lmsys/wan22-t2v-a14b-modelopt-fp8-sglang-transformer primary transformer quantized, transformer\_2 kept BF16 primary-transformer-only path; keep transformer\_2 on the base checkpoint, and do not describe this as dual-transformer full-model FP8 unless that path is validated separately
FP8 hunyuanvideo-community/HunyuanVideo --transformer-path lmsys/hunyuanvideo-modelopt-fp8-sglang-transformer single-transformer override, BF16-vs-FP8 video comparison, H100 benchmark, torch-profiler trace HunyuanVideo uses different ModelOpt/diffusers and SGLang runtime module names; the converter maps those names before writing FP8 scale tensors and BF16 fallback ignores
FP8 Qwen/Qwen-Image --transformer-path lmsys/qwen-image-modelopt-fp8-sglang-transformer single-transformer override, BF16-vs-FP8 image comparison, H100 benchmark, torch-profiler trace shares the Qwen Image FP8 fallback preset; keep img\_in, txt\_in, timestep embedder, norm\_out.linear, proj\_out, img\_mod/txt\_mod, and img\_mlp.net.2 in BF16
FP8 Qwen/Qwen-Image-Edit-2511 --transformer-path lmsys/qwen-image-edit-modelopt-fp8-sglang-transformer TI2I edit path, BF16-vs-FP8 image comparison, H100 benchmark shares QwenImageTransformer2DModel with Qwen Image and uses the same Qwen Image FP8 fallback preset
NVFP4 black-forest-labs/FLUX.1-dev --transformer-path lmsys/flux1-dev-modelopt-nvfp4-sglang-transformer mixed BF16+NVFP4 transformer override, correctness validation, 4x RTX 5090 benchmark, torch-profiler trace use build\_modelopt\_nvfp4\_transformer.py; validated builder keeps selected FLUX.1 modules in BF16 and sets swap\_weight\_nibbles=false
NVFP4 black-forest-labs/FLUX.2-dev --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 packed-QKV load path official raw export repo; validated packed export detection and runtime layout handling
NVFP4 Wan-AI/Wan2.2-T2V-A14B-Diffusers --transformer-path lmsys/wan22-t2v-a14b-modelopt-nvfp4-sglang-transformer primary transformer quantized with ModelOpt NVFP4, transformer\_2 kept BF16 primary-transformer-only path; keep transformer\_2 on the base checkpoint; the default FP4 GEMM backend is flashinfer\_trtllm
NVFP4 Qwen/Qwen-Image --model-path lmsys/qwen-image-modelopt-nvfp4-sglang full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison full repo loaded directly; exported with ModelOpt PR #1706 SVDQuant NVFP4 (--format fp4, max calibration, block size 16) and BF16 fallbacks for attention-sensitive modules plus first/last transformer blocks
NVFP4 Qwen/Qwen-Image-2512 --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison, B200 CI case same full-repo loader path as Qwen Image; this is the Qwen Image NVFP4 representative in multimodal-gen-test-1-b200
NVFP4 Qwen/Qwen-Image-Edit --model-path lmsys/qwen-image-edit-modelopt-nvfp4-sglang TI2I edit full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison full repo loaded directly with normal image-edit inputs; exported with the same ModelOpt PR #1706 NVFP4 recipe
NVFP4 Qwen/Qwen-Image-Edit-2511 --model-path lmsys/qwen-image-edit-2511-modelopt-nvfp4-sglang TI2I edit full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison full repo loaded directly with normal image-edit inputs; exported with the same ModelOpt PR #1706 NVFP4 recipe
These thirteen checkpoints are the intended ModelOpt documentation support set. The B200 diffusion CI job (`multimodal-gen-test-1-b200`) uses a representative NVFP4 subset and includes `lmsys/qwen-image-2512-modelopt-nvfp4-sglang` for Qwen Image coverage. ## ModelOpt FP8 ### Usage Examples Converted ModelOpt FP8 transformer repos should be loaded as transformer component overrides. If the repo or local directory already contains `config.json`, use `--transformer-path`. Full Diffusers repos such as the NVIDIA Wan2.2 FP8 checkpoint can be passed directly with `--model-path`. ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.2-dev \ --transformer-path lmsys/flux2-dev-modelopt-fp8-sglang-transformer \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --save-output ``` ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --transformer-path lmsys/wan22-t2v-a14b-modelopt-fp8-sglang-transformer \ --prompt "a fox walking through neon rain" \ --save-output ``` ```bash theme={null} sglang generate \ --model-path hunyuanvideo-community/HunyuanVideo \ --transformer-path lmsys/hunyuanvideo-modelopt-fp8-sglang-transformer \ --height 544 --width 960 --num-frames 17 \ --prompt "A cinematic shot of a red sports car driving through rain at night" \ --save-output ``` ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --transformer-path lmsys/qwen-image-modelopt-fp8-sglang-transformer \ --prompt "A tiny astronaut reading a book under a glass greenhouse" \ --save-output ``` ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image-Edit-2511 \ --transformer-path lmsys/qwen-image-edit-modelopt-fp8-sglang-transformer \ --image-path /path/to/input.png \ --prompt "Turn the scene into a warm watercolor illustration" \ --save-output ``` ### Notes * `--transformer-path` is the primary-DiT convenience spelling for converted ModelOpt FP8 component repos or directories that already carry `config.json`. * If the override repo or local directory contains its own `config.json`, SGLang reads the quantization config from that override instead of relying on the base model config. * `--transformer-weights-path` still works when you intentionally point at raw weight files or a directory that should be metadata-probed as weights first. * `dit_layerwise_offload` is supported for ModelOpt FP8 checkpoints. * `dit_cpu_offload` still stays disabled for ModelOpt FP8 checkpoints. * The layerwise offload path now preserves the non-contiguous FP8 weight stride expected by the runtime FP8 GEMM path. * On disk, the quantization config stays `quant_method=modelopt` with `quant_algo=FP8`; the `modelopt-fp8` label in this document is a support family name, not a serialized config key. * To build the converted checkpoint yourself from a ModelOpt diffusers export, use `python -m sglang.multimodal_gen.tools.build_modelopt_fp8_transformer`. ## ModelOpt NVFP4 ### Usage Examples For mixed ModelOpt NVFP4 transformer overrides that already contain `config.json`, keep the base model and quantized transformer separate and use `--transformer-path`: ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.1-dev \ --transformer-path lmsys/flux1-dev-modelopt-nvfp4-sglang-transformer \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --save-output ``` For raw NVFP4 exports such as the official FLUX.2 release, use `--transformer-weights-path`: ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.2-dev \ --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --save-output ``` SGLang also supports passing the NVFP4 repo or local directory directly as `--model-path`: ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.2-dev-NVFP4 \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --save-output ``` For a dual-transformer Wan2.2 export where only the primary `transformer` was quantized: ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --transformer-path lmsys/wan22-t2v-a14b-modelopt-nvfp4-sglang-transformer \ --prompt "a fox walking through neon rain" \ --save-output ``` For full Qwen Image NVFP4 exports, load the published repo directly: ```bash theme={null} sglang generate \ --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \ --prompt "A tiny astronaut reading a book under a glass greenhouse" \ --save-output ``` For high-resolution Qwen-Image-family generations on B200, the FlashInfer CUTLASS FP4 GEMM backend can be faster than the default TensorRT-LLM backend: ```bash theme={null} SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass \ sglang generate \ --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \ --width 2048 --height 2048 \ --prompt "A tiny astronaut reading a book under a glass greenhouse" \ --save-output ``` Qwen-Image 2.1 also supports independent NVFP4 DiT and native language-encoder component directories through `--component-paths.transformer` and `--component-paths.text_encoder`. Private max-calibrated exports passed B200 generation, editing, transparent RGBA, offload, and TP2 checks. These checks do not cover arbitrary exports or RTX 5090. See its [cookbook](/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1#nvfp4-components) for component-loading instructions and hardware requirements. ### Notes * Use `--transformer-path` for mixed ModelOpt NVFP4 transformer repos or local directories that already include `config.json`. * Use `--transformer-weights-path` for raw NVFP4 exports, individual safetensors files, or repo layouts that should be treated as weights first. * For dual-transformer pipelines such as `Wan2.2-T2V-A14B-Diffusers`, the primary `--transformer-path` override targets only `transformer`. Use a per-component override such as `--transformer-2-path` only when you intentionally want a non-default `transformer_2`. * On B200, the diffusion ModelOpt NVFP4 path defaults to FlashInfer TensorRT-LLM FP4 GEMM (`flashinfer_trtllm`). On SM120, including RTX 5090, the default is `auto`; TensorRT-LLM FP4 GEMM does not support that architecture. * The published Qwen Image NVFP4 exports keep the `img_mod`/`txt_mod` modulation projections and first/last transformer blocks in BF16. * Qwen-Image NVFP4 does not always improve latency at 1024x1024. On B200, the validated ModelOpt exports were faster than BF16 at 2048x2048 with `SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass`, while 1024x1024 remained BF16-faster. * Direct `--model-path` loading is the canonical path for full Qwen Image ModelOpt NVFP4 repos and a compatibility path for FLUX.2 NVFP4-style repos or local directories. * If `--transformer-weights-path` is provided explicitly, it takes precedence over the compatibility `--model-path` flow. * For local directories, SGLang first looks for `*-mixed.safetensors`, then falls back to loading from the directory. * To force the diffusion ModelOpt FP4 path onto a different FlashInfer backend, set `SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND`. Supported values include `flashinfer_cudnn`, `flashinfer_cutlass`, and `flashinfer_trtllm`. * On disk, the quantization config stays `quant_method=modelopt` with `quant_algo=NVFP4`; the `modelopt-nvfp4` label here is again a documentation family name rather than a serialized config key. ## GGUF GGUF loads a community-quantized transformer from a single `.gguf` file while the rest of the pipeline — VAE, text encoder, scheduler, tokenizer — keeps loading from the base model. GGUF primarily reduces checkpoint, host-memory, and resident-weight size. For example, MiniMax-H3's transformer is 17.5 GiB as Q4\_K\_M versus 61.7 GiB as BF16. With full layerwise offload, VAE decode and offload buffers can still dominate peak GPU memory, but each streamed DiT layer also transfers fewer bytes. Packed linears reuse SRT's GGUF type definitions and CUDA dequantization, then run the native GEMM. SRT's fused MMVQ/MMQ kernels target the low-token LLM regime and are slower at diffusion sequence lengths. GGUF remains a capacity-oriented option; latency depends on the quantization type, activation shape, and placement policy. Layers the checkpoint stores unquantized (F32/F16/BF16) take the ordinary linear path rather than the packed one. Their precision is then whatever the model declares for that layer, exactly as on the safetensors path — a checkpoint cannot raise a layer above the model's own dtype by storing it wider. For MiniMax-H3 the two agree: the layers it pins to FP32 are the ones the validated checkpoint leaves unquantized, and `post_load_weights` fails the load if any of them ends up narrower. ### Usage No extra install: `gguf` is already a core SGLang dependency. `--model-path` stays the base model; `--component-weights-paths.` takes the GGUF. A local path, `owner/repo/file.gguf`, or `owner/repo:QUANT_TYPE` all work. The quant-type shorthand is accepted only when exactly one repository file matches it; otherwise SGLang lists the candidates and asks for a full path. ```bash theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --component-weights-paths.transformer \ leejet/MiniMax-H3-GGUF/minimax_h3_fl2va-Q4_K_M.gguf \ --num-gpus 1 \ --attention-backend fa \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder \ --dit-offload-prefetch-size 1 \ --dit-layerwise-resident-layers 0 \ --enable-torch-compile false \ --port 30010 ``` Here `transformer` is H3's registered DiT component name. The `--transformer-weights-path` alias produces the same result. Note that `--quantization gguf` is not the selector: the quantization is read from the file itself, so passing the file is what enables the path. MiniMax-H3 GGUF checkpoints may use either the original timestep MLP or the pruned AdaLN curve architecture. Repositories often contain both FL2VA and Ref2VA files, so use the full Hub file reference instead of an ambiguous `owner/repo:QUANT_TYPE` selector. The [H3 compatibility table](/cookbook/diffusion/MiniMax/MiniMax-H3#checkpoint-and-adapter-formats) links both published layouts without duplicating their commands here. The pruned architecture keeps its sampled curve and reduced AdaLN projections in FP32, matching the published checkpoint implementation. This precision island deliberately bypasses the BF16-only fused modulation kernels. #### Whether to offload the VAE Adding `vae` to `--layerwise-offload-components` trades a lot of latency for some peak VRAM, because the video VAE decoder is re-streamed per decode tile. Measured on 1× RTX 5090 with this checkpoint, 1344×768 × 107 frames: | `--layerwise-offload-components` | Peak VRAM | Denoise | VAE decode | | - | -: | -: | -: | | `dit,text_encoder` | 26.3 GiB | 39.0 s | **9.5 s** | | `dit,text_encoder,vae` | **19.6 GiB** | 39.0 s | 57.2 s | Denoise is unaffected, and the output is bit-identical either way. Leave the VAE resident unless the 6.7 GiB matters — on a 24 GB card by this measurement it does, and the 6× slower decode is the price of fitting. ### Constraints | Constraint | Reason | | - | - | | TP shard boundaries must align to GGML blocks | Column-parallel rows shard directly; row-parallel packed columns require each local input partition to contain whole quantization blocks | | No `--use-fsdp-inference` | FSDP does not preserve the GGUF packed-block layout | | CUDA only | The reused SRT GGML dequantization kernel currently ships for CUDA | | Native byte order only | A quantized block embeds its scales, so a non-native file cannot be byte-swapped as a whole | | No LoRA, and none of the H3 AdaLN cache flags | An adapter cannot be merged into packed blocks, and the AdaLN paths read the transformer's safetensors | | No `--quantization` | The checkpoint fixes the quantization; the flag would be a second, conflicting selector | Every constraint above fails at startup with an explanatory error rather than silently producing wrong output. Sequence parallelism (`--ulysses-degree` / `--ring-degree`) remains available because it shards activations rather than packed weights. TP is also available; startup rejects a degree that cuts a row-parallel matrix inside a GGML block. ### Validated scope | Model | Checkpoint | Hardware | Result | | - | - | - | - | | MiniMax-H3 `fl2va` | [`leejet/MiniMax-H3-GGUF`](https://huggingface.co/leejet/MiniMax-H3-GGUF) `minimax_h3_fl2va-Q4_K_M.gguf` (17.5 GiB) | 1x RTX 5090 (32 GiB) | t2va 1344x768, 107 frames, video + audio; 19.6-26.3 GiB peak depending on VAE offload | | MiniMax-H3 `fl2va` | `minimax_h3_fl2va-Q4_K_M.gguf` (17.5 GiB) | 1x H200 (141 GiB) | 2-step t2va 1344x768, 107 frames, H.264 + AAC; 10.13 s and 17.17 GiB peak | | MiniMax-H3 `fl2va`, pruned AdaLN curve | [`unsloth/MiniMax-H3-GGUF`](https://huggingface.co/unsloth/MiniMax-H3-GGUF) `minimax_h3_fl2va_pruned-Q4_K.gguf` (10.7 GiB loaded DiT) | 1x H200 (141 GiB) | 2-step t2va 1344x768, 107 frames, H.264 + AAC | | MiniMax-H3 `fl2va`, pruned AdaLN curve | `minimax_h3_fl2va_pruned-Q4_K.gguf` | 1x GB300 (CUDA 13, PyTorch 2.13) | 50-step t2va 1344x768, 107 frames, H.264 + AAC; 105.38 s and 80.88 GB peak | | MiniMax-H3 `fl2va`, pruned AdaLN curve | `minimax_h3_fl2va_pruned-Q4_K.gguf` | 2x GB300, TP2 (CUDA 13, PyTorch 2.13) | 2-step t2va 1344x768, 107 frames, H.264 + AAC; 7.55 s and 51.90 GB peak per rank | | Qwen-Image 2.1 | Private native-name Q4\_0 exports; DiT 3.91 GiB, encoder 7.03 GiB | 1x B200; separate TP2 check | 1024px/40-step generation, editing, and transparent RGBA; each component and both together; combined offload matched resident pixels | | Qwen-Image 2.1 | Community ComfyUI-GGUF DiT [`abenzerps/Qwen-Image-2.1-Uncensored-GGUF`](https://huggingface.co/abenzerps/Qwen-Image-2.1-Uncensored-GGUF) `qwen-image-2.1-UC-Q4_K_M.gguf` (4.6 GB) | 1x RTX 5090 (32 GiB) | 1024px/40-step text-to-image; 8.25 GB peak vs 17.0 GB for the BF16 DiT, 0.47 vs 0.43-0.48 s/step | The unpruned MiniMax-H3 DiT loads at 17.5 GiB against 61.7 GiB for the BF16 checkpoint. Weight fidelity was checked tensor-by-tensor against the BF16 reference: cosine 1.00000 for the F32/BF16 tensors and 0.9973 for Q4\_K/Q4\_0. The MiniMax-H3 measurements do not validate other quantization types, the `ref2va` partition, or a BF16-vs-GGUF output comparison. Qwen-Image 2.1's Q4\_0 exports use the base component configs and native tensor names. Its [cookbook](/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1#gguf-components) describes how to load compatible component files and choose precision settings. These private exports establish loading and execution compatibility, not general output quality or compatibility with other GGUF exports. ## Nunchaku (SVDQuant) ### Install Install the runtime dependency first: ```bash theme={null} pip install nunchaku ``` For platform-specific installation methods and troubleshooting, see the [Nunchaku installation guide](https://nunchaku.tech/docs/nunchaku/installation/installation.html). ### File Naming and Auto-Detection For Nunchaku checkpoints, `--model-path` should still point to the original base model, while `--transformer-weights-path` points to the quantized transformer weights. If the basename of `--transformer-weights-path` contains the pattern `svdq-(int4|fp4)_r{rank}`, SGLang will automatically: * enable SVDQuant * infer `--quantization-precision` * infer `--quantization-rank` Examples:
checkpoint name fragment inferred precision inferred rank notes
svdq-int4\_r32 int4 32 Standard INT4 checkpoint
svdq-int4\_r128 int4 128 Higher-quality INT4 checkpoint
svdq-fp4\_r32 nvfp4 32 fp4 in the filename maps to CLI value nvfp4
svdq-fp4\_r128 nvfp4 128 Higher-quality NVFP4 checkpoint
Common filenames:
filename precision rank typical use
svdq-int4\_r32-qwen-image.safetensors int4 32 Balanced default
svdq-int4\_r128-qwen-image.safetensors int4 128 Quality-focused
svdq-fp4\_r32-qwen-image.safetensors nvfp4 32 RTX 50-series / NVFP4 path
svdq-fp4\_r128-qwen-image.safetensors nvfp4 128 Quality-focused NVFP4
svdq-int4\_r32-qwen-image-lightningv1.0-4steps.safetensors int4 32 Lightning 4-step
svdq-int4\_r128-qwen-image-lightningv1.1-8steps.safetensors int4 128 Lightning 8-step
If your checkpoint name does not follow this convention, pass `--enable-svdquant`, `--quantization-precision`, and `--quantization-rank` explicitly. ### Usage Examples Recommended auto-detected flow: ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --transformer-weights-path /path/to/svdq-int4_r32-qwen-image.safetensors \ --prompt "a beautiful sunset" \ --save-output ``` Manual override when the filename does not encode the quant settings: ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --transformer-weights-path /path/to/custom_nunchaku_checkpoint.safetensors \ --enable-svdquant \ --quantization-precision int4 \ --quantization-rank 128 \ --prompt "a beautiful sunset" \ --save-output ``` ### Notes * `--transformer-weights-path` is the primary-DiT convenience spelling used by Nunchaku examples. Older config names such as `quantized_model_path` are treated as compatibility aliases. * Auto-detection only happens when the checkpoint basename matches `svdq-(int4|fp4)_r{rank}`. * The CLI values are `int4` and `nvfp4`. In filenames, the NVFP4 variant is written as `fp4`. * Lightning checkpoints usually expect matching `--num-inference-steps`, such as `4` or `8`. * Current runtime validation only allows Nunchaku on NVIDIA CUDA Ampere (SM8x) or SM12x GPUs. Hopper (SM90) is currently rejected. ## [ModelSlim](https://gitcode.com/Ascend/msmodelslim) MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware. * **Installation** ```bash theme={null} # Clone repo and install msmodelslim: git clone https://gitcode.com/Ascend/msmodelslim.git cd msmodelslim bash install.sh ``` * **Multimodal\_sd quantization** Download the original floating-point weights of the large model. Taking Wan2.2-T2V-A14B as an example, you can go to [Wan2.2-T2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-T2V-A14B) to obtain the original model weights. Then install other dependencies (related to the model, refer to the modelscope model card). > Note: You can find pre-quantized validated models on [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech). Run quantization using one-click quantization (recommended): ```bash theme={null} msmodelslim quant \ --model_path /path/to/wan2_2_float_weights \ --save_path /path/to/wan2_2_quantized_weights \ --device npu \ --model_type Wan2_2 \ --quant_type w8a8 \ --trust_remote_code True ``` For more detailed examples of quantization of models, as well as information about their support, see the [examples](https://gitcode.com/Ascend/msmodelslim/blob/master/example/multimodal_sd/README.md) section in ModelSLim repo. > Note: SGLang does not support quantized embeddings, please disable this option when quantizing using msmodelslim. * **Auto-Detection and different formats** For msmodelslim checkpoints, it's enough to specify only `--model-path`, the detection of quantization occurs automatically for each layer using parsing of `quant_model_description.json` config. In the case of `Wan2.2` only `Diffusers` weights storage format are supported, whereas modelslim saves the quantized model in the original `Wan2.2` format. For conversion, use the one-step `wan_repack.py` script: ```bash theme={null} python wan_repack.py \ --model-type Wan2.2-TI2V-5B \ --original-model-path {path_to_original_diffusers_model} \ --quant-path {path_to_quantized_model} \ --output-path {path_to_converted_model} ``` Supported `--model-type` values: `Wan2.2-TI2V-5B` (single-transformer), `Wan2.2-T2V-A14B` and `Wan2.2-I2V-A14B` (Cascade dual-transformer). The script automatically handles: copying the base model, converting quantized weights to Diffusers format, and restoring `config.json`. * **Usage Example** With auto-detected flow: ```bash theme={null} sglang generate \ --model-path Eco-Tech/Wan2.2-T2V-A14B-Diffusers-w8a8 \ --prompt "a beautiful sunset" \ --save-output ``` * **Available Quantization Methods**: * [x] `W4A4_DYNAMIC` linear with online quantization of activations * [x] `W8A8` linear with offline quantization of activations * [x] `W8A8_DYNAMIC` linear with online quantization of activations * [x] `W8A8_MXFP8` linear with offline quantization (msmodelslim pre-quantized weights) * [x] `mxfp8` linear with online quantization (`--quantization mxfp8`) * [x] `W4A4_MXFP4` / `W4A4_MXFP4_DUALSCALE` linear with offline quantization (msmodelslim pre-quantized weights) * [x] `mxfp4_npu` linear with online quantization (`--quantization mxfp4_npu`) ## MXFP8 Quantization Self-describing serialized MXFP8 transformers are inferred from their checkpoint metadata. For online MXFP8 quantization, load the original FP16/BF16 model and add `--quantization mxfp8`. NVIDIA and ROCm reuse SRT's dense MXFP8 implementation; Ascend uses `npu_dynamic_mx_quant` and `npu_quant_matmul` with a block size of 32. ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --quantization mxfp8 \ --prompt "a fox walking through neon rain" \ --save-output ``` The selected SRT backend must provide an MXFP8 kernel on NVIDIA or ROCm. On Ascend, the hardware requirement remains 950PR/DT Series or newer; `npu_dynamic_mx_quant` is not available on A2/A3 Series. ## MXFP8 Offline Quantization (msmodelslim) Pre-quantized MXFP8 weights exported by msmodelslim are auto-detected via `quant_model_description.json` (`W8A8_MXFP8` scheme). Use `wan_repack.py` to convert the quantized weights to Diffusers format, then load the converted model with `--model-path`: ```bash theme={null} sglang generate \ --model-path Eco-Tech/Wan2.2-T2V-A14B-Diffusers-mxfp8 \ --prompt "a beautiful sunset" \ --save-output ``` ## MXFP4 Online Quantization For online MXFP4 quantization on Ascend NPU, load the original FP16/BF16 model and add `--quantization mxfp4_npu`. The `mxfp4_npu` key is used for Ascend because `mxfp4` is reserved for the ROCm/aiter backend. Weights are quantized at load time via `npu_dynamic_dual_level_mx_quant`, and activations are quantized per-token during inference before `npu_dual_level_quant_matmul`. MXFP4 uses dual-level block scales with an L1 block size of 32 and an L0 block size of 512. ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --quantization mxfp4_npu \ --prompt "a fox walking through neon rain" \ --save-output ``` > **Hardware requirement:** Ascend 950PR/DT Series or newer. `npu_dynamic_dual_level_mx_quant` > and `npu_dual_level_quant_matmul` are not available on A2/A3 Series. > > **Note:** Online MXFP4 weight quantization is experimental. The offline msmodelslim > flow uses pre-quantized weights and may produce different numerical results. ## MXFP4 Offline Quantization (msmodelslim) Pre-quantized MXFP4 weights exported by msmodelslim are auto-detected via `quant_model_description.json` (`W4A4_MXFP4` / `W4A4_MXFP4_DUALSCALE` scheme). Use `wan_repack.py` to convert the quantized weights to Diffusers format, then load the converted model with `--model-path`: ```bash theme={null} sglang generate \ --model-path {path_to_converted_mxfp4_model} \ --prompt "a beautiful sunset" \ --save-output ``` The offline MXFP4 checkpoint stores weights in an FP8 container and includes dual-level scales (`weight_scale`, `weight_dual_scale`). If exported with smooth quantization, `mul_scale` is loaded and applied before activation quantization to keep activations aligned with the calibrated weights. # Realtime and Causal Video Models Source: https://docs.sglang.io/docs/sglang-diffusion/realtime_models Realtime and causal video pipelines generate video incrementally and reuse state across chunks. This differs from offline diffusion pipelines, which denoise one bounded latent sequence and release all request state when generation finishes. ## Execution Modes SGLang Diffusion exposes two related but distinct modes: | Mode | Lifetime | Interface | Examples | | - | - | - | - | | Realtime session | State persists until the client disconnects or the session ends | `/v1/realtime_video/generate` WebSocket | LingBot World, SANA-WM realtime | | Request-based causal generation | State is reused across chunks within one request, then released | Standard video generation API | LongLive 2.0, batch-streaming SANA-WM | The realtime server retains model-specific state such as the causal self-attention KV cache, cross-attention cache, decoder history, and pending control events. State is isolated per session and is not reused by unrelated requests. A causal DiT is not automatically a realtime session model. The pipeline must also register a realtime adapter and implement the WebSocket session lifecycle. ## Supported Realtime Pipelines | Model family | Pipeline | Live controls | QVG KV-cache quantization | | - | - | - | - | | LingBot World | `LingBotWorldCausalDMDPipeline` | Camera actions an